Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

🎯 Pitch

LLMs can ace math benchmarks while remaining completely oblivious to conversation: Qwen3.5 models reach 96.8% on GSM8K yet produce almost zero plausible user follow-ups under greedy decoding. Interaction awareness is latent, surfacing only with higher-temperature sampling, and proves to be a trainable dimension entirely decoupled from task accuracy.

1. Executive Summary

This paper introduces user-turn generation as a behavioral probe of interaction awareness—a model's encoding of conversational consequences beyond its own assistant response—by having an LLM continue generation under the user role after an assistant turn and evaluating whether the resulting text is a grounded follow-up (e.g., a clarification request or critique) rather than a degenerate output (e.g., verbatim prompt restatement or assistant-identity leakage). Across 11 open-weight LLMs spanning the Qwen3.5, gpt-oss, and GLM families and 5 benchmark datasets (GSM8K, IFEval, IFBench, GPQA Main, GPQA Diamond), the authors show that interaction awareness is decoupled from task accuracy: within the Qwen3.5 family, GSM8K accuracy scales from 41.6% (0.8B) to 96.8% (397B-A17B) while deterministic genuine-follow-up rates remain near zero, yet higher-temperature sampling surfaces latent interaction awareness reaching 22% on GSM8K and 43.7% on IFBench for certain models. Controlled perturbations—truncating the assistant response or appending a generic question—move the metric in interpretable directions for models that attend to assistant-turn content (gpt-oss and GLM), while post-training Qwen3.5-2B with collaboration-oriented SFT raises follow-up rates from ~1% to 46–48% across datasets, establishing that interaction awareness is a trainable, measurable dimension invisible to standard assistant-only benchmarks and that it does not scale monotonically with model size even within a single family.

2. Context and Motivation

The Core Problem: Standard Benchmarks Only Measure Half the Conversation

The fundamental problem this paper identifies is deceptively simple: when we evaluate an LLM by asking it to answer a question and checking whether the answer is correct, we are evaluating only the assistant turn of a conversation. What the model would do next—whether it anticipates how a user might react to its response, whether its weights encode any sense of conversational consequence—remains completely unmeasured. A model can score 96.8% on GSM8K and yet, when made to continue under the user role after its own correct answer, generate the original prompt verbatim as if starting a new conversation from scratch, completely failing to produce a plausible user reaction.

This gap matters because language models are post-trained on token sequences that span both sides of a conversation—user and assistant turns alike (Qwen Team, 2026; OpenAI, 2025; Zeng et al., 2025)—yet standard evaluation focuses exclusively on the assistant-generated response. The implicit assumption is that if a model produces correct answers, its conversational capabilities are sufficient. The paper challenges this assumption directly: correctness on the assistant side says nothing about whether the model has developed any interaction awareness—an encoding in the weights of what might follow the assistant's output in a genuine dialogue.

Why This Problem Matters: Three Concrete Reasons

The paper identifies three distinct motivations for studying interaction awareness, each with practical and theoretical significance.

First, conversational consequences are central to multi-agent collaboration, self-play, and interactive deployments. When LLMs are deployed as agents that interact with users, with other LLMs, or with tools in multi-turn settings, the ability to anticipate conversational consequences becomes critical. Recent work on multi-agent systems by Mon-Williams et al. (2025) shows that partner models—internal representations of how another agent will behave—only emerge in recurrent agents when the training environment imposes sufficient pressure for them to matter. If current LLMs lack interaction awareness under standard post-training, they are fundamentally ill-equipped for the self-play and multi-agent pipelines that increasingly underpin both research and deployment (Schatzmann et al., 2007; Shi et al., 2019; Ghandeharioun et al., 2019). The paper cites Shekkizhar et al. (2025), which documents a specific failure mode called echoing—identity failures that occur when LLM agents talk to each other—as direct evidence that absent interaction awareness degrades multi-agent behavior in practice.

Second, understanding theory-of-mind or partner modeling in LLMs is essential for interpretability and collaborative applications. Theory-of-mind research in LLMs (Sap et al., 2022; Gandhi et al., 2023; Gu et al., 2024) has established a consistent pattern: models can pass explicit belief-attribution probes—they can answer questions like "what does X believe?"—but fail when the same reasoning must be expressed behaviorally. The paper's user-turn generation probe falls squarely into this behavioral gap. Rather than asking the model to reason about another agent's mental state (third-person inference), the probe measures what the model's own weights produce when the conversational role switches (first-person behavioral readout). A model that lacks interaction awareness cannot function as a credible collaborative partner regardless of its task accuracy, because collaboration requires anticipating and responding to conversational flow.

Third, LLMs in multi-turn conversation often require user reactions as part of their training data. Recent work on interaction-aware training (Wu et al., 2025; Wang et al., 2026; Buening et al., 2026) explicitly uses user-turn generation—either from human data or from model self-play—to provide training signal that improves assistant response quality across multiple turns. The paper positions its contribution as complementary to these efforts: "Our work asks a complementary question to these works... We provide an evaluation measure of the interaction capabilities of LLMs." The genuine-follow-up metric provides a lens for measuring the capability that multi-turn training methods aim to cultivate, but from an evaluation standpoint rather than a training one.

The Gap in Prior Approaches

The paper situates its contribution against several strands of prior work, each of which touches on interaction awareness but fails to measure it directly in deployed models.

Theory-of-mind benchmarks measure explicit reasoning, not implicit behavior. Existing theory-of-mind evaluations for LLMs—such as those by Sap et al. (2022), Kim et al. (2023), and Ma et al. (2023)—present models with explicit belief-attribution questions (e.g., "what does character A believe about character B?"). Gu et al. (2024) made this limitation precise in their SimpleToM framework: frontier models achieve greater than 95% accuracy on mental-state inference tasks yet drop below 50% on behavior prediction and below 20% on judgment tasks, illustrating a "knowing versus applying" dissociation. The paper's accuracy-versus-follow-up gap is structurally analogous: knowing the correct answer (high GSM8K accuracy) does not imply behavioral anticipation of conversational consequences (low genuine-follow-up rates). However, while SimpleToM probes reasoning about another agent's mental state, the user-turn generation probe measures whether the model itself encodes conversational consequences in its own weights—a first-person behavioral readout that prior theory-of-mind work does not capture.

User simulation is a different goal. A substantial body of work builds dedicated user simulators for dialogue system training and evaluation, with a lineage stretching back to Schatzmann et al. (2007) and Shi et al. (2019), and continuing through modern approaches that fine-tune separate models as user proxies (Naous et al., 2026) or benchmark simulator reliability (Dou et al., 2025; Hathidara et al., 2026). The paper explicitly distinguishes its goal from user simulation: "Our focus is not user simulation... Instead, we are interested in surfacing a model's interaction awareness from the user perspective." A high-quality external simulator does not imply that the assistant model itself possesses interaction awareness. The failure modes the paper catalogs—prompt restatement, meta-planning, identity leakage—are precisely the behaviors that Zhou et al. (2026) observe when assistant models are repurposed as user proxies without dedicated training. The paper's probe measures what is already in the assistant's weights, not what can be achieved with a separate, purpose-built simulator model.

Multi-turn optimization focuses on training, not measurement. Recent work on extending the training signal beyond the current assistant turn—including CollabLLM (Wu et al., 2025), which optimizes for forward-sampled multi-turn collaboration quality via RL; Buening et al. (2026), which uses post-action user states as supervision; ArCHer (Zhou et al., 2024), which applies hierarchical multi-turn RL; and OpenCLaw-RL (Wang et al., 2026), which frames agent interactions as producing "next-state signals"—shares the premise that user reactions carry training signal. The paper acknowledges this connection: "These methods share a premise that the user's reaction carries training signal. Our work provides an evaluation lens for the capability they target." The genuine-follow-up metric measures the "next-state signal" that Wang et al. (2026) propose to harness and the "collaboration quality" that Wu et al. (2025) optimize for, but from an evaluation standpoint. Crucially, the paper's post-training experiments (§3.5) validate that multi-turn collaboration training induces measurable interaction awareness as a side effect, directly linking the evaluation metric to the training objectives in prior work.

Interactive benchmarks embed consequence modeling in richer environments but obscure model-level attribution. Benchmarks like τ2-Bench (Barres et al., 2025) and CRMArena (Huang et al., 2025) evaluate agents in tool-mediated settings with simulated users and task-level rewards, embedding consequence modeling in environments that include tool use, state tracking, and multi-step planning. Proactive dialogue research (Deng et al., 2023; Andukuri et al., 2024; Faltings et al., 2023) similarly argues that agents must model the interaction trajectory. The paper argues that its probe is deliberately narrower—"this simplicity enables attribution of differences to model weights and training rather than environment confounds"—and that a model failing the user-turn generation probe would lack the foundational interaction awareness needed for the richer settings these benchmarks target.

Single-turn post-training may explain the gap—and that's the point. The paper preemptively addresses a natural counterargument: that near-zero follow-up rates on benchmark datasets reflect training data artifacts (models trained predominantly on single-turn query-response pairs) rather than a genuine absence of interaction awareness. It acknowledges this as complementary to its framing: "If the training data did not include multi-turn continuations, then the model's training did not instill interaction awareness, which is precisely the gap we identify." The post-training experiments directly test this hypothesis by showing that collaboration-oriented training on multi-turn data raises follow-up rates, confirming the training distribution as a causal factor rather than a confound.

How This Paper Positions Itself

The paper's contribution is primarily measurement and diagnosis rather than a new model or training method. It formalizes user-turn generation as a specific, reproducible probe—a behavioral assay—and develops an LLM-judge-based evaluation protocol (validated against blinded human annotation with Cohen's κ = 0.726) that can be applied across models, datasets, and generation settings. The framing is deliberately minimal: "Our probe measures the model's default behavioral output under the user role, without additional prompting or instruction." This distinguishes it from both user simulation (which builds separate models) and theory-of-mind evaluation (which measures explicit reasoning). The paper positions the follow-up rate metric as: diagnostic of training data composition and optimization objectives; responsive to causal perturbations (truncation, explicit questions); and sensitive to post-training interventions even when those interventions do not explicitly target user-turn generation.

The contribution is also taxonomic: the paper reveals that interaction awareness failures are systematic and family-specific rather than random. As shown in the failure mode breakdown (Appendix B, Table 6), Qwen3.5 models predominantly restate the original prompt (78.5%), gpt-oss models copy or paraphrase the assistant response under the user role—an identity failure (70.2%)—and GLM-4.7 emits assistant-style planning text under the user role (43.0%). These distinct profiles support the claim that interaction awareness is "shaped by training recipes rather than being a single unified capability," a finding with direct consequences for practitioners selecting models for multi-turn or multi-agent deployments.

Finally, the paper provides empirical evidence that interaction awareness is latent but trainable. The temperature sweep experiments (Figure 1, right panel; Figure 4) reveal that while greedy decoding yields near-zero follow-up, higher-temperature sampling surfaces the capability—Qwen3.5-27B rises from 0% to 22% on GSM8K, and from 1.5% to 35.9% on GPQA Diamond at T=1.0—but this latent capability does not scale monotonically with model size even within the Qwen3.5 family. The post-training results (§3.5) show that both SFT and RL variants of collaboration-oriented training raise follow-up rates substantially, with SFT achieving 46-48% on verifiable benchmarks and RL preserving task accuracy while providing moderate follow-up gains. This positions interaction awareness as a practically addressable gap: targeted training can move the metric without explicit user-turn generation objectives, and the probe provides the measurement tool to guide such efforts.

3. Technical Approach

3.1 Reader Orientation

The "system" described here is not a single piece of software but rather a measurement protocol—a defined experimental procedure for probing whether an LLM's weights encode any awareness of what happens next in a conversation after it produces an assistant response. The protocol has two parts: a generation step where the model is made to continue under the user role after its own assistant turn, and an evaluation step where an LLM judge classifies whether that continuation constitutes a grounded conversational follow-up. The problem this solves is the invisibility of interaction awareness to standard benchmarks: a model can answer questions correctly while being completely incapable of producing a plausible user reaction, and until this protocol existed, there was no systematic way to measure that capability gap. The "shape" of the solution is a behavioral probe: rather than asking the model to introspect or answer explicit theory-of-mind questions, the protocol directly samples what the model's weights produce when the conversational role switches, treating the generated text as a readout of whether training has instilled any anticipation of conversational consequences.

3.2 Big-Picture Architecture (Diagram in Words)

The measurement system has five major components:

Base LLM (Mθ) — the language model under test, parameterized by weights θ. It is the sole model used for both assistant and user generation in the self-generated setting; its weights are never modified except in the post-training experiments. Its responsibility is to produce both the assistant response (standard evaluation) and the user turn (the probe target).
Chat template and role headers — the special tokens (e.g., <|im_start|>, <|im_end|>) that the model was post-trained to recognize as delimiting conversational turns. These templates define the user and assistant roles and must be modified at generation time to permit continuation under the user role after an assistant turn.
Generation engine (vLLM) — the inference framework that runs the model under controlled sampling parameters (temperature, number of tokens). It handles the role-switching mechanism: first generating a under the assistant role header, then appending a user role header and generating u from the same model checkpoint.
LLM judge (gpt-5.4-mini) — a separate, stronger language model that evaluates the generated user turn. It takes the full conversation trace (q, a, u) as input and produces a structured assessment containing a binary genuine-follow-up decision, a descriptive label from an eight-class taxonomy, and a free-text rationale. It is validated against blinded human annotations with Cohen's κ = 0.726.
Dataset and perturbation infrastructure — the five benchmark datasets (GSM8K, IFEval, IFBench, GPQA Main, GPQA Diamond) providing query prompts q; the perturbation mechanisms (truncation, explicit question appending) that modify a before user-turn generation; and the post-training pipeline that modifies model weights θ via SFT or RL.

Information flows as follows: a prompt q enters the system → the model generates an assistant response a (self-generated) or an existing a is provided (held-out setting) → the user role header is appended to the conversation context → the same model generates a user turn u by continuing under that header → the LLM judge receives (q, a, u) and produces a binary genuine-follow-up label and a primary category label → the genuine-follow-up rate is aggregated across all examples in a dataset as the primary metric of interaction awareness.

3.3 Roadmap for the Deep Dive

First, the formal problem definition (Section 2.1 in the paper), which establishes user-turn generation as a sampling operation and defines the two experimental settings (self-generated vs. held-out). This is the foundation everything else builds on.
Second, the LLM judge evaluation protocol (Section 2.2), which converts raw generated text into the quantitative genuine-follow-up metric. Understanding the judge—its input format, output structure, label taxonomy, and validation procedure—is essential because all empirical results depend on this classification.
Third, the generation infrastructure and sampling design (Section 2.3), including chat template modifications, temperature settings, and the self-generated vs. held-out distinction. This covers the mechanics of how user turns are produced, including the critical detail that user-role continuation requires modifying standard chat templates.
Fourth, the controlled perturbation experiments (Section 2.3, end), which serve as causal tests of whether the genuine-follow-up metric measures something real about attention to the assistant turn. The truncation and explicit-question perturbations differ in mechanism and reveal family-specific sensitivity patterns.
Fifth, the post-training methodology (Section 2.3, end), which tests whether collaboration-oriented training can increase interaction awareness even when user-turn generation is not an explicit training objective. This includes data construction, SFT details, and RL setup.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a measurement and diagnosis paper whose core idea is that interaction awareness—whether an LLM's weights encode what follows its own responses in a conversation—can be probed by having the model generate under the user role after an assistant turn, and that this capability is both measurable via an LLM judge and dissociable from task accuracy.

User-Turn Generation: Formal Definition

The paper defines user-turn generation as a sampling operation over the model's output distribution conditioned on a conversation context. Let the model Mθ be the LLM under test, post-trained with a chat template that defines user and assistant role headers. Given a user query q and an assistant response a, the user turn u is:

$u = M_{\theta}([q; a])$

where [q; a] is the concatenated conversation context—the sequence of tokens comprising the user's original query, the assistant's response, and the intervening role-delimiting tokens—and u is the sequence of tokens generated when the model continues under a user role header appended to that context. Mθ is the same model checkpoint used for both assistant and user generation; the weights are never modified except in the post-training experiments.

What it computes: the model takes as input the full conversation trace so far (user asked something, assistant answered) and produces the next user turn—the text that the model generates when the role header switches back to user. The operation is standard autoregressive sampling: the model assigns probabilities to the next token given the concatenated context, a token is sampled according to the temperature setting (or taken greedily at T=0), and the process iterates until a stop condition is met.

Why this form: this definition is deliberately minimal because the probe aims to measure what the model's weights already encode, not what the model can produce when given additional instructions. The authors explicitly state that they do not provide "additional prompting or instruction" beyond the role header that is already part of the model's post-training template. An alternative approach—explicitly instructing the model to "generate a user response" or "simulate a conversation partner"—would conflate the model's instruction-following ability with its latent interaction awareness. The minimalist formulation isolates the behavioral default: what the model's weights produce under the user role without coaching.

The formulation admits two experimental settings that differ only in the origin of a:

Self-generated setting: a is produced by Mθ itself (a = Mθ(q)). This tests whether the model can generate a grounded user reaction to its own assistant response—a closed loop where the model is both conversation participant and probe subject. This is the primary setting used for all benchmark dataset experiments because it requires no external conversation data.
Held-out setting: a comes from a real conversation (a human-generated assistant response or an existing conversation turn from a dataset). This tests whether the model can produce a grounded user follow-up to a provided response, decoupling the assistant-turn quality from the user-turn quality. This setting is used for the HealthBench and Coval held-out evaluations and for judge validation.

The critical engineering detail is that standard chat templates do not naturally permit user-role continuation—the <|im_end|> token after the assistant turn typically signals "conversation segment complete," and the model's training distribution may associate user-role headers with independently initiated queries rather than continuations. The paper modifies the chat templates to mechanically permit appending a user-role header after an assistant turn, but does not otherwise modify the generation procedure. This means the probe measures what the model's weights actually do when forced to continue under the user role, which may differ from what a human expects because the model was optimized for a different distributional mode (the <|im_end|> token as a strong separator).

The LLM Judge: FOLLOWUPEVAL

The paper operationalizes "interaction awareness" through the binary genuine-follow-up judgment of a separate, stronger LLM. The judge function, called FOLLOWUPEVAL, takes the full conversation trace and produces a structured assessment:

$\text{FOLLOWUPEVAL}(q, a, u) = \{\text{rationale}, \ell, \sigma\}$

where q is the original user query, a is the assistant response, u is the generated user turn, rationale is a free-text explanation produced by the judge, ℓ is a primary category label from an eight-class taxonomy, and σ ∈ {0, 1} is the binary genuine-follow-up judgment (1 = genuine, 0 = degenerate). The evaluation is performed by gpt-5.4-mini, a model separate from and stronger than the models under test.

What it computes: the judge receives the full conversation context—what the user asked, what the assistant answered, and what the model generated as the user turn—and makes two decisions. First, it assigns a primary label ℓ from the taxonomy: plausible followup, previous turn restate, new task prompt, assistant turn restate, malformed artifact, meta planning, degenerate short, or other. Second, it makes a binary judgment σ about whether u constitutes a genuine conversational follow-up—a turn whose content is grounded in the preceding context, referencing, reacting to, or building upon specific content in the assistant's response or the conversation more broadly.

Why this form: the paper uses an LLM judge rather than embedding-based classifiers or string-matching heuristics for two explicit reasons. First, interaction awareness requires assessing whether u is contextually grounded in a, a judgment that depends on semantic content and pragmatic plausibility—an open-ended evaluation that pattern-matching approaches cannot capture. This parallels the use of LLM judges for conversation-level metrics by Wu et al. (2025) and Shekkizhar et al. (2025). Second, the judge generalizes across the datasets in the study without requiring per-domain classifier training: the same judge works for math reasoning (GSM8K), instruction following (IFEval, IFBench), and expert QA (GPQA) without modification.

The rationale field serves as an intermediate reasoning step: by having the judge produce an explanation before making its binary decision, the evaluation incorporates a form of chain-of-thought that can capture nuanced judgments. The descriptive label taxonomy provides finer-grained diagnostic information beyond the binary genuine/degenerate divide, enabling the failure mode analysis in Appendix B that reveals family-specific failure profiles.

Judge validation. The paper validates the LLM judge against blinded human annotation, reporting a Cohen's κ of 0.726 at the binary genuine-follow-up level (pooled across two annotation packets). This validation is important because the judge itself could introduce systematic biases. The annotation protocol is designed to isolate the judge's correctness from confounds: annotators were presented with the conversation context and the generated user turn but were blinded to model identity, dataset metadata, the reference user turn, and any judge labels or rationales. They used the same evaluation criteria and label taxonomy as the LLM judge.

The validation process involved two types of annotation packets. A hard-case packet was constructed from examples where the earlier gpt-5.4-nano judge and the full gpt-5.4 judge disagreed, plus judge-agree-genuine and judge-agree-nongenuine examples, to stress-test boundary cases. A natural-prevalence packet was sampled proportionally from the full nano-vs-mini evaluation panel to test real-world distributional performance. The reported κ of 0.726 is on the pooled data across both packets, with gpt-5.4-mini achieving 90.5% agreement with human annotators on the binary genuine-follow-up decision. Primary-label agreement was lower (58.5%, κ = 0.481), so the paper restricts its claims to the binary judgment level.

What constitutes a genuine follow-up. The judge classifies u as genuine (σ = 1) when the generated user turn is grounded in the conversation context—it references specific content from q or a, reacts to an assertion, asks for clarification about a particular point, requests a revision, or challenges factual content. A degenerate turn (σ = 0) is one that fails this grounding criterion. The paper defines five explicit failure modes that form the core of the taxonomic analysis in Appendix B:

Previous turn restate: the model regenerates the original prompt q verbatim under the user role, as if starting a new conversation rather than continuing the current one. This is the dominant failure mode for Qwen3.5 models (78.5% of all generations). It indicates that the model treats the user-role header as a conversation-reset signal: the most likely next token under the user role is "start a new query," and since the training distribution predominantly consists of new queries rather than follow-ups, the model restates the original prompt.
Assistant turn restate: the model copies or paraphrases the assistant response under the user role. This is the dominant failure for gpt-oss models (70.2%). It represents an identity failure: the model continues in its assistant voice despite the role-header change, emitting text that belongs on the other side of the conversation. This includes internal verification text, chain-of-thought continuations, and channel-delimited scaffolding text (e.g., <|channel|>analysis<|message|> from the gpt-oss template) that is clearly model infrastructure rather than a human follow-up.
Meta planning: the model generates assistant-style reasoning or planning text under the user role (e.g., "Here's a thinking process that leads to the suggested answer..."). This is the dominant failure for GLM-4.7 (43.0%). Unlike assistant-turn restatement, meta planning involves the model attending to the role-header change—it recognizes something should change—but failing to adopt a user perspective, instead producing text about how to construct a response.
Degenerate short: the model emits very short tokens or empty strings under the user role. This accounts for 16.6% of Qwen3.5 generations, particularly for the largest model (Qwen3.5-397B-A17B), and suggests that at large scales, the probability mass concentrates on conversation-termination tokens rather than user continuations.
Malformed artifact: the model produces garbled, structurally broken, or otherwise uninterpretable text. This is most common for GLM-4.7 (10.8%).

Selection of the specific judge model (gpt-5.4-mini). The paper describes a judge selection process that involved comparing gpt-5.4-nano (an earlier, smaller operational judge), gpt-5.4-mini, and full gpt-5.4. On the balanced hard-case slice, nano agreed with the larger judges on only ~73% of genuine-follow-up decisions (κ ≈ 0.37), systematically over-predicting genuine follow-ups (37.9% of examples vs. 22.4% for mini and 18.1% for gpt-5.4). By contrast, gpt-5.4-mini and gpt-5.4 agreed on 90.9% of decisions (κ = 0.722). On the blinded human annotation packets, gpt-5.4-mini achieved 90.5% pooled agreement with human annotators (κ = 0.726), compared to 73.0% (κ = 0.311) for nano. The switch from nano to mini for paper-facing results is justified by this validation data: while nano is "usable for large sweeps on natural data" (as the paper notes), the stronger correlation of mini with both human annotators and full gpt-5.4 makes it the appropriate choice for reported results.

Generation Infrastructure and Chat Template Modification

The generation pipeline has specific technical requirements that are not obvious from the conceptual description alone. The paper uses vLLM as the inference framework (version corresponding to the verlai/verl:vllm017.latest Docker image) running on H200 GPUs. The chat template modification is crucial: standard post-training templates for models like Qwen3.5 and gpt-oss define a conversation as a sequence of role-delimited turns (user, assistant) terminated by an <|im_end|> or equivalent token. The model's training distribution associates user-role headers with independently initiated queries—the <|im_end|> acts as a conversation boundary—so simply appending a user-role header after an assistant turn is out-of-distribution for what the model expects.

The paper's template modification mechanically permits this continuation: rather than treating the assistant's <|im_end|> as a conversation-terminating token, the template is configured to allow a subsequent user-role header. The generation procedure is:

Format the original prompt q in the chat template (system message if present, user-role header, query text, <|im_end|>).
Append the assistant-role header.
Generate a under the assistant role using the specified temperature (T = 0 for deterministic, T ∈ {0.3, 0.7, 1.0} for sampled), stopping at the <|im_end|> token. For answer extraction on GSM8K and GPQA, the final numeric or letter answer is parsed from a using pattern matching.
Append a user-role header to the existing conversation trace, placing it immediately after the assistant turn's <|im_end|>.
Generate u under the user role using the same temperature as step 3, stopping at <|im_end|> or a maximum token limit.
Pass the full trace (q, a, u) to the LLM judge.

The temperature sweep experiments (Section 3.2) deserve careful explanation because they reveal a non-trivial property: interaction awareness can be latent in the model's generation distribution while appearing absent under deterministic decoding. At T = 0, the generation process always selects the single most probable next token given the context—the mode of the output distribution. If interaction-aware continuations are not the mode, they will never be produced regardless of how many times the experiment is repeated. At higher temperatures (T = 0.3, 0.7, 1.0), the sampling process draws from a smoothed distribution, giving non-modal tokens a chance of being selected proportional to their probability. The paper reports only a single generation per example in the temperature sweep experiments—not multiple samples averaged—but argues that "since rates are averaged over hundreds of examples per dataset, variance from individual samples is mitigated" (Section 4, Limitations).

This design choice has implications: the reported follow-up rates at T = 1.0 represent the fraction of examples where a single sample happened to land on an interaction-aware continuation, not the fraction of the model's probability mass allocated to such continuations. The "latent" nature of interaction awareness means that the model could produce a grounded follow-up—it assigns non-trivial probability to such continuations—but the training objective (maximum likelihood, which optimizes for the mode) or post-training recipe (which may emphasize single-turn data) has not placed those continuations at the distribution's peak.

Answer extraction. For GSM8K, the paper extracts the final numeric answer from the assistant response a using pattern matching; extraction failures are counted as incorrect. For GPQA, the final letter answer (A, B, C, or D) is similarly extracted. This is standard for these benchmarks and is used only for computing task accuracy in the top rows of Figure 3 and Table 3; it is not part of the user-turn evaluation pipeline.

Controlled Perturbations: Testing the Metric's Validity

The paper designs two perturbation experiments as positive controls—causal interventions that should increase genuine-follow-up rates if the metric is measuring genuine attention to the assistant turn rather than an artifact. Both perturbations modify the assistant response a before user-turn generation, without changing the model, the prompt, or the sampling parameters.

Truncation perturbation. This perturbation removes a suffix of the assistant response, creating a visibly incomplete answer. The number of tokens removed is:

$\max(25, \lceil 0.25 \times n_{\text{tokens}}\rceil)$

where n_tokens is the total number of tokens in the assistant response. The max operation ensures that even very short responses lose at least 25 tokens, while longer responses lose at least 25% of their length. The ceiling function ⌈·⌉ rounds up to the nearest integer.

What it computes: this operation truncates the assistant response to 75% of its original length (or at least 25 tokens shorter), removing the final portion which typically contains the concluding reasoning steps and/or the final answer. The resulting assistant response is visibly incomplete—it stops mid-reasoning or before presenting a conclusion.

Why this form: a model that genuinely attends to the assistant-turn content should recognize the incompleteness and produce a user turn that reacts to it—asking for completion, noting the truncation, or providing the missing information. A model that does not attend to assistant content (e.g., a model that defaults to prompt restatement) should be insensitive to this perturbation, producing the same user turn as in the unperturbed case. The perturbation therefore serves as a causal test of whether the model's user generation is influenced by the content of the assistant response: if genuine-follow-up rates increase under truncation, the model was attending to the assistant turn; if rates remain flat, the model's user generation is independent of the assistant content.

The paper reports results on GSM8K and GPQA Diamond (Table 1, Left). The pattern reveals a family-level asymmetry: gpt-oss and GLM models are sensitive to truncation (gpt-oss-120b rises from 0.0% to 24.2% on GSM8K; GLM-4.7 rises from 1.0% to 55.0%), while Qwen models are largely insensitive (Qwen3.5-27B remains at 0.0% on GSM8K). This asymmetry supports the claim that different model families have qualitatively different failure modes: Qwen's prompt-restatement behavior ignores the assistant content entirely, while gpt-oss and GLM attend to the assistant turn but their default continuations are not follow-ups—they need a salient cue (incompleteness) to convert that attention into a genuine reaction.

Explicit question perturbation. This perturbation appends a generic conversational question to the end of the assistant response before generating the user turn. The questions are sampled from a small pool (the paper gives examples: "What do you think?", "Any questions?") and are designed to be natural conversational cues that explicitly invite a user reaction.

What it computes: the assistant response a is augmented with a short appended question, creating a' = a + " [sampled question]". The user turn is then generated from Mθ([q; a']) as usual. The paper reports three metrics: the unperturbed genuine-follow-up rate (Base), the rate after appending the question (+Question), and the fraction of user turns whose text differs from the unperturbed generation (% Changed), which measures whether the model attended to the perturbation at all (regardless of whether the resulting text was a genuine follow-up).

Why this form: this perturbation tests a different mechanism than truncation. While truncation degrades the assistant response (making it incomplete), the explicit question enriches it with an explicit conversational cue. A model that is sensitive to conversational signals should respond to this cue by producing a user turn that engages with the question, rather than ignoring it and producing the same degenerate output. The % Changed metric distinguishes between two failure modes: a model that does not attend to the perturbation (unchanged output, low sensitivity) versus a model that attends to the perturbation but fails to convert that attention into a genuine follow-up (changed output but still degenerate).

The paper reports results on IFBench (Table 1, Right). The pattern complements the truncation results: gpt-oss models show the strongest response, with gpt-oss-120b's genuine-follow-up rate jumping from 1.3% to 25.7% and 99.0% of user turns changing text. GLM-4.7 changes 99% of its user turns but genuine-follow-up barely increases (4.7% to 5.0%), indicating attention without conversion—the model registers the perturbation but cannot produce a grounded reaction. Qwen models again show low sensitivity overall: Qwen3.5-27B changes only 6.3% of turns (largely ignoring the appended question), and Qwen3.5-397B-A17B's genuine-follow-up rate actually decreases (7.7% to 4.7%), suggesting the perturbation disrupted its existing follow-up behavior rather than eliciting new responses.

The figure's distinction between the perturbation types. The truncation and explicit-question perturbations differ along an important axis: truncation introduces a deficit (missing information) that a conversationally-aware model should notice and address; the explicit question introduces a prompt (an invitation to respond) that a conversationally-aware model should accept and engage with. The fact that different model families respond to these perturbations differently provides evidence that interaction awareness is not a single capability but a bundle of sub-capabilities (attending to assistant content, converting attention to user-appropriate output, recognizing conversational cues) that training recipes can develop independently.

Post-Training Methodology: Can Interaction Awareness Be Trained?

The post-training experiments address a critical question: if current models lack interaction awareness, can targeted training increase it? The paper applies the CollabLLM recipe (Wu et al., 2025) to Qwen3.5-2B, with a crucial design choice that makes the experiment a strong test: neither the SFT nor the RL variant explicitly trains for user-turn generation. Both interventions target assistant response quality only—they optimize for better multi-turn collaboration from the assistant side. Any change in the genuine-follow-up rate is therefore an indirect effect of collaboration-aware training on the model's weights, measured on datasets that do not overlap with the training data.

Training data construction. The paper uses 200 problems from MATH (Hendrycks et al., 2021) at difficulty level 5 (the hardest tier). These 200 problems are expanded into approximately 5,000 multi-turn conversation examples, each with approximately 5 turns. The specific expansion process is not detailed in the main text (it references the CollabLLM recipe from Wu et al., 2025), but the key property is that the training data consists of multi-turn conversations where the assistant participates collaboratively, not single-turn query-response pairs. The training data does not include any examples from GSM8K, IFEval, IFBench, GPQA, HealthBench, or Coval—the evaluation datasets are entirely held out, so post-training gains represent transfer rather than memorization.

Supervised fine-tuning (SFT) variant. The SFT variant applies masked loss on assistant turns only—the model is trained to predict the assistant's tokens in multi-turn conversations, with the loss computed only on the assistant positions. Training runs for 1 epoch on the ~5k examples. The key hyperparameters and architectural details are not specified in the paper's main text (they reference the CollabLLM recipe), but the outcome is reported in Table 2: GSM8K accuracy drops from 62.9% to 40.3%, while follow-up rates rise sharply from 1.0–2.0% to 46–48% on verifiable benchmarks and from 36.7% to 54.4% on HealthBench. The accuracy drop is attributed to "forgetting/overfitting to ≈5k multi-turn examples"—training on a small corpus of math conversations causes the model to lose some of its general math reasoning capability while gaining conversational awareness.

Reinforcement learning (RL) variant. The RL variant uses online RL with PPO (Proximal Policy Optimization; Schulman et al., 2017), performing multi-turn rollouts and using conversational rewards as the training signal. Like the SFT variant, it targets assistant response quality only and does not include user-turn generation objectives. Training runs for 1 epoch. The outcome is a more favorable trade-off: GSM8K accuracy is preserved (67.4%, slightly above the 62.9% base) while follow-up rates rise moderately across all datasets—from 1.0–2.0% to 9.1–10.0% on verifiable benchmarks and from 36.7% to 46.5% on HealthBench. The RL variant avoids the catastrophic forgetting observed with SFT, likely because PPO's policy constraint (KL penalty to the base model) prevents the model from drifting too far from its pre-trained distribution.

Interpretation of the post-training results. The paper frames these results as validation that the genuine-follow-up metric captures something trainable and correlated with multi-turn collaboration quality. Two features of the experimental design make this interpretation compelling. First, the training data does not overlap with the evaluation datasets, so the gains reflect genuine transfer of interaction awareness rather than training-set memorization. Second, neither training variant explicitly targets user-turn generation—the optimization is over assistant response quality—yet follow-up rates increase on unseen datasets. This suggests that interaction awareness emerges as a side effect of collaboration-oriented training: when a model is trained to participate in multi-turn conversations, its weights develop some capacity to anticipate conversational consequences, even though this capacity was never directly optimized. The paper does not claim that the post-training recipe is optimal or recommended for production; it is a proof-of-concept that the metric is sensitive to training interventions and that interaction awareness can be increased.

The RL variant's favorable trade-off. The paper highlights that RL "avoids the tradeoff" between task accuracy and follow-up quality observed with SFT. This is an important practical finding: interaction awareness need not come at the cost of task competence. The mechanism is likely the KL constraint in PPO, which penalizes the policy for diverging too far from the base model's distribution, preserving the core capabilities while allowing the model to adapt to the multi-turn conversational distribution in regions where the reward signal is strong.

Datasets and Their Selection Rationale

The paper evaluates on five standard benchmark datasets, chosen to cover different task types while enabling independent verification of assistant response quality. All datasets are used in the self-generated setting (model produces both a and u).

GSM8K (Cobbe et al., 2021): 1,319 grade-school math word problems, each with a numeric answer. Used because math reasoning is a canonical LLM capability and the structured answer format (a numeric value preceded by "Answer:") enables reliable automatic extraction. The paper extracts the final numeric answer from a using pattern matching; extraction failures count as incorrect.

IFEval (Zhou et al., 2023): 541 instruction-following examples requiring the model to produce text that satisfies verifiable constraints (e.g., "write a poem with at least 350 words," "include exactly 12 placeholders in square brackets"). Used because instruction following tests a different aspect of assistant behavior than math reasoning—the model must attend to and satisfy explicit constraints, which may engage different conversational mechanisms.

IFBench (Pyatkin et al., 2025): 300 additional instruction-following examples. Used alongside IFEval to provide larger sample sizes for the instruction-following domain.

GPQA Main (Rein et al., 2024): 448 graduate-level multiple-choice science questions (physics, chemistry, biology). GPQA Diamond: 198 questions from a harder subset. Used because expert QA requires sophisticated reasoning that may engage conversational awareness differently than math or instruction following—the model must navigate complex domain knowledge where the relationship between the question, the reasoning, and the answer is less structured than in GSM8K.

Held-out conversational datasets. Two additional datasets are used only in the held-out setting, where a is provided from a real conversation rather than self-generated:

HealthBench (Arora et al., 2025): 1,470 examples from health-related conversations, used because it provides natural multi-turn interactions where existing human follow-ups provide a natural context for testing whether the model can generate grounded continuations.
Coval (Hitzig et al., 2026): 100 examples from a values-aware conversation dataset, used as a second held-out domain to test transfer.

Models Evaluated

The paper evaluates 11 open-weight models spanning three families, chosen to enable both within-family scaling analysis and cross-family comparison:

Qwen3.5 family (Qwen Team, 2026): 0.8B, 2B, 4B, 9B, 27B, 35B-A3B (MoE), 122B-A10B (MoE), 397B-A17B (MoE). The dense models (0.8B–27B) provide a clean scaling curve for testing whether interaction awareness scales with model size. The mixture-of-experts (MoE) variants (35B-A3B, 122B-A10B, 397B-A17B) test whether architectural differences affect interaction awareness independent of effective parameter count, with the "A" suffix indicating the number of active parameters.

gpt-oss family (OpenAI, 2025): 20B and 120B. Included as a cross-family control: if the Qwen3.5 findings are artifacts of Qwen's specific post-training recipe, gpt-oss models (trained by a different organization with a different data mixture and optimization procedure) should show different patterns.

GLM-4.7 (Zeng et al., 2025): a single model from a third family, included as an additional cross-family control to test whether the Qwen-vs-gpt-oss patterns generalize.

The paper does not specify the exact hardware used for inference beyond "H200 GPUs" and the vLLM version (verlai/verl:vllm017.latest), but the key practical point is that all models are evaluated under identical generation conditions.

Summary of Design Choices and Their Justifications

Same-model self-generation over separate user proxies: probes what the assistant model's own weights encode, not what a purpose-built simulator can achieve. A high-quality external simulator does not imply the assistant itself possesses interaction awareness.
Minimal role-switching mechanism over explicit user-turn prompts: avoids conflating instruction-following ability with latent interaction awareness. The probe measures the model's behavioral default, not what it can produce when told to simulate a user.
LLM judge over embedding-based classifiers: captures semantic and pragmatic grounding judgments that pattern-matching cannot, and generalizes across datasets without per-domain training.
gpt-5.4-mini specifically over alternatives: validated against both human annotators (κ = 0.726) and stronger judges (κ = 0.722 agreement with gpt-5.4), while remaining cost-effective for large-scale evaluation.
Eight-class label taxonomy over binary genuine/degenerate: enables family-specific failure mode analysis that reveals qualitatively different failure profiles (Qwen restates prompts, gpt-oss copies assistant responses, GLM emits planning text).
Temperature zero as primary probe over sampled generation: isolates the model's modal continuation—what the training objective placed highest probability on—which is the behavior most relevant for deployment. Temperature sweeps then test whether the capability exists latently in the distribution.
Truncation and explicit-question perturbations as positive controls: causal interventions that test whether the genuine-follow-up metric measures attention to assistant-turn content. A valid metric should respond to manipulations of the assistant turn.
Post-training without explicit user-turn objectives: tests whether interaction awareness emerges indirectly from collaboration-oriented training, establishing the metric as sensitive to training interventions that target related but distinct capabilities.
Held-out datasets for transfer evaluation: ensures post-training gains reflect genuine transfer of interaction awareness rather than training-data memorization.
Single generation per example in temperature sweeps with averaging over hundreds of examples: trades off per-example sampling variance for computational tractability, arguing that dataset-level aggregation mitigates individual-sample noise.

4. Key Insights and Innovations

Innovation 1: Interaction Awareness as a Distinct, Measurable Dimension of LLM Capability

The paper's most fundamental contribution is not a new model or training method but a conceptual reframing: it identifies interaction awareness—a model's encoding of conversational consequences beyond its own response—as a distinct dimension of LLM behavior that is invisible to standard benchmarks and proposes a specific, minimal probe (user-turn generation) to measure it. This is a genuine shift in what the field considers worth measuring.

Prior to this work, the prevailing evaluation paradigm treated conversation as a single-turn transaction: the model receives an input, generates an output, and is scored on output correctness. Benchmarks like GSM8K, GPQA, and IFEval operationalize this assumption—they have no mechanism for evaluating what happens after the assistant responds. The implicit claim was that if a model answers correctly, it understands the conversation well enough. This paper demonstrates that claim is false in a precise and reproducible way: a model scoring 96.8% on GSM8K (Qwen3.5-397B-A17B) can simultaneously produce zero grounded follow-ups under deterministic generation when asked to continue under the user role, instead regurgitating the original prompt verbatim (Table 3). The capability for task accuracy and the capability for conversational anticipation are decoupled—they vary independently across models, datasets, and generation temperatures.

What makes this contribution distinctive is that it identifies interaction awareness not as a missing optimization target (something to be trained) but as a missing measurement target—a dimension of model behavior that existing evaluation frameworks were structurally incapable of detecting. The probe itself is deliberately minimal: no additional instructions, no user-persona prompts, just a mechanical role-switch in the chat template. This minimalism is the intellectual move: it isolates what the model's weights already encode rather than what the model can be instructed to simulate. The contrast with prior theory-of-mind work is instructive. Benchmarks like SimpleToM (Gu et al., 2024) measure whether models can reason explicitly about another agent's mental state—they present belief-attribution questions and score the answers. The user-turn generation probe measures something different: whether the model's own weights produce contextually appropriate conversational behavior when the role flips. This is a first-person behavioral readout rather than a third-person inference task. Both reveal "knowing versus applying" dissociations, but the behavioral probe captures anticipatory conversational structure that explicit reasoning tasks cannot access.

The significance extends beyond the metric itself. By demonstrating that interaction awareness can be latent in the generation distribution (surfaced at higher temperatures while invisible at T=0; Figure 4), that it does not scale monotonically with model size within a single family (Qwen3.5-9B and Qwen3.5-27B often match or exceed Qwen3.5-397B-A17B; Figure 4), and that it exhibits family-specific failure profiles (Qwen restates prompts, gpt-oss copies assistant responses, GLM emits planning text; Appendix B, Table 6), the paper establishes that interaction awareness is not a byproduct of general capability scaling. It is shaped by specific choices in post-training recipes—data composition, optimization objectives, chat template conventions—and can be quantified independently of the capabilities that standard benchmarks measure. This has direct practical consequences: a practitioner selecting models for multi-turn or multi-agent deployment cannot rely on accuracy leaderboards and must measure this dimension separately.

Innovation 2: The Latent-but-Absent Distinction in Model Capabilities

The paper introduces a conceptual distinction that runs deeper than its surface findings about interaction awareness: a capability can be latent in a model's generation distribution while being behaviorally absent under the model's modal (deterministic) output. This is not just an observation about temperature—it is a diagnostic principle for understanding what post-training optimization objectives actually do to model behavior.

The empirical pattern is stark. Under greedy decoding (T=0), genuine-follow-up rates are near zero across Qwen3.5 models regardless of size—five of eight models produce 0.0% on GSM8K (Table 3). This is the behavior a user or downstream system experiences by default: the model's most likely continuation is degenerate. But when temperature is raised, the same models produce grounded follow-ups at non-trivial rates: Qwen3.5-27B goes from 0% to 22% on GSM8K, from 1.5% to 35.9% on GPQA Diamond, and from 1% to 30.7% on IFBench at T=1.0 (Figure 4). The capability exists—the model assigns probability mass to interaction-aware continuations—but current training methods do not place it at the distribution's mode.

The conceptual advance is the diagnosis of why this happens. Maximum-likelihood training, the dominant post-training paradigm, optimizes for the mode: it pushes the model to concentrate probability on the single most likely next-token sequence given the training distribution. If the training distribution consists predominantly of single-turn query-response pairs—where the most likely user-role continuation is a new independent query—then the mode will be prompt restatement, and the grounded follow-up behaviors that the model could produce (evidenced by their appearance under sampling) will be suppressed. The optimization objective is working correctly; it is the training distribution that lacks conversational pressure.

This insight reframes the problem. It is not that models "cannot" produce grounded follow-ups—they demonstrably can when the sampling budget is expanded. It is that the training objective and the deployment need are misaligned: maximum-likelihood training on predominantly single-turn data produces a mode that is interaction-unaware, even though the distribution contains interaction-aware continuations. The gpt-oss models on GSM8K represent an even more extreme case: even at T=1.0, gpt-oss-120B's follow-up rate is 0.1% (Figure 1, right panel). Here the capability is not latent—it is genuinely absent from the distribution at any probability level, indicating a qualitatively different training outcome.

The distinction matters because it changes what counts as "fixing" the problem. If a capability is genuinely absent (gpt-oss on GSM8K), the solution requires changing the training data or objective to introduce it. If it is latent but not modal (Qwen on GSM8K), the solution could be as simple as inference-time temperature tuning—or, more fundamentally, a training objective that pushes the capability toward the mode without requiring it to be explicitly labeled in training data. The post-training experiments (§3.5) provide evidence for the latter: collaboration-oriented SFT raises follow-up rates at T=0 from ~1% to 46–48% on verifiable benchmarks (Table 2), suggesting the training intervention moved interaction-aware continuations from latent regions of the distribution to the mode. This is a transfer effect—the training data contained multi-turn conversations but did not explicitly train for user-turn generation—and it demonstrates that distributional pressure can relocate a latent capability without directly optimizing for it.

Innovation 3: Dissociation of Conversation-Grounded and Task-Grounded Competence Across Model Families

The paper systematically reveals that different model families develop interaction awareness on different input contexts, and that this context-specificity is unrelated to task accuracy. This finding reframes interaction awareness from a monolithic capability (something a model either has or lacks) to a family of context-dependent behaviors shaped by training recipes.

The evidence is multi-layered. At the aggregate level (Figure 3), gpt-oss models produce 17-21% follow-ups on GPQA Diamond despite lower answer accuracy (61-67%) than Qwen3.5-397B-A17B (86.1% accuracy, 0.5% follow-up) or GLM-4.7 (66.4% accuracy, 1.5% follow-up). On GSM8K, the pattern reverses: Qwen3.5-397B produces more follow-ups (0.8%) than gpt-oss-120B (0.0%), though both are near zero. On IFBench, Qwen3.5-397B leads at 9.7%, while gpt-oss models are at 0.0-3.3%. The ranking of models by follow-up quality depends on the dataset, and the dataset at which a model performs best varies by family.

The perturbation experiments (§3.4) provide mechanistic evidence for why this happens. Truncation increases genuine follow-up for gpt-oss and GLM models on GSM8K (gpt-oss-120B: 0.0% → 24.2%; GLM-4.7: 1.0% → 55.0%; Table 1, Left) but has negligible effect on Qwen (Qwen3.5-27B: 0.0% → 0.0%). The explicit-question perturbation shows the converse pattern: gpt-oss and GLM change nearly all their user turns (97-99%) in response to the appended question, while Qwen models change only 6-40% (Table 1, Right). These asymmetries reveal that Qwen's failure is a failure to attend to the assistant-turn content—the model treats the user-role header as a conversation-reset signal and regenerates the prompt regardless of what the assistant said—while gpt-oss and GLM's failure is a failure to convert attention into appropriate user-role output. gpt-oss and GLM register the assistant turn's content, but their default continuation under the user role is degenerate (copying the assistant response or emitting planning text) unless a salient cue like truncation or an explicit question forces a different behavior.

This distinction—failure to attend versus failure to convert—is a conceptual contribution that goes beyond documenting low follow-up rates. It implies that interaction awareness is not one thing that post-training either instills or fails to instill. It is at least two things: the capacity to process the assistant turn as conversational content (present in gpt-oss and GLM, largely absent in Qwen on the tested datasets) and the capacity to produce user-role-appropriate output conditioned on that content (present in all families at some level, but suppressed at the mode in different ways). The failure mode taxonomy in Appendix B (Table 6) operationalizes this: Qwen's 78.5% prompt restatement rate, gpt-oss's 70.2% assistant-turn restatement rate, and GLM's 43.0% meta-planning rate are systematic, family-level signatures that reflect different underlying deficits.

The practical implication is that no single model family dominates interaction awareness across contexts. A practitioner choosing between Qwen3.5 and gpt-oss for a multi-turn deployment needs to know which family's context-specific follow-up profile matches the application domain. The paper does not provide a general recipe for making this choice—it measures the profiles on five specific benchmarks—but it establishes the principle that the choice matters and cannot be inferred from accuracy leaderboards.

Innovation 4: Positive Controls via Causal Perturbation as a Validation Methodology

The paper introduces a methodological move that is underused in LLM evaluation: causal perturbation of the input as a validation that the metric measures a real property of the model rather than a spurious correlation. The truncation and explicit-question experiments (§3.4) are not just additional measurements—they are diagnostic tests designed to rule out alternative explanations for the near-zero follow-up rates.

Consider the alternative explanations that these perturbations test. One could argue that near-zero follow-up rates reflect a chat-template artifact: the <|im_end|> token signals "conversation complete," so the model treats the subsequent user-role header as starting a new conversation. Under this explanation, the follow-up rate measures nothing about interaction awareness—it is just an artifact of the template convention. The truncation experiment falsifies this for gpt-oss and GLM models: if the template convention were the sole explanation, truncating the assistant response (which does not change the template) would not affect follow-up rates. But it does—sharply. The model is attending to the assistant-turn content and responding to its incompleteness, which requires processing the assistant turn as a conversational object, not just as a sequence-terminating boundary.

A second alternative explanation is that the model lacks the capacity for grounded follow-ups—the weights simply do not encode conversational anticipation in any form. The temperature sweep experiments (Figure 4) falsify this for Qwen models: at T=1.0, follow-up rates rise substantially, proving the capacity exists in the distribution. The explicit-question perturbation provides converging evidence: for gpt-oss models, appending a generic question converts a high fraction of user turns from degenerate to genuine (gpt-oss-120B: 1.3% → 25.7% on IFBench; Table 1, Right), showing that the capacity is present but requires a salient conversational cue to be expressed.

The methodological insight is that these perturbations serve as positive controls—they validate that the metric is sensitive to manipulations that should matter if the construct is real, while being insensitive to manipulations that should not matter if the construct is an artifact. This is a standard scientific practice (causal manipulation to establish construct validity) that is surprisingly rare in LLM evaluation, where metrics are typically validated against human judgment (as the paper also does with the judge validation) but rarely against causal interventions on the model's input. The paper demonstrates that both forms of validation are complementary: human agreement establishes that the judge is measuring something reliably; causal perturbation establishes what is being measured.

The asymmetry in perturbation responses across families is itself a substantive finding: Qwen models are largely insensitive to both perturbations, suggesting that their user-turn generation is driven primarily by the role-header transition rather than the assistant-turn content. This is consistent with the failure mode analysis (78.5% prompt restatement) and implies that Qwen's post-training recipe has trained the model to treat user-role headers as hard conversation resets—a design choice with consequences for multi-turn deployment that the perturbation methodology makes visible.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses five benchmark datasets in the self-generated setting (model produces both assistant response a and user turn u): GSM8K (Cobbe et al., 2021) with 1,319 math word problems; IFEval (Zhou et al., 2023) with 541 instruction-following examples; IFBench (Pyatkin et al., 2025) with 300 additional instruction-following examples; GPQA Main (Rein et al., 2024) with 448 graduate-level multiple-choice science questions; and GPQA Diamond with 198 harder questions from the same source. Two held-out conversational datasets are used only for post-training transfer evaluation: HealthBench (Arora et al., 2025) with 1,470 health-related conversation examples, and Coval (Hitzig et al., 2026) with 100 values-aware conversation examples. All datasets are evaluated in full—no subsampling is mentioned—and GSM8K and GPQA answers are extracted using pattern matching (final numeric answer for GSM8K, final letter for GPQA), with extraction failures counted as incorrect.
Base model(s). Eleven open-weight models from three families are evaluated: eight Qwen3.5 models spanning 0.8B, 2B, 4B, 9B, 27B, 35B-A3B (MoE with 3B active parameters), 122B-A10B (MoE), and 397B-A17B (MoE) (Qwen Team, 2026); two gpt-oss models at 20B and 120B (OpenAI, 2025); and GLM-4.7 (Zeng et al., 2025). The Qwen3.5 family is chosen to provide within-family scaling analysis from 0.8B to 397B active parameters, spanning both dense and mixture-of-experts architectures. The gpt-oss and GLM models serve as cross-family controls to test whether findings are Qwen-specific artifacts of training data composition and post-training recipes.
Metrics. The primary metric is the genuine-follow-up rate (%): the fraction of generated user turns u classified as σ = 1 (genuine) by the LLM judge gpt-5.4-mini. This is computed per dataset by aggregating the judge's binary decisions across all examples. A secondary metric is task accuracy on standard benchmarks (GSM8K numeric answer extraction, GPQA final-letter extraction, IFEval/IFBench constraint satisfaction), reported to demonstrate the dissociation between assistant-response correctness and interaction awareness. For the perturbation experiments, an additional metric % Changed is reported: the fraction of user turns whose text differs from the unperturbed generation, measuring whether the model attended to the perturbation regardless of whether the resulting text was judged genuine.
Baselines. The paper does not compare against alternative interaction-awareness metrics or prior probe designs—there is no existing standard for this measurement. The baseline is implicitly the deterministic (T=0) generation condition, which probes the model's modal user-turn continuation. Cross-family and cross-dataset comparisons serve as relative baselines: models are compared to each other at the same temperature on the same dataset. For the post-training experiments (§3.5), the untrained Qwen3.5-2B base model (62.9% GSM8K accuracy, 1.0–2.0% follow-up on verifiable benchmarks) serves as the baseline.
Generation budget / compute accounting. The paper measures compute implicitly through the number of generations: one full user-turn generation per example in the self-generated and held-out settings. Temperature sweeps use a single generation per example per temperature point, with the authors arguing that "since rates are averaged over hundreds of examples per dataset, variance from individual samples is mitigated" (Section 4, Limitations). There is no explicit FLOP counting or budget comparison because the probe is a measurement protocol, not a training method—the cost is one assistant generation plus one user generation per evaluated example. All generation uses vLLM on H200 GPUs with the verlai/verl:vllm017.latest Docker image. Assistant and user turns are both generated under the same temperature setting (matching T for a and u).
Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense because model weights are never trained in the main evaluation experiments—all models are evaluated off-the-shelf. The LLM judge's reliability is validated against blinded human annotation: two annotation packets (a hard-case packet constructed from nano-vs-gpt-5.4 disagreements plus judge-agree examples, and a natural-prevalence packet sampled proportionally from the full panel) were annotated by peer AI researchers blinded to model identity, dataset metadata, and judge labels. Pooled across both packets, gpt-5.4-mini achieved 90.5% binary genuine-follow-up agreement with human annotators (Cohen's κ = 0.726). Per-dataset genuine-follow-up rates are simple proportions with no confidence intervals reported, which is a limitation the paper does not discuss.

Main Quantitative Results

Cross-Family Dissociation: Task Accuracy vs. Follow-Up Quality

The paper's headline finding is that interaction awareness—as measured by genuine-follow-up rate under deterministic generation (T=0)—is decoupled from task accuracy across model families on verifiable benchmarks. Figure 3 (top row) reports task accuracy for five representative models on three datasets, while the bottom row reports the corresponding genuine-follow-up rates. Exact numbers across all five datasets are in Table 3 for Qwen3.5 and Table 5 for cross-family results.

On GSM8K, task accuracy scales cleanly with model size within Qwen3.5: 41.6% (0.8B), 89.5% (9B), 95.8% (27B), 96.8% (397B-A17B). The cross-family models are comparably strong: gpt-oss-20B at 93.9%, gpt-oss-120B at 95.8%, GLM-4.7 at 95.3%. Yet genuine-follow-up rates under T=0 remain near zero across all models: Qwen3.5-9B at 0.5%, Qwen3.5-27B at 0.0%, Qwen3.5-397B at 0.8%, gpt-oss-20B at 1.4%, gpt-oss-120B at 0.0%, GLM-4.7 at 1.2%. The near-perfect accuracy of Qwen3.5-397B (96.8%) coexists with a 0.8% follow-up rate—the model almost always answers correctly, and almost never produces a grounded conversational continuation afterward.

On GPQA Diamond, the pattern inverts in a revealing way: gpt-oss models, which underperform on task accuracy (gpt-oss-20B at 61.1%, gpt-oss-120B at 67.2%) relative to Qwen3.5-397B (86.1%), substantially outperform on follow-up quality. gpt-oss-20B reaches 20.7% genuine follow-ups and gpt-oss-120B reaches 16.7%, while Qwen3.5-397B achieves only 0.5% despite 18.9 percentage points higher accuracy. Qwen3.5-9B is a notable within-family outlier at 13.1% follow-up on GPQA Diamond—nearly 26× the rate of the 397B model—despite lower accuracy (78.6% vs. 86.1%). GLM-4.7 (66.4% accuracy) produces only 1.5% follow-ups.

On IFBench, the pattern shifts again: Qwen3.5-397B leads follow-up rates at 9.7%, followed by Qwen3.5-9B at 2.0% and gpt-oss-120B at 3.3%. No model family dominates across all datasets. The key structural finding is that task accuracy rankings and follow-up quality rankings are uncorrelated: the best model for answering GPQA questions (Qwen3.5-397B) is among the worst at generating user follow-ups after answering them, while gpt-oss models show the reverse pattern. IFEval (Table 5) mirrors this: Qwen3.5-397B leads at 10.5%, while gpt-oss-120B manages 3.7%.

The paper's claim that "interaction awareness is decoupled from task accuracy" is supported by these cross-family and cross-dataset dissociations, but the evidence is strictly correlational: it shows that high accuracy does not imply high follow-up, not that the two are causally independent. The post-training experiments (§3.5) provide stronger evidence for causal independence by showing that SFT training raises follow-up rates while reducing accuracy, demonstrating they can move in opposite directions under an intervention.

Temperature Scaling: Interaction Awareness Is Latent in the Distribution

The cross-family results use deterministic generation (T=0), which probes only the mode of the model's output distribution. The paper tests whether near-zero follow-up rates reflect genuine absence of the capability or whether interaction-aware continuations exist in the distribution but are suppressed at the mode. Figure 1 (right panel) presents the temperature sweep (T ∈ {0, 0.3, 0.7, 1.0}) for six representative models on three datasets. Figure 4 and Table 4 provide the complete sweep for all eight Qwen3.5 models.

The results reveal a clear separation between model families. Qwen3.5 and GLM show strong temperature scaling: Qwen3.5-27B rises from 0.0% at T=0 to 0.1% at T=0.3, 4.5% at T=0.7, and 22.0% at T=1.0 on GSM8K (Table 4). On GPQA Diamond, the same model scales from 1.5% (T=0) to 7.1% (T=0.3), 17.2% (T=0.7), and 35.9% (T=1.0). On IFBench, it rises from 1.0% to 30.7%. GLM-4.7 shows a similar trajectory: from 1.2% to 15.2% on GSM8K, and from 1.5% to 35.4% on GPQA Diamond at T=1.0 (Figure 1, right panel). These increases are smooth and monotonic with temperature across all three datasets.

gpt-oss models show mixed or absent temperature response. On GSM8K, gpt-oss-120B remains at 0.1% even at T=1.0—essentially zero at all temperatures. On IFBench, both gpt-oss models stay below 4% across all temperatures. On GPQA Diamond, however, gpt-oss-20B scales from 20.7% (T=0) to 47.0% (T=1.0), the highest genuine-follow-up rate reported in the paper for any model-dataset-temperature combination. The asymmetry is instructive: gpt-oss models possess latent interaction awareness for GPQA science questions but not for GSM8K math problems or IFBench instruction-following tasks.

The temperature-response pattern varies substantially even within the Qwen3.5 family (Figure 4, Table 4). On GSM8K at T=1.0, the 27B model leads at 22.0%, followed by 397B-A17B at 18.0% and 0.8B at 13.7%, while 9B (7.7%) and 35B-A3B (5.5%) lag. On GPQA Diamond, 27B again leads at 35.9%, but 0.8B shows a non-monotonic pattern (0.5% at T=0, rising to 32.3% at T=0.7, then falling to 16.2% at T=1.0), while 397B-A17B reaches only 12.1%. On IFBench, 397B-A17B dominates at 43.7%, with 9B at 36.0% and 27B at 30.7%. Model size does not predict follow-up rate at any temperature: the ranking of models changes depending on the dataset, and mid-sized models (9B, 27B) frequently match or exceed the largest model (397B-A17B).

The MoE models consistently underperform their dense counterparts. At T=1.0 on GSM8K, the dense 27B reaches 22.0% while the MoE 35B-A3B (with 3B active parameters) reaches only 5.5%, the MoE 122B-A10B reaches 3.3%, and the MoE 397B-A17B reaches 18.0%—all below the dense 27B despite some having more active parameters. The same pattern holds on GPQA Diamond (27B at 35.9% vs. 35B-A3B at 2.5%, 122B-A10B at 3.5%, 397B-A17B at 12.1%) and IFBench (27B at 30.7% vs. 35B-A3B at 13.3%, 122B-A10B at 25.3%, 397B-A17B at 43.7%—the one exception where the largest MoE barely exceeds the largest dense model).

The interpretation these results support is nuanced. For Qwen3.5 models, interaction awareness is latent in the distribution but suppressed at the mode: the models assign non-trivial probability mass to grounded follow-up continuations, but the training objective has placed those continuations in the distribution's tail, not at its peak. The smooth temperature scaling suggests the capability is genuinely learned—the probability mass allocated to interaction-aware continuations increases systematically as temperature broadens the sampling distribution. For gpt-oss on GSM8K and IFBench, the capability appears genuinely absent: even at T=1.0, where sampling draws from the full distribution, grounded follow-ups almost never appear, suggesting the models have not learned this behavior for these contexts at all. For gpt-oss on GPQA Diamond, the capability is latent and strong, possibly reflecting domain-specific conversational patterns in the training data that transferred to science Q&A contexts.

Within-Family Scaling Analysis: Model Size Does Not Predict Interaction Awareness

The Qwen3.5 family provides a clean test of whether interaction awareness scales with model size, since all eight models share the same training recipe and architecture (with the MoE variants providing a secondary test of architecture vs. effective parameter count). Table 3 reports deterministic (T=0) results across all five benchmark datasets. Table 4 and Figure 4 report the full temperature sweep.

At T=0, the answer is unambiguous: model size does not predict follow-up rate. On GSM8K, five of eight models produce exactly 0.0% genuine follow-ups; the non-zero models are Qwen3.5-0.8B at 1.0%, Qwen3.5-9B at 0.5%, and Qwen3.5-397B-A17B at 0.8%. The scaling of task accuracy from 41.6% to 96.8% is almost perfectly monotonic; the scaling of follow-up rate is flat near zero with no size-dependent trend. On GPQA Diamond, Qwen3.5-9B at 13.1% stands as a dramatic outlier—the 2B, 4B, 35B-A3B, and 122B-A10B models are all at 0.0%, and the 27B and 397B models are at 1.5% and 0.5% respectively. The 9B model produces more than 26× the follow-ups of the 397B model despite being 44× smaller in active parameters. On IFBench, the 397B-A17B leads at 9.7%, followed by the 0.8B at 3.0% and 9B at 2.0%, with intermediate models clustered around 0.3–1.0%. IFEval shows a similar pattern: 397B-A17B at 10.5%, 9B at 5.4%, all others below 1.2%.

The temperature sweep (Figure 4) reveals that this non-scaling persists even when interaction awareness is surfaced through sampling. At T=1.0 on IFBench, the 397B-A17B model reaches 43.7%, but the 9B model is at 36.0%, the 27B at 30.7%, and the 0.8B at 27.7%—the gap between the smallest and largest model is only 16 percentage points, far smaller than the 30-point gap in task accuracy (21.6% vs. 51.6% by the paper's reported IFBench accuracy numbers). On GSM8K, the 27B (22.0%) and 0.8B (13.7%) bracket the 397B (18.0%). On GPQA Diamond, the 27B (35.9%) substantially exceeds the 397B (12.1%).

The qualitative example in Figure 5 illustrates the non-monotonicity concretely. On the same GPQA Diamond chemistry question about the Corey-Chaykovsky reagent, both Qwen3.5-9B and Qwen3.5-27B answer correctly (Answer: D), but their generated user turns are qualitatively different. Qwen3.5-9B produces a critical engagement with the assistant's reasoning—"In the previous turn, you identified the reagent... However, the reaction conditions specified 'in DMSO at elevated temperature'. Let's re-evaluate the reagent name...". Qwen3.5-27B, a 3× larger model, restates the original prompt verbatim. The model with lower task accuracy produces a more interaction-aware user turn.

The MoE architectures consistently underperform their dense counterparts of comparable or smaller active parameter counts. At T=1.0 across all three datasets: the 35B-A3B MoE trails the 2B dense model on GSM8K (5.5% vs. 4.7%), though both are low; on GPQA Diamond, the 35B-A3B (2.5%) trails the 0.8B dense model (16.2%); the 122B-A10B MoE (25.3%) trails the 9B dense (36.0%) on IFBench. The MoE degradation is particularly stark on GPQA Diamond, where every dense model from 0.8B upward exceeds every MoE model at most temperatures. The paper does not investigate the mechanism behind this MoE deficit, but it is a consistent empirical signal that interaction awareness may be sensitive to how parameters are routed in sparse architectures.

Controlled Perturbations: Validating That the Metric Measures Attention to Assistant Content

Table 1 summarizes the perturbation experiments. The truncation perturbation (Left panel) test whether genuinely-follow-up rates respond to degrading the assistant response by removing the final max(25, ⌈0.25 × n_tokens⌉) tokens, creating a visibly incomplete answer. On GSM8K, where unperturbed follow-up is near zero for all models, truncation produces sharp increases for gpt-oss and GLM models: GLM-4.7 jumps from 1.0% to 55.0%, gpt-oss-120B from 0.0% to 24.2%, gpt-oss-20B from 1.2% to 5.8%. Qwen models are flat: Qwen3.5-27B remains at 0.0%, Qwen3.5-397B rises modestly from 0.8% to 4.9%. On GPQA Diamond, the pattern holds: GLM-4.7 rises from 2.0% to 39.4%, gpt-oss-120B from 20.7% to 65.7% (the highest genuine-follow-up rate in the paper), gpt-oss-20B from 1.0% to 18.7%. Qwen3.5-27B and Qwen3.5-397B show negligible change (1.0% → 1.0% and 2.0% → 2.0%).

The explicit-question perturbation (Right panel) appends a generic conversational question (e.g., "What do you think?", "Any questions?") to the assistant response. On IFBench, gpt-oss-120B jumps from 1.3% to 25.7% with 99.0% of user turns changing text, gpt-oss-20B from 1.0% to 6.0% with 97.3% changed, and GLM-4.7 changes 99.0% of turns but genuine-follow-up barely increases (4.7% → 5.0%). Qwen models again show low sensitivity: Qwen3.5-27B changes only 6.3% of user turns (0.0% → 6.3% genuine), and Qwen3.5-397B's genuine-follow-up rate decreases from 7.7% to 4.7% with 40.3% of turns changed, suggesting the appended question disrupted its existing (already modest) follow-up behavior.

These results serve as positive controls that validate the metric: if genuine-follow-up rate measured something unrelated to assistant-turn content, perturbing the assistant response would not systematically change it. The fact that it does—but only for models that have been independently shown to attend to assistant content (gpt-oss and GLM, evidenced by their assistant-turn-restatement failure mode in Table 6)—confirms that the metric captures a real property of the model's processing of conversational context. The family-level asymmetry (Qwen insensitive, gpt-oss/GLM sensitive) is consistent with the failure mode taxonomy: Qwen's prompt-restatement behavior is driven by the role-header transition, not the assistant content, so manipulating the assistant content has little effect. gpt-oss and GLM attend to the assistant turn but default to degenerate continuations (assistant-turn copying, meta-planning); the perturbations provide salient cues that redirect this attention toward grounded output.

Post-Training: Interaction Awareness Can Be Increased

Table 2 reports results from post-training Qwen3.5-2B with collaboration-oriented training (the CollabLLM recipe, Wu et al., 2025) using 200 MATH level-5 problems expanded into ~5K multi-turn examples. Two variants are evaluated: supervised fine-tuning (SFT) with masked loss on assistant turns, and online reinforcement learning (RL) with PPO using multi-turn rollouts and conversational rewards. Neither variant explicitly trains for user-turn generation; both target assistant response quality only. Evaluation is on three verifiable benchmarks (IFBench, GPQA Diamond) and two held-out conversational datasets (HealthBench, Coval), none of which overlap with the MATH training data.

SFT results. The SFT variant produces dramatic follow-up gains at the cost of task accuracy. Genuine-follow-up rates on IFBench rise from 1.0% (base) to 48.0%, on GPQA Diamond from 2.0% to 46.0%. On the held-out datasets, HealthBench rises from 36.7% to 54.4%, and Coval from 19.4% to 45.2%. GSM8K accuracy drops from 62.9% to 40.3%—a 22.6 percentage-point decrease attributed by the paper to "forgetting/overfitting to ≈5k multi-turn examples." The SFT variant demonstrates that strong interaction awareness can be induced through multi-turn training, but the accuracy-for-follow-up trade-off suggests the small training corpus (5K examples) caused catastrophic forgetting of general math reasoning capabilities.

RL results. The RL variant avoids this trade-off: GSM8K accuracy improves slightly from 62.9% to 67.4%, while follow-up rates rise moderately across all datasets: IFBench from 1.0% to 10.0%, GPQA Diamond from 2.0% to 9.1%, HealthBench from 36.7% to 46.5%, Coval from 19.4% to 29.0%. The gains are smaller than SFT (10% vs. 48% on IFBench) but do not degrade task accuracy. The paper attributes this to PPO's KL constraint, which prevents the policy from diverging too far from the base model's distribution, preserving core capabilities while allowing adaptation to the multi-turn conversational distribution where reward signal is present.

The key structural finding is that interaction awareness increases as a side effect of training that targets assistant-response quality in multi-turn conversations, without ever explicitly optimizing for user-turn generation. This validates the paper's central claim that the genuine-follow-up metric measures something trainable that is correlated with multi-turn collaboration quality, and that current models' near-zero follow-up rates are a consequence of training data and objectives, not an inherent limitation of the architecture.

Failure Mode Taxonomy: Systematic, Family-Specific Signatures

Table 6 reports the distribution of primary labels across all five self-generated benchmark datasets (5,612 generations for Qwen3.5 and gpt-oss, 2,806 for GLM-4.7), pooled across models and datasets within each family. The three families exhibit qualitatively distinct failure profiles that are systematic rather than random.

Qwen3.5 is dominated by previous turn restate at 78.5%: nearly four out of five user turns are verbatim copies of the original prompt. The second most common category is degenerate short at 16.6%—particularly for large models like Qwen3.5-397B, which frequently emits single tokens or empty strings. Plausible followup accounts for only 2.1% of generations. Meta planning is nearly absent at 0.4%, and assistant turn restate at 0.0%.

gpt-oss shows a fundamentally different profile: assistant turn restate dominates at 70.2%, followed by previous turn restate at 11.9%, meta planning at 6.8%, and degenerate short at 5.1%. Plausible followup is 4.2%. The dominant failure is not ignoring the conversation (as with Qwen) but continuing in the assistant voice—copying or paraphrasing the assistant response under the user role, often including internal chain-of-thought verification text and channel-delimited scaffolding like <|channel|>analysis<|message|>.

GLM-4.7 is dominated by meta planning at 43.0%, with assistant turn restate at 23.0%, previous turn restate at 20.2%, and malformed artifact at 10.8%. Plausible followup is 1.9%. The high meta-planning rate indicates that GLM-4.7 attends to the role-header change (unlike Qwen's prompt restatement) but fails to convert that attention into a user-appropriate response, instead generating assistant-style reasoning text (e.g., "Here's a thinking process that leads to the suggested answer...").

These distinct profiles support two claims. First, interaction awareness failure is not a single phenomenon—it is a family of behaviors shaped by training recipes, with each family defaulting to a different degenerate continuation pattern. Second, the failure is systematic: each family's dominant failure mode accounts for 43–78% of all generations, meaning these are not random errors but consistent behavioral defaults that reflect how the training data and optimization objectives have shaped the model's weights for the user-role continuation task.

Ablation Studies and Robustness Checks

Judge model selection. The paper conducted a same-family judge comparison between gpt-5.4-nano (initial operational judge), gpt-5.4-mini, and full gpt-5.4. On the full evaluation panel rerun, nano and mini reached 81.7% genuine-follow-up agreement with κ = 0.480, with the overall genuine-follow-up rate dropping from 26.6% (nano) to 18.5% (mini). On a balanced hard-case slice (nano-vs-gpt-5.4 disagreements plus judge-agree examples), nano agreed with mini at only 72.8% (κ = 0.373) and with full gpt-5.4 at 73.2% (κ = 0.366), while mini and gpt-5.4 agreed at 90.9% (κ = 0.722). The systematic direction of disagreement was that "nano is looser on the hard slice, marking 37.9% of examples as genuine follow-up, compared with 22.4% for mini and 18.1% for gpt-5.4" (Appendix D.1). This justifies the switch to gpt-5.4-mini for paper-facing results.
Human annotation validation. Two blinded annotation packets were distributed to peer AI researchers: a hard-case packet (nano-vs-gpt-5.4 disagreements plus judge-agree genuine and nongenuine examples) and a natural-prevalence packet (proportional sample from the full panel). Annotators were blinded to model identity, dataset metadata, reference user turn, and all judge labels. Pooled across both packets, gpt-5.4-mini achieved 90.5% binary genuine-follow-up agreement with human annotators (κ = 0.726). Primary-label agreement was lower at 58.5% (κ = 0.481), indicating that the judge's fine-grained label assignment is less reliable than its binary judgment—the paper appropriately restricts its main claims to the binary level. The nano judge achieved only 73.0% pooled agreement (κ = 0.311), further supporting the switch to mini.
Held-out vs. self-generated settings. The paper evaluates follow-up rates in both the self-generated setting (a produced by Mθ) and the held-out setting (a from real conversation data). The held-out evaluation on HealthBench and Coval is reported only for post-training experiments (Table 2), not for baseline models. This limits the ability to assess whether low benchmark follow-up rates are specific to self-generated assistant responses or generalize to human-generated assistant responses. The paper acknowledges this limitation: the held-out evaluation "partially addresses" whether the metric measures recovery of human continuations, but both datasets are English-only conversational domains (Section 4, Limitations).
Within-family architecture comparison: dense vs. MoE. The Qwen3.5 family sweep includes both dense models (0.8B, 2B, 4B, 9B, 27B) and MoE models (35B-A3B, 122B-A10B, 397B-A17B). The consistent underperformance of MoE variants relative to dense counterparts of comparable or smaller active parameter counts (Figure 4, Table 4) serves as an implicit ablation of architecture type. On GSM8K at T=1.0, the dense 27B reaches 22.0% while MoE 35B-A3B reaches 5.5%, MoE 122B-A10B reaches 3.3%, and MoE 397B-A17B reaches 18.0%. The paper does not investigate or hypothesize a mechanism for this MoE deficit, but it is a robust empirical signal across all three datasets.
Single-generation temperature sweeps. The temperature sweep experiments use a single generation per example per temperature, rather than multiple samples averaged. The paper argues this is acceptable because "rates are averaged over hundreds of examples per dataset" (Section 4). However, the variance at the per-example level is high—a single sample at T=1.0 either lands on a grounded follow-up or does not, with no information about the proportion of probability mass allocated to such continuations. A more rigorous design would sample multiple user turns per example and report the pass@k rate, which would directly measure the probability mass allocated to interaction-aware continuations rather than the single-sample hit rate. The paper does not include this experiment.
Chat template modifications. The paper's generation pipeline requires modifying standard chat templates to permit user-role continuation after an assistant turn, since <|im_end|> tokens are typically treated as conversation boundaries. The paper does not ablate whether the specific template modification (as opposed to the role-switching concept) affects results. A potential confound is that the modification itself—appending a user-role header after an assistant's <|im_end|>—places the model in an out-of-distribution input format, and the degenerate continuations (prompt restatement, assistant-turn restatement) may be partially caused by this distribution shift rather than by absent interaction awareness. The perturbation experiments partially address this by showing that manipulating the assistant turn's content changes follow-up behavior for some models, indicating that the model is processing that content rather than just reacting to the template irregularity.
Training data composition for post-training. The post-training experiments use only 200 MATH level-5 problems expanded into ~5K multi-turn examples. This is a narrow training distribution—hard math problems in a multi-turn format—and the transfer to IFBench, GPQA Diamond, HealthBench, and Coval is therefore a test of generalization across domains. The SFT variant's catastrophic forgetting of GSM8K accuracy (62.9% → 40.3%) suggests overfitting to the small training corpus, while the RL variant's preserved accuracy suggests the KL constraint mitigated overfitting. The paper does not ablate training data size, problem difficulty, or domain, leaving open whether larger-scale multi-turn training across diverse domains would produce more robust interaction awareness gains.
The paper does not report the temperature sweep for the perturbation experiments. All perturbation results are at T=0. This is a notable omission: if interaction awareness is latent at T=0 but surfaced at higher temperatures, the perturbation effects might be amplified or qualitatively different at T=1.0. It is possible that Qwen models, which are insensitive to perturbations at T=0, would show sensitivity at higher temperatures where their attention to assistant content is less masked by prompt-restatement behavior.

Critical Assessment

Claim 1: "Interaction awareness is decoupled from task accuracy across model families."

What the experiments demonstrate. Figure 3 and Tables 3–5 show that models with high task accuracy on a dataset can have near-zero genuine-follow-up rates on that same dataset under deterministic generation (Qwen3.5-397B-A17B: 96.8% GSM8K accuracy, 0.8% follow-up). Conversely, models with lower accuracy can have higher follow-up rates (gpt-oss-120B: 67.2% GPQA Diamond accuracy, 16.7% follow-up vs. Qwen3.5-397B: 86.1% accuracy, 0.5% follow-up). This is clear evidence that high accuracy does not imply high follow-up rates.

What the experiments do not demonstrate. The claim of "decoupling" could be interpreted as implying that interaction awareness and task accuracy are independent dimensions that can be optimized separately. The experiments do not establish this. They show that at a single point in model space (off-the-shelf checkpoints), the ranking on accuracy does not predict the ranking on follow-up. They do not show that the two dimensions can be independently varied through interventions—the post-training experiments demonstrate the opposite: the SFT intervention simultaneously raised follow-up rates and lowered accuracy, showing the two are coupled under this particular training recipe. The RL intervention preserved accuracy while raising follow-up, which provides partial evidence for separability, but from a single training recipe and a single model size (2B). The claim as stated is supported for the cross-sectional comparison but the causal independence implied by "decoupled" is not fully established.

What would strengthen the claim. A systematic sweep of post-training recipes—varying the proportion of multi-turn vs. single-turn data, the conversational reward structure, the KL penalty strength—showing that follow-up rates and accuracy trace out a frontier where different recipes achieve different points on the accuracy-follow-up trade-off curve. The current post-training experiments (two data points: SFT and RL) are suggestive but insufficient to map this frontier.

Claim 2: "Interaction awareness is latent in the generation distribution, surfaced with higher temperature sampling."

What the experiments demonstrate. Figure 4 and Table 4 show smooth, monotonic increases in genuine-follow-up rates as temperature increases for Qwen3.5 models. Qwen3.5-27B goes from 0.0% to 22.0% on GSM8K as temperature rises from 0 to 1.0. This is strong evidence that the model's generation distribution contains interaction-aware continuations that are not selected under greedy decoding—the capability is meaningfully latent.

Limitations of the evidence. The temperature sweep uses a single generation per example per temperature point. The reported rate at T=1.0 (e.g., 22.0% for Qwen3.5-27B on GSM8K) is the fraction of examples where one sample happened to be a genuine follow-up. This is not the same as the probability mass allocated to interaction-aware continuations. A model could assign low but non-zero probability to follow-ups on many examples and high probability on few; a single-sample measurement conflates these. A pass@k measurement with k > 1 would provide a more direct estimate of the probability mass allocated to interaction-aware continuations. The current evidence shows the capability exists in the distribution (latent) but does not quantify how much of the distribution it occupies.

The gpt-oss asymmetry is under-explored. gpt-oss models show strong temperature scaling on GPQA Diamond (gpt-oss-20B: 20.7% → 47.0%) but essentially no scaling on GSM8K (gpt-oss-120B: 0.0% → 0.1%) or IFBench (gpt-oss-120B: 3.3% → 3.7% at T=0.0 → 1.0 from Figure 1, though Table 4 shows it stays below 4%). The paper notes this as a "clear separation between model families" but does not investigate why domain-specificity of latent awareness exists. Is it about the format of the assistant response (math step-by-step vs. prose science reasoning)? The presence of specific conversational patterns in training data? The paper documents the phenomenon without explaining it, which is reasonable for a measurement paper but limits the practical guidance for practitioners.

Claim 3: "Interaction awareness does not scale with model size even within a single family."

What the experiments demonstrate. This claim is among the best-supported in the paper. Table 3 shows that at T=0, follow-up rates are near zero for most Qwen3.5 models regardless of size—the scaling curve is flat, not increasing. Figure 4 shows that even at higher temperatures, the largest model does not dominate: on GSM8K at T=1.0, the 27B model (22.0%) exceeds the 397B (18.0%); on GPQA Diamond, the 27B (35.9%) more than doubles the 397B (12.1%); on IFBench, the 397B leads at 43.7% but the 0.8B model reaches 27.7%, and the 9B reaches 36.0%—far from monotonic scaling. The qualitative example in Figure 5 makes this vivid: Qwen3.5-9B produces a sophisticated critique while Qwen3.5-27B restates the prompt on the same question.

What would strengthen the claim. The paper reports accuracy at T=0 only (Table 3) but follow-up rates across temperatures. A parallel temperature sweep for accuracy would rule out the possibility that the non-scaling of follow-up rates is an artifact of temperature differentially affecting models of different sizes. If, for instance, the 27B model's accuracy degrades more slowly with temperature than the 397B model's, the follow-up rate scaling pattern might partially reflect accuracy scaling patterns in the sampled regime. The paper does not report temperature-dependent accuracy.

The MoE vs. dense finding is robust but unexplained. The consistent MoE deficit across all three datasets and all temperatures is striking: 35B-A3B and 122B-A10B rarely exceed smaller dense models despite having comparable or greater active parameters. This is a genuine empirical contribution—it suggests that interaction awareness is sensitive to architectural choices in ways that task accuracy scaling is not—but the paper does not investigate the mechanism. Is it that MoE routing disrupts the formation of conversational representations? That the MoE models were post-trained with a different data mixture? That the effective parameter count metric does not capture the relevant capacity? Without ablation or analysis, the MoE deficit remains an intriguing observation rather than an actionable finding.

Claim 4: "Controlled perturbations validate that the proposed metric measures a real property of the model."

What the experiments demonstrate. The perturbation experiments (Table 1) are well-designed positive controls. The truncation perturbation produces large increases in genuinely-follow-up rates for gpt-oss and GLM models (GLM-4.7: 1.0% → 55.0% on GSM8K; gpt-oss-120B: 20.7% → 65.7% on GPQA Diamond), confirming that these models attend to assistant-turn content and that the metric captures this attention. The explicit-question perturbation shows that gpt-oss models respond to a conversational cue by changing their user turns (99.0% changed for gpt-oss-120B on IFBench) and increasing genuine-follow-up rates (1.3% → 25.7%). In both cases, the perturbation causes a change that is consistent with the model processing assistant content.

What the experiments do not demonstrate. The perturbations validate the metric only for models that already show some sensitivity to assistant content. For Qwen models, which are insensitive to both perturbations (Qwen3.5-27B: 0.0% → 0.0% on GSM8K truncation, 6.3% changed on IFBench explicit question), the null result is consistent with two interpretations: (1) the Qwen models genuinely do not attend to assistant content, or (2) the perturbations were not strong enough to overcome Qwen's dominant prompt-restatement behavior, and a different perturbation (e.g., a much larger truncation, a more directive explicit question) would have elicited a response. The design does not distinguish these interpretations because the perturbation parameters (25% truncation, generic question pool) were not calibrated per family. A dose-response curve—varying the fraction of tokens truncated or the directiveness of the appended question—would provide stronger evidence that Qwen's insensitivity is fundamental rather than a failure to find the right perturbation strength.

The perturbation results are at T=0 only. This is a significant gap. The temperature sweep shows that interaction awareness is latent in Qwen models but surfaced at higher temperatures. The perturbation experiments at T=0 show no response from Qwen models. If the perturbations were run at T=1.0, where Qwen models produce non-trivial follow-up rates, the effect might be substantially larger or qualitatively different. The paper does not report this, leaving open the question of whether Qwen's perturbation insensitivity is a fundamental property or specific to the T=0 regime where all behavior is compressed to the mode.

Claim 5: "Post-training can increase interaction awareness."

What the experiments demonstrate. Table 2 shows that both SFT and RL training on multi-turn collaboration data raise genuine-follow-up rates on all five evaluation datasets, with SFT producing 46-48% on verifiable benchmarks and RL producing 9-10% while preserving task accuracy. The training data (200 MATH problems) has no overlap with the evaluation datasets, so the gains represent transfer. This is the paper's strongest causal evidence: an intervention that does not explicitly target user-turn generation nonetheless moves the follow-up metric in a positive direction, validating that the metric captures something correlated with multi-turn conversational training.

Limitations that constrain the claim. The post-training experiments are on a single model (Qwen3.5-2B) with a single training recipe (the CollabLLM approach, using 200 hard math problems expanded to ~5K multi-turn examples with ~5 turns each). The paper does not establish that the gains would generalize to other model families, other model sizes, other training data compositions, or other multi-turn training recipes. The SFT variant's catastrophic accuracy loss (62.9% → 40.3% on GSM8K) suggests the training was not well-regularized for this model size and data quantity—"forgetting/overfitting" is a diagnosis, not an explanation, and the paper does not investigate whether larger training corpora or different learning rates would mitigate the trade-off.

The held-out results are stronger than the benchmark results for the base model. On HealthBench and Coval (Table 2), the base Qwen3.5-2B model already achieves 36.7% and 19.4% follow-up rates at T=0—substantially higher than the ~1% on verifiable benchmarks. This suggests that conversation-domain data (health conversations, values discussions) contains richer multi-turn structure that elicits follow-up behavior even without post-training. The post-training gains on these datasets (to 54.4% and 45.2% for SFT) represent moderate absolute improvements (~18 and ~26 percentage points) rather than the dramatic relative improvements on benchmarks (from ~1% to ~47%). The paper does not discuss this baseline difference, which is important for understanding the conditions under which interaction awareness is already present versus needs to be trained.

Overall Strengths

The measurement protocol is well-validated. The judge selection process (comparing nano, mini, and full gpt-5.4), the human annotation study (blinded, with both hard-case and natural-prevalence packets), and the perturbation experiments (positive controls demonstrating causal sensitivity) together provide unusually thorough validation of a new metric for an LLM evaluation paper. The Cohen's κ of 0.726 on the binary genuine-follow-up decision is a credible level of human-judge agreement.

The family-level failure mode analysis is a genuine contribution. Table 6 and the accompanying qualitative examples (Appendix C) reveal that interaction awareness failures are systematic and family-specific—Qwen restates the prompt (78.5%), gpt-oss copies the assistant response (70.2%), GLM emits planning text (43.0%). These are not random errors but behavioral signatures that reflect training data and objective choices. This taxonomy is practically useful: a practitioner experiencing echo failures in multi-agent deployments (Shekkizhar et al., 2025) can now diagnose whether the underlying deficit is attention to the assistant turn (Qwen-style) or conversion of attention to appropriate output (gpt-oss/GLM-style).

The temperature sweep reveals a non-obvious property of current models. The smooth scaling of follow-up rates with temperature for Qwen models (Figure 4) demonstrates that maximum-likelihood training on predominantly single-turn data has suppressed, but not eliminated, interaction-aware behavior. This is a structural finding about how post-training objectives interact with capabilities—it implies that many capabilities may be "latent at the mode" and only detectable through sampling, which has implications for evaluation methodology beyond interaction awareness.

Overall Weaknesses

The benchmark evaluation is at T=0 and does not include a pass@k measurement at higher temperatures. The paper's core diagnostic—interaction awareness is latent but surfaced at higher temperatures—is based on single-sample generation. A pass@k measurement (sample k user turns per example, count the fraction of examples where at least one is genuine) would directly estimate the probability mass allocated to interaction-aware continuations and provide a more stable metric. The single-sample approach conflates probability mass with sampling luck, particularly at T=1.0 where the distribution is broad.

The perturbation experiments are at T=0 only and not calibrated per family. Running perturbations only at T=0 means they test whether the model's modal user continuation responds to assistant content changes, not whether the model's full distribution responds. For Qwen models, whose modal continuation is prompt restatement regardless of assistant content, the null perturbation result is expected even if the distribution contains assistant-sensitive continuations at lower probability. A temperature sweep over perturbations would provide stronger evidence about whether the insensitivity is a fundamental property or specific to the mode.

The post-training experiments are preliminary (single model, single recipe, small data). While the paper is appropriately cautious about the post-training results—calling them "preliminary" in Section 4—the conclusion that "interaction awareness can be increased through post-training" implicitly generalizes beyond the tested conditions. The SFT accuracy collapse (62.9% → 40.3%) raises questions about whether the training recipe is practical at scale, and the paper does not explore whether this trade-off can be mitigated through larger training corpora, different learning rates, or regularization.

The paper does not characterize what a "good" follow-up rate would be. All genuine-follow-up rates are reported in absolute terms, but there is no human baseline—what fraction of human-generated user turns would the LLM judge classify as genuine? Without a human ceiling, a follow-up rate of 47% (gpt-oss-20B on GPQA Diamond at T=1.0) or 48% (SFT model on IFBench) is difficult to interpret: is this approaching human-level conversational grounding, or is it still far below? The held-out datasets (HealthBench, Coval) contain human user turns but the paper does not report human follow-up rates as a comparison point.

Dataset sizes are small for some analyses. The per-difficulty-bin analysis is not applicable here (the paper does not bin by difficulty), but the perturbation experiments on GPQA Diamond (198 examples) and IFBench (300 examples) have relatively small sample sizes for the percentage-change metrics. A change from 1.0% to 18.7% on GPQA Diamond for gpt-oss-20B truncation represents roughly 2 → 37 examples. The paper does not report confidence intervals, making it difficult to assess whether cross-model differences (e.g., Qwen3.5-397B: 0.8% → 4.9% vs. Qwen3.5-27B: 0.0% → 0.0% on GSM8K truncation) are statistically reliable.

The gpt-oss channel-delimited scaffolding is not analyzed as a potential confound. The gpt-oss models' assistant responses include internal verification text delimited by <|channel|>analysis<|message|> tokens (visible in the qualitative examples in Appendix C). This scaffolding is not standard chat-template content and may interact with the user-turn generation mechanism in ways the paper does not investigate. If the model was trained to associate <|channel|> tokens with internal reasoning, their presence in the conversation context when the user-role header is appended might bias the model toward continuing in the assistant voice—which is precisely the dominant gpt-oss failure mode (70.2% assistant-turn restatement). The paper's perturbation experiments show that gpt-oss models can produce grounded follow-ups when given the right cues, but the role of the scaffolding tokens in suppressing that behavior at T=0 is unexplored.

The metric's practical utility is not demonstrated. The paper establishes that the genuine-follow-up rate measures something real and dissociable from accuracy, but does not show that it predicts downstream performance on tasks that require interaction awareness—multi-agent collaboration quality, self-play training data quality, user satisfaction in conversational deployments. The post-training experiments provide indirect evidence (training for multi-turn collaboration raises follow-up rates) but do not establish a predictive relationship: does a model with higher follow-up rate actually perform better as a conversational partner or a self-play training signal generator? Without this, the metric remains a diagnostic of unclear practical significance.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Metric

The assumption or constraint. The paper's primary metric—genuine-follow-up rate—is measured under a protocol where the assistant response a is already generated before the user turn u is produced. The cost of generating a (one assistant generation per evaluated example) is part of the probe by design, but the paper's central claim—that interaction awareness is invisible to standard benchmarks—implicitly treats the probe as a measurement protocol whose cost is separable from the model's deployment cost. The paper does not account for the LLM judge's inference cost in any compute budget, does not compare the total FLOPs of the probe against the cost of standard accuracy evaluation, and does not discuss the practical overhead of running a second, stronger model (gpt-5.4-mini) to evaluate every generated user turn.

The consequence. A practitioner wanting to routinely measure interaction awareness in their model candidates would need to run three inference passes per example (assistant generation, user generation, judge evaluation) rather than the single pass of standard accuracy evaluation. For large-scale model comparisons or continuous integration pipelines, this triples the inference compute relative to accuracy-only evaluation. More critically, the judge model (gpt-5.4-mini) is itself a proprietary, closed-weight model—deploying the probe in environments without API access to this specific judge or in air-gapped settings would require either accepting a different, unvalidated judge or forgoing the measurement entirely. The paper's judge validation (Appendix D) is specific to gpt-5.4-mini against gpt-5.4 and human annotators; the Cohen's κ of 0.726 does not transfer to other judge models. A practitioner substituting a different judge (e.g., an open-weight alternative) would have unknown measurement reliability.

What evidence exists in the paper. The paper does not measure or discuss the computational cost of the LLM judge relative to the models under test. The judge selection process (Appendix D.1) compares nano, mini, and full gpt-5.4 on agreement metrics but does not report their relative inference costs, latencies, or hardware requirements. The human annotation protocol (Appendix D.2) involved distributing annotation packets to "peer AI researchers with expertise in LLMs" and receiving 5 completed returns—a process whose cost and turnaround time is not quantified but is clearly orders of magnitude more expensive per example than automated evaluation. The paper's temperature sweep experiments (Figure 4, Table 4) multiply the evaluation cost further: each temperature point requires a separate user generation and judge evaluation pass per example.

Mitigation status. The paper does not address this limitation. Section 4 (Limitations) notes that "future studies should explore alternative detection strategies and more thorough analysis with other judge models," implicitly acknowledging that the current judge dependency is a constraint, but does not frame this as a practical deployment cost. There is no discussion of judge distillation (training a smaller model to replicate gpt-5.4-mini's judgments), no measurement of how judge quality degrades with cheaper alternatives, and no analysis of whether simpler heuristics (e.g., embedding similarity between u and q, or keyword-based detection of prompt restatement) could substitute for some fraction of the judge calls. The human annotation validation (κ = 0.726) establishes the judge's reliability but not its necessity—we do not know whether a cheaper method would achieve comparable agreement with the same human annotations.

A Single Generation Per Example Cannot Distinguish Probability Mass from Sampling Noise

The assumption or constraint. The paper's core finding—that interaction awareness is "latent in the distribution" but surfaced at higher temperatures—is based on a temperature sweep that uses a single generated user turn per example per temperature point. The reported follow-up rate at T=1.0 (e.g., 22.0% for Qwen3.5-27B on GSM8K; Table 4) is the fraction of examples where one sample happened to be classified as a genuine follow-up. The paper argues, in Section 4 (Limitations), that "since rates are averaged over hundreds of examples per dataset, variance from individual samples is mitigated." This argument is valid for estimating the mean follow-up rate across examples but does not address a deeper measurement problem: the single-sample design cannot distinguish between a model that assigns 5% probability to follow-up continuations on many examples and a model that assigns 80% probability on a few examples and near-zero on the rest. Both would produce similar single-sample hit rates, but the models have fundamentally different distributions of interaction awareness.

The consequence. The paper's interpretation of temperature scaling—that higher temperatures "surface latent interaction awareness"—assumes that the increased follow-up rate reflects a genuine broadening of the sampling distribution to include interaction-aware continuations that were previously suppressed. But the single-sample measurement is also consistent with a different interpretation: at higher temperatures, the model simply produces a wider variety of completions, and by chance, more of them happen to satisfy the judge's criteria for genuine follow-up (e.g., the model occasionally emits a question-like string that the judge classifies as a clarification request, even though the model's weights do not encode any systematic anticipation of conversational consequences). The paper's qualitative examples (Figure 5; Appendix C) suggest the follow-ups are genuinely grounded rather than coincidental—Qwen3.5-9B's critique of the Corey-Chaykovsky reagent reasoning is specific and contentful—but without a pass@k measurement, we cannot quantify how much of the probability distribution is allocated to these grounded continuations versus to other plausible-sounding outputs at T=1.0.

What evidence exists in the paper. The paper's temperature sweep results (Figure 4, Table 4) report only the single-sample genuine-follow-up rate. There is no pass@k measurement, no estimation of the per-example probability of generating a genuine follow-up, and no analysis of how many distinct genuine follow-ups the model can produce for the same (q, a) pair. The Gaussian noise argument (rates averaged over hundreds of examples) addresses the variance of the mean estimator but does not address the fundamental limitation of the single-sample design for distinguishing between models with different per-example follow-up probability distributions. For the perturbation experiments (Table 1), the same limitation applies: we see that truncation increases the single-sample follow-up rate for gpt-oss-120B on GSM8K from 0.0% to 24.2%, but we do not know whether this represents a model that now assigns high probability to follow-up continuations on 24.2% of examples, or a model that assigns moderate probability on a larger fraction of examples where the single sample happened to land on a follow-up.

Mitigation status. The paper acknowledges the variance concern in Section 4 (Limitations) but only to dismiss it via the large-N averaging argument, not to address the deeper measurement design limitation. The paper does not propose or conduct pass@k experiments, does not estimate per-example follow-up probabilities, and does not discuss whether the single-sample design is sufficient for the claims being made. A pass@k measurement (sample k user turns per example, count the fraction of examples where at least one is genuine) would directly address this limitation by estimating the probability mass allocated to interaction-aware continuations, and the paper's generation infrastructure (vLLM on H200 GPUs) would support this with a linear increase in compute cost. The absence of this measurement is a gap between what the experimental design can demonstrate and what the claims assert about "latent" capabilities.

Evaluation Is Restricted to English-Only, Single-Turn Follow-Ups on a Narrow Set of Task Types

The assumption or constraint. The paper evaluates user-turn generation on five benchmark datasets (GSM8K, IFEval, IFBench, GPQA Main, GPQA Diamond) that share a common structure: they present a single-turn query (a math problem, an instruction-following request, a multiple-choice science question) to which the model produces a single assistant response, followed by a single generated user turn. All datasets are English-only. The paper explicitly acknowledges this in Section 4 (Limitations): "generalization to multilingual settings, code generation, or longer-horizon multi-turn interactions remains untested." The two held-out datasets (HealthBench, Coval) are also English-only conversational domains, and their results are reported only for the post-training experiments (Table 2), not for baseline model characterization.

The consequence. The paper's findings about interaction awareness—that it is decoupled from task accuracy, latent in the distribution, family-specific, and non-scaling with model size—are established for a specific interaction pattern: the model answers a self-contained question, then is asked to continue under the user role. This pattern does not generalize to the multi-turn scenarios where interaction awareness would be most consequential. In a genuine multi-turn conversation (e.g., a user asking follow-up questions, the model requesting clarification, multiple rounds of revision), the model's user-turn generation behavior may be qualitatively different because the conversational context is richer and the model has been exposed to more multi-turn data during post-training. The HealthBench baseline follow-up rate of 36.7% for Qwen3.5-2B (Table 2)—compared to ~1% on the benchmark datasets—already suggests that conversation-domain data elicits substantially more follow-up behavior. The paper documents this difference but does not characterize the conditions under which benchmark-follow-up rates predict conversational-follow-up rates. A practitioner evaluating models for a multi-turn deployment cannot infer from the paper's benchmark results how the models would perform in their actual use case, because the probe's task distribution (self-contained single-turn queries) differs systematically from the deployment distribution (interleaved, multi-turn conversations).

What evidence exists in the paper. The held-out dataset results (Table 2) provide partial evidence that follow-up behavior differs across domains: Qwen3.5-2B achieves 36.7% on HealthBench and 19.4% on Coval in the self-generated setting, versus 1.0-2.0% on IFBench and GPQA Diamond. However, these results are reported only for one model (Qwen3.5-2B) and only in the context of the post-training experiments—the paper does not provide a cross-family or within-family analysis of held-out follow-up rates for off-the-shelf models. The perturbation experiments (Table 1) are conducted only on GSM8K, GPQA Diamond, and IFBench at T=0, further narrowing the scope of the causal validation. The failure mode taxonomy (Table 6) is pooled across all five benchmark datasets but not compared against held-out dataset failure modes. The paper's qualitative examples (Appendix C) are drawn from the benchmark datasets and illustrate single-turn follow-up patterns; there are no examples of user-turn generation in multi-turn conversational contexts.

Mitigation status. The paper acknowledges the scope limitation explicitly: "generalization to multilingual settings, code generation, or longer-horizon multi-turn interactions remains untested" (Section 4). The held-out datasets (HealthBench, Coval) partially address the domain generalization concern by showing that follow-up rates differ across task types, but the paper does not analyze this difference systematically or characterize when benchmark results are predictive of conversational behavior. The post-training experiments demonstrate transfer from MATH training data to held-out conversational datasets—suggesting that interaction awareness can generalize across domains—but this is a training intervention result, not a characterization of off-the-shelf model behavior across task distributions. The paper does not propose or evaluate a protocol for measuring interaction awareness in multi-turn settings, which would be the natural extension required for deployment-relevant evaluation.

Interaction Awareness at T=0 Remains Near Zero Even for the Best Models, and the Paper Does Not Establish Whether This Matters for Deployment

The assumption or constraint. The paper's central empirical finding is that genuine-follow-up rates under deterministic generation (T=0)—the default inference mode for most deployments—are near zero for the vast majority of model-dataset combinations. For Qwen3.5 models on GSM8K, five of eight models produce 0.0% follow-ups; the maximum is 1.0% (Table 3). On GPQA Diamond, the strongest Qwen model at 397B-A17B achieves 0.5% despite 86.1% task accuracy. The cross-family story is similar: on GSM8K, all non-Qwen models are at 0.0-1.4% follow-up (Table 5). The highest T=0 follow-up rate reported in the paper is 20.7% for gpt-oss-20B on GPQA Diamond (Table 5), an outlier that still means ~79% of generated user turns are degenerate. The paper demonstrates that interaction awareness can be surfaced at higher temperatures and increased through post-training, but does not establish whether near-zero T=0 follow-up rates actually degrade performance on downstream tasks that require interaction awareness—multi-agent collaboration, self-play training data quality, conversational user satisfaction, or theory-of-mind behavioral applications.

The consequence. The paper makes a compelling case that interaction awareness is a real, measurable dimension of model behavior that is invisible to standard benchmarks. But it does not close the loop between measurement and practical impact. A model that restates the user's prompt under the user role at T=0 (Qwen3.5's dominant behavior) might still perform adequately as a conversational partner if the deployment's interaction pattern never requires the model to generate user turns—for instance, if the model only ever responds to human queries and never participates in self-play or multi-agent scenarios. Conversely, a model that shows 20% follow-up on GPQA Diamond (gpt-oss-20B) might still fail catastrophically in a multi-agent setting if the 80% of degenerate user turns cause conversation collapse (echoing, as documented by Shekkizhar et al., 2025). Without a demonstration that follow-up rates predict downstream interaction quality, the metric's practical value is unproven—it diagnoses a gap but does not quantify the gap's consequences.

What evidence exists in the paper. The post-training experiments (Table 2) provide the closest thing to a downstream validation: training that targets multi-turn collaboration quality (CollabLLM recipe, Wu et al., 2025) increases follow-up rates, suggesting that follow-up rate and collaboration quality are correlated under this training intervention. But this is a correlation between training-induced changes in follow-up rate and collaboration quality (the latter measured indirectly through the training objective's success, not through a separate downstream evaluation). It is not a demonstration that pre-existing differences in follow-up rate between off-the-shelf models predict their relative performance on a downstream interaction task. The paper does not run a multi-agent evaluation (e.g., having the model participate in self-play dialogues and measuring conversation success rate) and correlate it with follow-up rate. The perturbation experiments (Table 1) show that the metric responds to causal manipulations of the assistant turn, validating that it measures attention to assistant content, but do not show that higher follow-up rates under perturbation correspond to better downstream interaction.

Mitigation status. The paper does not claim that follow-up rate predicts downstream performance—it positions the metric as a diagnostic that "surfaces a dimension of LLM behavior... that is unexplored and invisible with current assistant-only benchmarks" (Abstract). This is a valid scientific contribution (identifying and measuring a new dimension) without requiring downstream validation. However, the paper's framing—"This capability is crucial for at least three reasons. First, conversational consequences are central to multi-agent collaboration, self-play, and interactive deployments..." (Section 1)—implies that low follow-up rates should concern practitioners deploying models in these scenarios. The paper does not provide evidence that they should. The gap between "this metric measures something real" (established by judge validation and perturbation experiments) and "this metric matters for deployment" (unestablished) is significant, particularly given the near-zero values at T=0: if the metric matters, then essentially all current models are severely deficient; if it does not matter, the metric is measuring an interesting but practically irrelevant dimension. The paper's post-training results partially bridge this gap by showing that follow-up rates respond to collaboration-oriented training, but do not close it by demonstrating that follow-up rates predict downstream interaction success in a held-out evaluation.

The Post-Training Experiments Are Preliminary, Cover a Single Model and Recipe, and Show a Severe Accuracy-Follow-Up Trade-Off Under SFT

The assumption or constraint. The paper's post-training experiments (Section 3.5) apply a single training recipe (the CollabLLM approach from Wu et al., 2025) to a single model (Qwen3.5-2B) using a single training data source (200 MATH level-5 problems expanded into ~5K multi-turn examples). The paper acknowledges the preliminary nature in Section 4: "Our post-training study was preliminary and generalization to other model families remain unexplored." Two training variants are evaluated: SFT with masked loss on assistant turns, and online RL with PPO. The SFT variant produces dramatic follow-up gains (1.0-2.0% → 46-48% on verifiable benchmarks; Table 2) but at the cost of catastrophic accuracy degradation (GSM8K: 62.9% → 40.3%, a 22.6 percentage-point drop). The RL variant preserves accuracy (62.9% → 67.4%) but produces only moderate follow-up gains (1.0-2.0% → 9.1-10.0% on verifiable benchmarks).

The consequence. The SFT-accuracy collapse is a severe practical limitation that the paper attributes to "forgetting/overfitting to ≈5k multi-turn examples" (Section 3.5) without further analysis. This attribution is a diagnosis, not an explanation: the paper does not investigate whether the accuracy loss is due to the small training corpus size (5K examples), the specific domain (hard math problems), the number of training epochs (1), the learning rate, or some fundamental tension between single-turn task accuracy and multi-turn conversational capability. A practitioner wanting to improve interaction awareness in their model cannot use the paper's SFT recipe without accepting a major regression in core capabilities, and cannot use the paper's RL recipe without accepting only modest follow-up gains—and does not know whether either trade-off can be improved through hyperparameter tuning, larger training corpora, or different data mixtures. The paper does not characterize whether the follow-up gains from SFT are robust: does the SFT model still produce 46-48% follow-up on IFBench after its GSM8K accuracy has collapsed, or has the model overfit to a narrow conversational pattern that does not generalize? The held-out dataset results (HealthBench: 36.7% → 54.4%, Coval: 19.4% → 45.2%) suggest generalization beyond the MATH training domain, but these are single-dataset point estimates without error bars or cross-validation.

What evidence exists in the paper. Table 2 reports all post-training results in four rows (Base, SFT, RL) across five columns (GSM8K accuracy, IFBench follow-up, GPQA Diamond follow-up, HealthBench follow-up, Coval follow-up). There is no learning curve (follow-up rate and accuracy as a function of training steps), no ablation of training data size or composition, no sweep of learning rates or KL penalty coefficients, and no comparison across model sizes. The paper reports that the SFT variant was trained for 1 epoch and the RL variant for 1 epoch, but does not report whether additional epochs would further increase follow-up rates, worsen accuracy collapse, or produce plateau behavior. The paper's characterization of the CollabLLM recipe references Wu et al. (2025) for details, meaning a reader cannot fully reproduce the training setup from the paper alone. The RL variant's preserved accuracy (62.9% → 67.4%, a 4.5 percentage-point improvement) is notable but unexplained—it is not clear whether this improvement is a genuine benefit of multi-turn RL training or a statistical fluctuation given the small number of evaluation examples (GSM8K: 1,319 examples, where a ~4 percentage-point change represents ~59 additional correct answers).

Mitigation status. The paper is transparent about the preliminary nature of the post-training experiments, explicitly stating they are "preliminary" and that generalization to other model families "remains unexplored" (Section 4). The claim made about post-training is appropriately narrow: "collaboration-oriented post-training can increase interaction awareness" (Section 3.5 heading), and the results are presented as a proof-of-concept that the metric responds to training interventions rather than as a recommended recipe for production. However, the SFT accuracy collapse is not discussed as a fundamental limitation of the approach—the paper frames it as "forgetting/overfitting" to the small training corpus, implying that larger corpora would solve the problem, without providing evidence for this. A dose-response experiment (varying training data size and measuring the accuracy-follow-up trade-off frontier) would characterize whether the trade-off is fundamental or an artifact of the specific experimental parameters, but such an experiment is not conducted. The paper does not suggest specific future work to address the trade-off, leaving practitioners with a diagnosis (low interaction awareness) but no validated prescription for fixing it without degrading other capabilities.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new measurement dimension for LLM evaluation—interaction awareness—that was previously invisible to the field's standard benchmarks. This is not a paradigm shift in how models are trained, but it is a significant reframing of what it means to evaluate a conversational model: the paper demonstrates that a model scoring 96.8% on GSM8K under the assistant role can simultaneously produce zero grounded conversational continuations under the user role (Qwen3.5-397B-A17B, Table 3), and that this gap is not an artifact of insufficient capability but a consequence of training data composition and optimization objectives that place interaction-unaware behavior at the distribution's mode.

The shift in thinking this enables is analogous to the introduction of calibration metrics (expected calibration error, reliability diagrams) into classification evaluation. Before calibration was measured, a model that produced correct answers 90% of the time was considered well-characterized. After calibration measurement, the field recognized that a model could be 90% accurate while being systematically overconfident or underconfident, and that this miscalibration had consequences for deployment reliability. Similarly, this paper establishes that a model can be highly accurate on assistant-turn benchmarks while being interaction-unaware—incapable of producing a grounded user reaction to its own responses—and that this unawareness has consequences for multi-turn, multi-agent, and self-play deployment scenarios. The genuine-follow-up rate is to conversational capability what calibration error is to classification confidence: a second dimension of evaluation that captures a qualitatively different property from task accuracy, is invisible to accuracy-only benchmarks, and varies independently across models and training recipes.

It reconciles conflicting intuitions about LLM conversational competence. Prior to this work, two contradictory narratives coexisted without a framework for resolving them. On one side, practitioners deploying LLMs in multi-agent and self-play scenarios documented systematic failures: Shekkizhar et al. (2025) showed that LLM agents fall into "echoing" loops when talking to each other, and Zhou et al. (2026) found that user simulators built from assistant models are overly cooperative and inflate agent scores. On the other side, standard benchmarks showed monotonic improvements in task accuracy, and models appeared to handle multi-turn conversations in chatbot interfaces reasonably well. The paper provides a reconciliation: benchmarks measure assistant-turn accuracy, which scales with model size and training compute; the user-turn generation probe measures interaction awareness, which does not scale with model size, is near-zero at T=0 for most models, and exhibits family-specific failure profiles that explain the echoing and simulator failures. The two narratives are not contradictory—they measure fundamentally different dimensions of the same models. The paper resolves the tension not by showing that one narrative is wrong, but by providing the measurement tool that makes the second dimension visible.

It redirects attention from model scale to training recipe for conversational capabilities. The within-family scaling analysis (Section 3.3, Figure 4) is the paper's most impactful single finding for research prioritization. Task accuracy scales cleanly with model size within the Qwen3.5 family (GSM8K: 41.6% → 96.8% from 0.8B to 397B-A17B; Table 3). Interaction awareness does not: Qwen3.5-9B produces 13.1% follow-up on GPQA Diamond at T=0, while Qwen3.5-397B-A17B produces 0.5%; at T=1.0 on GSM8K, the 27B model (22.0%) exceeds the 397B model (18.0%), and the 9B model on IFBench (36.0%) is competitive with the 397B (43.7%) despite a roughly 44× difference in active parameters. The implication for the scaling-laws research program is clear: if a capability does not scale with model size within a single architecture family trained by the same organization, then "train a bigger model" is not a reliable strategy for acquiring it. The capability is shaped by post-training data composition and optimization objectives—factors that are independent of model scale. This redirects research attention from pretraining compute allocation (the dominant focus since Kaplan et al., 2020, and Hoffmann et al., 2022) toward post-training data curation, multi-turn conversation design, and training objectives that impose conversational pressure on the model.

It establishes that maximum-likelihood training on predominantly single-turn data produces interaction-unaware modes. The temperature sweep experiments (Figure 4, Table 4) reveal a structural property of current post-training: the models can generate grounded follow-ups—the probability mass exists in the distribution—but these continuations are not at the mode. The training objective (next-token prediction on single-turn query-response pairs) has pushed the mode toward prompt restatement (Qwen), assistant-turn copying (gpt-oss), or meta-planning (GLM), while interaction-aware continuations are suppressed to lower-probability regions of the distribution. This is a concrete diagnosis of a training-data-optimization mismatch: maximum-likelihood training is doing exactly what it is designed to do (place highest probability on the most common training-set continuations), but the training set's most common continuations under the user role are new independent queries, not conversational follow-ups. The temperature-sweep methodology—testing whether a capability is latent in the distribution versus genuinely absent—is a general diagnostic that extends beyond interaction awareness to any capability that might be suppressed at the mode by training data composition.

It provides a taxonomy of interaction awareness failures that explains cross-family differences in multi-agent behavior. The failure mode analysis (Table 6) reveals that Qwen3.5, gpt-oss, and GLM-4.7 have systematically different default behaviors under the user role: Qwen restates the prompt (78.5%), gpt-oss copies the assistant response (70.2%), and GLM emits planning text (43.0%). These are not random errors—they are behavioral signatures of the training recipe. A practitioner who observed echo failures in a multi-agent Qwen deployment (Shekkizhar et al., 2025) can now diagnose the underlying mechanism: Qwen models treat the user-role header as a conversation-reset signal and regenerate the original prompt, which, when fed back to another agent, creates a loop of repeated queries. A practitioner using gpt-oss models would experience a different failure mode: identity leakage, where the model continues in the assistant voice under the user role, producing text that belongs on the wrong side of the conversation. The taxonomy enables failure-mode-aware model selection and suggests family-specific mitigation strategies (e.g., for Qwen, the problem is attention to assistant-turn content; for gpt-oss, the problem is converting attention to user-appropriate output once attention is achieved).

Follow-Up Research This Work Enables

Pass@k measurement of latent interaction awareness to distinguish probability concentration from sampling luck. The paper's temperature sweep uses a single generation per example per temperature point, which conflates two models: one that assigns 5% probability to follow-up continuations across many examples, and one that assigns 80% probability on a few examples and near-zero on the rest. Both would produce similar single-sample hit rates but have fundamentally different interaction awareness distributions. A direct follow-up would sample k user turns per example (k ∈ {4, 8, 16, 32}) at T=1.0 and report the pass@k rate—the fraction of examples where at least one of k samples is a genuine follow-up. This would estimate the per-example probability mass allocated to interaction-aware continuations and distinguish "broad but shallow" awareness (probability mass spread across many examples) from "narrow but deep" awareness (high probability on few examples). The same pass@k methodology could be applied to the perturbation experiments (truncation and explicit question) to test whether perturbations shift the per-example probability distribution or only increase the hit rate through sampling variance. The paper's existing generation infrastructure (vLLM on H200 GPUs) supports this with a linear increase in compute cost, and the LLM judge (gpt-5.4-mini) is already validated for the binary decision.

Temperature-dependent perturbation experiments to test whether Qwen's perturbation insensitivity is fundamental or mode-specific. The perturbation experiments (Section 3.4, Table 1) are conducted only at T=0, where Qwen models produce near-zero follow-up rates even without perturbation. The null result for Qwen (truncation: Qwen3.5-27B remains at 0.0% on GSM8K; explicit question: Qwen3.5-27B changes only 6.3% of turns on IFBench) is consistent with two interpretations: Qwen models fundamentally do not attend to assistant-turn content, or the perturbations are not strong enough to overcome the dominant prompt-restatement mode at T=0. The temperature sweep (Figure 4) shows that Qwen models do produce follow-ups at T=1.0 (Qwen3.5-27B: 22.0% on GSM8K, 35.9% on GPQA Diamond), suggesting the capability is present in the distribution. Running the truncation and explicit-question perturbations at T=1.0 would distinguish the two interpretations: if Qwen's follow-up rate increases under perturbation at T=1.0 (e.g., truncation raises GSM8K follow-up from 22% to something higher), the insensitivity at T=0 is mode-specific; if Qwen remains insensitive even at T=1.0, the model's user-turn generation is fundamentally assistant-content-independent, and the follow-ups observed at T=1.0 are generated through a mechanism that does not involve processing the assistant turn. A dose-response curve—varying the truncation percentage from 10% to 90% and measuring follow-up rate at T=0 and T=1.0—would further characterize the attention-to-content relationship per family.

Cross-family post-training to test whether interaction awareness gains transfer across architectures and training recipes. The post-training experiments (Section 3.5) apply the CollabLLM recipe (Wu et al., 2025) to Qwen3.5-2B only. A natural extension would apply the same recipe—same training data (200 MATH level-5 problems, ~5K multi-turn examples), same SFT and RL variants, same number of epochs—to gpt-oss-20B and GLM-4.7, and measure the change in follow-up rates on the same benchmark and held-out datasets. This would test whether the post-training gains are Qwen-specific (due to the interaction between the CollabLLM recipe and Qwen's pre-existing prompt-restatement failure mode) or family-general (the recipe improves interaction awareness regardless of the starting failure profile). The gpt-oss family is particularly interesting: gpt-oss models already attend to assistant-turn content (evidenced by the 70.2% assistant-turn restatement rate in Table 6 and the strong perturbation responses in Table 1) but fail to convert that attention into user-appropriate output. Would CollabLLM training teach gpt-oss to redirect its existing attention toward grounded follow-ups, producing even larger gains than observed for Qwen? Conversely, would CollabLLM training on GLM-4.7 reduce its meta-planning rate (43.0%) by teaching the model to distinguish user-role output from assistant-role planning? This experiment would reveal whether interaction awareness is trainable through a common recipe across families or requires family-specific post-training strategies tailored to each family's dominant failure mode.

Downstream validation: does genuine-follow-up rate predict multi-agent conversation success? The paper measures interaction awareness but does not establish that higher follow-up rates causally improve performance on downstream tasks that require conversational anticipation. A direct validation would correlate genuine-follow-up rate (measured at T=0 and T=1.0, and under the SFT and RL post-trained variants) with performance on a multi-agent benchmark where the model must participate in self-play dialogues. A concrete design: take the base Qwen3.5-2B, the SFT variant (48% follow-up on IFBench, degraded accuracy), and the RL variant (10% follow-up on IFBench, preserved accuracy), and have each participate in a self-play evaluation where two instances of the same model converse to solve a collaborative task (e.g., the τ2-Bench from Barres et al., 2025, or a simpler multi-turn negotiation or question-answering task). Measure conversation success rate (fraction of dialogues that reach a correct collaborative outcome), conversation length before breakdown (as in Shekkizhar et al., 2025), and human preference ratings. If the SFT variant (highest follow-up rate, lowest task accuracy) outperforms the base model on conversation success despite lower accuracy, it establishes that interaction awareness independently contributes to multi-agent performance. If the RL variant (moderate follow-up, preserved accuracy) outperforms both, it suggests that both accuracy and awareness are needed, with the optimal trade-off being application-specific. If neither post-trained variant outperforms the base model on conversation success, the follow-up metric may measure a real but practically irrelevant property—an equally important finding.

Difficulty-conditioned interaction awareness: does awareness depend on assistant response correctness? The paper evaluates follow-up rates aggregated across all examples in each dataset, but does not condition on whether the assistant response a was correct or incorrect. A natural follow-up would stratify follow-up rates by assistant answer correctness: are genuine follow-ups more common when the assistant is correct (the model produces a critique or clarification of a correct answer) or when the assistant is incorrect (the model catches the error and requests correction)? This connects to the literature on self-correction and self-critique (Huang et al., 2023; Madaan et al., 2023): if models produce more follow-ups when the assistant is incorrect, interaction awareness may partially manifest as an implicit verification mechanism. The paper's qualitative examples hint at both patterns—Qwen3.5-9B's GPQA Diamond follow-up (Figure 5) critically engages with a correct answer's reasoning, while the IFEval example (Appendix C, GLM-4.7) requests a revision of an unsatisfactory response—but there is no systematic quantification. The existing experimental data (generated assistant responses with known correctness, LLM judge labels) already contains the necessary information; the analysis requires only stratifying the existing follow-up rates by the already-computed assistant accuracy.

Domain-general interaction awareness: testing on code generation and multilingual conversation. The paper's evaluation is restricted to English-language math reasoning, instruction following, and expert QA. Two domains are particularly important for testing generality. Code generation (e.g., HumanEval, MBPP) is a natural extension because it involves structured outputs where a user follow-up might request a bug fix, ask for an explanation of a particular line, or suggest an alternative implementation—patterns that are analogous to the math reasoning follow-ups but in a domain where multi-turn debugging conversations are common in both training data and deployment. Multilingual conversation would test whether family-specific failure profiles (Qwen prompt restatement, gpt-oss assistant-turn copying) are language-dependent or reflect deeper architectural/training properties that persist across languages. The paper acknowledges both as untested (Section 4, Limitations). A minimal extension would evaluate the same 11 models on 100-example subsets of HumanEval and a multilingual conversation dataset using the same judge protocol (the LLM judge would need to handle code and non-English text, which may require judge re-validation).

Practical Applications and Downstream Use Cases

Model selection for multi-agent and self-play pipelines. The paper's family-level failure mode taxonomy (Table 6) provides actionable guidance for practitioners building systems where LLMs interact with each other. In a self-play setup where one model generates training data by simulating user feedback to another model's responses, the choice of model for the user-simulation role matters: a model that restates prompts (Qwen: 78.5% prompt restatement) will produce degenerate training data that does not provide useful feedback; a model that copies assistant responses (gpt-oss: 70.2% assistant-turn restatement) will produce training data that reinforces the assistant's existing outputs without adding new information; a model that emits planning text (GLM: 43.0% meta-planning) will produce malformed training data that may degrade the assistant model. The paper's perturbation experiments (Table 1) suggest a mitigation strategy for gpt-oss models: appending a generic conversational question to the assistant response increases genuine-follow-up rates (gpt-oss-120B: 1.3% → 25.7% on IFBench) with 99% of user turns changing text, suggesting that a lightweight prompt-engineering intervention can substantially improve user-simulation quality for this family. For Qwen models, which are insensitive to both perturbations at T=0, prompt engineering alone is insufficient—either temperature tuning (Table 4: Qwen3.5-27B reaches 22% on GSM8K at T=1.0) or post-training (Table 2: SFT reaches 48% on IFBench) would be required.

Post-training data curation for conversational applications. The paper's finding that interaction awareness is "shaped by training recipes rather than being a single unified capability" (Section 4) has direct implications for teams curating post-training data. If a model's deployment involves multi-turn conversations, self-play, or agent-agent interaction, the post-training data mixture should include multi-turn conversational examples where the model sees user reactions to assistant responses—not just single-turn query-response pairs. The paper's post-training experiments (Table 2) quantify the benefit: adding ~5K multi-turn examples (even from a narrow domain of hard math problems) raised follow-up rates on held-out conversational datasets from 36.7% to 54.4% (HealthBench, SFT) and from 19.4% to 45.2% (Coval, SFT). The RL variant suggests that this can be done without sacrificing task accuracy (GSM8K: 62.9% → 67.4%), though the follow-up gains are more modest (HealthBench: 36.7% → 46.5%). The practical takeaway is that multi-turn data should be included in the post-training mixture even for models whose primary evaluation is single-turn accuracy, because the inclusion transfers interaction awareness to conversational domains without necessarily degrading core capabilities.

Difficulty estimation for user-facing evaluation of conversational competence. The temperature-sweep methodology (Figure 4) doubles as a diagnostic for whether a deployed model is likely to produce interaction-aware behavior in practice. At T=0 (the default for most production APIs), most models produce near-zero follow-up rates. If a deployment cannot tolerate degenerate user-facing behavior (e.g., a customer-support chatbot that occasionally echoes the user's query back to them, or a multi-agent system where agent miscommunication causes task failure), the deployment must either accept non-zero temperature (trading off determinism for interaction awareness) or invest in multi-turn post-training (trading off training compute). The paper's temperature-follow-up curves provide a quantitative basis for this decision: for Qwen3.5-27B on GSM8K, T=0.7 gives 4.5% follow-up, while T=1.0 gives 22.0% (Table 4). A practitioner can choose the temperature that achieves the minimum acceptable follow-up rate for their application, calibrated to the specific model and task domain. The gpt-oss results on GSM8K (gpt-oss-120B: 0.1% even at T=1.0; Figure 1) provide a cautionary counterexample: for some model-dataset combinations, even T=1.0 does not surface interaction awareness, meaning temperature tuning alone is insufficient and post-training or model-family switching is required.

When to Prefer This Method

This paper does not propose a method for practitioners to choose among; it proposes a measurement protocol (the user-turn generation probe and LLM judge evaluation) and a diagnostic dimension (interaction awareness as measured by genuine-follow-up rate). The paper does not position this probe against alternative metrics for conversational competence, because no such standard metrics exist—the probe defines a new evaluation dimension rather than competing with an established one. A "when to prefer" matrix would be artificial because the probe is diagnostic, not prescriptive, and the paper does not articulate trade-offs against named alternatives for measuring interaction awareness. The appropriate framing is: when should a practitioner run this probe on their model? The paper's results imply it is most valuable when the model will be deployed in multi-turn, multi-agent, or self-play settings where conversational anticipation matters, when the practitioner is curating post-training data and wants to measure whether data composition affects interaction awareness, or when comparing models from different families for a conversational application where accuracy-only benchmarking would miss the interaction awareness dimension. It is less valuable when the model is deployed in single-turn, fire-and-forget settings with no conversational continuation (e.g., one-shot classification, single-turn information extraction), because the absence of interaction awareness in such settings has no practical consequence.