Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

🎯 Pitch

LLMs fine-tuned to follow instructions actually collapse into brittle pattern-matching: one leading model plunges from 5th to 15th place when told to write an unstructured essay or give deliberately wrong answers. This paper introduces Inverse IFEval, showing that while thinking models partially resist this cognitive inertia, today’s alignment methods leave even the best systems deaf to commands that clash with their training.

1. Executive Summary

This paper introduces Inverse IFEval, a benchmark designed to evaluate what the authors term Counter-intuitive Ability—a model's capacity to override training-induced biases and comply with adversarial instructions that conflict with standardized patterns learned during supervised fine-tuning (e.g., requiring an unstructured essay with no paragraph breaks, or deliberately incorrect answers to simple factual questions). The benchmark constructs 1012 Chinese and English questions across eight adversarial instruction types—ranging from Question Correction to Counterfactual Answering—and evaluates 15 leading LLMs using an optimized LLM-as-a-Judge framework that achieves 98% scoring accuracy. The best-performing model, o3-high, achieves top scores while fine-tuned non-thinking models drop dramatically—Qwen3-235B-A22B-Instruct plunges from 5th on IFEval to 15th on Inverse IFEval—establishing that current alignment methods produce brittle instruction-following that collapses when instructions deviate from training conventions, even while thinking mechanisms and increased test-time compute (Best-of-32 pushing four of five models near or above 90) offer partial mitigation.

2. Context and Motivation

The Core Problem: LLMs Follow Training Patterns, Not Instructions

The fundamental problem this paper tackles is deceptively simple: LLMs fine-tuned to follow instructions often fail precisely when those instructions conflict with the patterns they were trained on. This isn't about models being incompetent or failing to understand what was asked — it's about a deeper phenomenon the authors call cognitive inertia: the persistent tendency to replicate training-induced behaviors even when explicitly told not to.

The paper crystallizes this with a confrontational user demand: "Do As I Say, Not As You Were Trained!!" This captures the frustration of deploying models in real-world scenarios where users issue instructions that deviate from the idealized formats present in training data. Figure 1 demonstrates this concretely: when asked to strictly avoid bullet point formatting — a directive that contradicts the structured, list-heavy responses common in SFT data — models frequently fail to comply, reverting to their trained patterns despite the explicit prohibition.

Why This Matters: The Gap Between Training Paradigms and Real-World Use

The significance of this problem extends well beyond academic curiosity. The authors ground their motivation in a detailed understanding of how SFT data is actually produced, based on their direct experience with annotation pipelines. They observed a systematic bias in data creation:

"SFT data annotation tends to adhere to an 'idealized paradigm', where annotators construct data by following a predefined, idealized response format."

This isn't a bug in any particular annotation process — it's a structural property of how instruction-tuning datasets are built. Annotators are typically instructed to produce well-formatted, factually correct, logically structured responses that follow best practices. The resulting datasets overwhelmingly reinforce conventions like:

Using bullet points and numbered lists for clarity
Adding explanatory comments in code
Providing factually correct answers
Structuring responses with paragraph breaks and section headers
Following standard formatting norms

When models are fine-tuned on this data, they don't just learn to produce good responses — they internalize these formatting patterns and correctness norms as default behaviors. The problem emerges when a user wants something that violates these defaults: an unstructured wall of text, an intentionally incorrect answer for testing purposes, or code deliberately stripped of documentation. The model's training has taught it that such outputs are "wrong," and it struggles to overcome that conditioning.

This tension is not hypothetical. The authors argue that long-tail user needs inevitably exist that post-training fails to cover. Real-world applications involve unconventional, ambiguous, or dynamically shifting requirements that no amount of careful annotation can fully anticipate. A model that can only follow instructions that align with its training distribution is brittle in precisely the ways that matter for deployment reliability.

The Analogy to Human IQ Testing

The paper makes an illuminating analogy to justify why contrived, unnatural instructions are the right evaluation tool:

"We do not assert these instructions are inherently meaningful in a practical sense. Rather, we argue that they reflect a model's generalization capability for following instructions. This is analogous to human IQ tests, which do not consist of problems encountered in daily life but can effectively measure human intelligence."

This framing is crucial for understanding the benchmark's philosophy. IQ tests deliberately present problems that are out-of-distribution relative to everyday experience — that's what makes them diagnostic of general cognitive ability rather than learned routines. Similarly, Inverse IFEval's instructions aren't meant to represent common user requests; they're diagnostic probes that reveal whether a model has genuinely learned to follow instructions versus merely learning to reproduce training patterns.

This distinction — between instruction-following as a generalized capability versus instruction-pattern-matching — is the theoretical heart of the paper. A model that scores well on conventional instruction-following benchmarks but poorly on Inverse IFEval hasn't truly learned to follow instructions; it has learned to recognize and comply with the specific types of instructions present in its training distribution.

Where Prior Benchmarks Fall Short

The paper identifies specific limitations in existing instruction-following evaluation:

IFEval and its derivatives test convention-aligned compliance. IFEval (Zhou et al., 2023) is the most directly comparable benchmark. It evaluates whether models can follow verifiable instructions — things like "write at least 400 words," "include the keyword X," or "use at least 3 bullet points." These are precisely the kinds of instructions that appear in training data. A model that performs well on IFEval might simply be pattern-matching to familiar directive types rather than exhibiting genuine instruction-following flexibility.

Multilingual extensions don't address the core issue. Benchmarks like M-IFEval (Dussolle et al., 2025) extend IFEval to French, Japanese, and Spanish but inherit the same limitation: they test convention-aligned instruction following across languages, not the ability to follow instructions that contradict conventions. The paper explicitly notes this gap:

"We noticed that no previous works focus on the inverse instruction situation, which is exactly the focus of our work."

Domain-specific benchmarks evaluate factual knowledge, not instruction flexibility. Benchmarks like MMLU (Hendrycks et al., 2021) assess whether models know facts and can reason about domain-specific content. They don't test whether models can deliberately depart from that knowledge when instructed to do so — for instance, whether a model that knows the capital of France is Paris can successfully answer "Berlin" when told to provide deliberately incorrect answers.

Vision-language and retrieval benchmarks address different modalities. Benchmarks like VisIT-Bench (Bitton et al., 2023) and InstructIR (Oh et al., 2024) evaluate instruction following in vision-language and retrieval contexts respectively, but they address multimodal capabilities rather than the fundamental flexibility problem of whether a model can override its training biases.

Table 1 comparison shows the structural gap. The paper's comparison table highlights that while IFEval evaluates instruction following, it has no "inverse instruction" dimension. Complex instruction-following benchmarks (e.g., FollowEval, ComplexBench) test multi-constraint compliance but still within conventional paradigms. None systematically invert training conventions to test whether models can unlearn their defaults.

Conflicting Evidence in Prior Work Creates Ambiguity

The paper doesn't frame its contribution as resolving an explicit empirical contradiction in the literature (unlike the test-time compute paper). Rather, it identifies a more subtle gap: the field has no measurement instrument for counter-conventional instruction following, so we don't even know how bad the problem is. Prior work has extensively documented that models can follow instructions and has gradually improved that capability through better instruction tuning methods. But the question of whether this capability generalizes to instructions that actively conflict with training patterns has been systematically unexamined.

This is significant because it means existing benchmarks may be producing false confidence. A model that achieves 85% on IFEval might appear to have strong instruction-following capabilities, but if that performance collapses to 30% when instructions contradict training conventions, the model isn't truly instruction-following — it's pattern-matching. The field simply hadn't built the instrument to detect this distinction.

How This Paper Positions Itself

The paper positions Inverse IFEval not as a replacement for existing benchmarks but as a complementary diagnostic tool that measures a distinct dimension of instruction-following capability. The introduction frames this dimension — Counter-intuitive Ability — as a critical but underexplored aspect of LLM evaluation that captures a specific type of robustness: the ability to unlearn training conventions when explicitly told to do so.

This positioning has several implications:

It defines a new evaluation axis, not a new training method. The paper doesn't propose techniques to improve counter-intuitive instruction following. It establishes that the capability gap exists, characterizes which types of inverse instructions cause the most trouble, and provides a standardized measurement instrument. This makes it foundational work — defining the problem that future methods will need to solve.

It connects to broader concerns about overfitting in post-training. The paper explicitly links cognitive inertia to overfitting risk: "when models become overly attuned to post-training patterns, they may lose flexibility and fail to generalize beyond the narrow conventions reinforced during annotation." This frames the problem within the well-understood concept of overfitting but applies it to instruction-following behavior rather than task accuracy, which is a novel extension.

It distinguishes between thinking and non-thinking models. A key positioning choice is the paper's emphasis on the role of reasoning mechanisms (chain-of-thought, extended thinking budgets) in mitigating cognitive inertia. By showing that thinking models consistently outperform their non-thinking counterparts (e.g., Qwen3 thinking vs. non-thinking variants, Gemini full-thinking vs. Flash with reduced thinking budget), the paper suggests that the process of deliberation helps models override their training defaults — an insight that connects to broader findings about reasoning models' improved robustness.

It frames the problem as generalization, not adversarial robustness. The paper is careful to note that the inverse instructions aren't adversarial attacks — they're legitimate (if unusual) user directives. The benchmark tests whether models can generalize instruction-following capability beyond the training distribution, which is a different framing than testing robustness against malicious inputs. This matters because it positions the work within the broader literature on out-of-distribution generalization rather than adversarial robustness, which carries different assumptions about what constitutes a valid test.

It aims to be both diagnostic and foundational. The conclusion explicitly states:

"We hope that Inverse IFEval serves as both a diagnostic tool and a foundation for developing methods that mitigate cognitive inertia, reduce overfitting to narrow patterns, and ultimately enhance the instruction-following reliability of LLMs in diverse and unpredictable real-world scenarios."

This dual-purpose framing — measurement and catalyst — is standard for benchmark papers but takes on additional weight here because the diagnostic reveals a capability gap that the field hadn't previously characterized. Before you can fix a problem, you need to measure it, and Inverse IFEval provides that measurement basis.

3. Technical Approach

3.1 Reader Orientation

This paper is a benchmark construction and evaluation paper that creates a diagnostic instrument—a dataset of adversarial instructions paired with a reliable automated evaluation pipeline—rather than proposing a new model or training method. The system being built is both the dataset itself (1012 question-answer pairs with evaluation rubrics) and the evaluation framework that grades model responses against those rubrics. The problem it solves is measurement: before the field can improve models' ability to follow counter-conventional instructions, we need a reliable way to measure how badly current models fail at this task. The "shape" of the solution is a carefully curated set of instructions that systematically invert standard training paradigms, coupled with a high-accuracy LLM-as-a-Judge pipeline that can automatically score compliance at scale.

3.2 Big-Picture Architecture (Diagram in Words)

The Inverse IFEval system has four major components connected in a pipeline:

Instruction Type Taxonomy — a classification system identifying eight distinct ways training conventions can be inverted, derived from systematic observation of SFT annotation practices. Each type specifies a pattern of counter-conventional behavior (e.g., "provide deliberately incorrect answers" or "write code with no comments") that conflicts with what models learn during fine-tuning.
Data Generation Pipeline — a multi-stage, human-in-the-loop process that produces the actual benchmark questions. It flows through five stages: (a) expert analysis of training paradigms to identify inversions, (b) manual seed question construction by domain experts, (c) large-scale LLM-based question generation using tailored prompting templates, (d) automatic filtering for quality control, and (e) rigorous human review and calibration.
The Benchmark Dataset — the final collection of 1012 high-quality questions (506 Chinese, 506 English) spanning 23 knowledge domains and eight instruction types, each annotated with type labels, domain labels, and standardized evaluation rubrics for automated scoring.
LLM-as-a-Judge Evaluation Framework — an optimized automated scoring system that uses different judge models for different instruction types, employs type-specific judging templates, and incorporates enhanced system prompts with scoring logic explanations and exemplar demonstrations. This component achieved 98% accuracy against human expert judgments.

Information flows through these components sequentially: first, the taxonomy defines what kinds of adversarial instructions to test; second, the generation pipeline produces questions conforming to that taxonomy; third, the curated dataset serves as the evaluation instrument; fourth, the judge framework scores model responses against predetermined rubrics.

3.3 Roadmap for the Deep Dive

First, the eight adversarial instruction types — their definitions, the training conventions they invert, and concrete examples — because the entire benchmark is built around this taxonomy and understanding the types is prerequisite to understanding everything else.
Second, the data construction pipeline in its five sequential stages, including the specific prompting strategies, filtering mechanisms, and expert review procedures — because the benchmark's validity depends entirely on the quality control embedded in this pipeline.
Third, the LLM-as-a-Judge evaluation framework and its three optimization strategies — because the benchmark's practical utility depends on automated scoring that matches human judgment.
Fourth, key design choices and their justifications — because understanding why particular decisions were made reveals the paper's methodology more deeply than merely describing what was done.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction paper whose core idea is that instruction-following evaluation must test not just compliance with training-aligned conventions but also the ability to override those conventions when explicitly instructed to do so.

The Eight Adversarial Instruction Types

The foundation of Inverse IFEval is a taxonomy of eight instruction types, each defined by identifying a standard training convention and then inverting it to create a counter-conventional directive. The authors derived these types through systematic analysis of widely used SFT datasets, identifying canonical response paradigms that annotators consistently follow, and then designing instructions that explicitly contradict those paradigms. The eight types, their regular training paradigms, and what they demand are as follows (with full demonstrations in Appendix Figures 7–14):

1. Question Correction. The regular training paradigm: models are trained to answer questions as asked, accepting the premise provided by the user. The inversion: the instruction asks the model to identify and correct factual errors or problematic assumptions embedded within the question itself before answering — or sometimes instead of answering. This requires the model to question the user's framing rather than naively accepting it, which conflicts with the standard assistant behavior of being helpful and accommodating.

2. Intentional Textual Flaws. The regular training paradigm: SFT data trains models to produce grammatically correct, well-punctuated, typo-free text that follows standard writing conventions. The inversion: the instruction explicitly demands that the model introduce specific types of textual errors — misspellings, grammar mistakes, missing punctuation, awkward phrasing — into its response. This directly conflicts with the model's deeply ingrained training to produce polished, correct text.

3. Code without Comments. The regular training paradigm: coding instruction data overwhelmingly teaches models to produce well-documented code with explanatory comments, following software engineering best practices that are heavily represented in training corpora. The inversion: the instruction demands code with absolutely no comments, no docstrings, and no inline explanations — a directive that feels "wrong" to a model trained on best-practices data where comments are consistently present and encouraged.

4. Counter-Conventional Formatting (Non-Code). The regular training paradigm: SFT responses are structured with paragraph breaks, bullet points, numbered lists, section headers, and clear visual organization — formatting conventions that annotators consistently apply to make responses readable. The inversion: the instruction demands an unstructured, continuous block of text with explicitly prohibited formatting elements — "no bullet points, no paragraph breaks, no numbered lists, no headers." This is the most direct assault on the formatting conventions that SFT data overwhelmingly reinforces. Figure 1 in the paper demonstrates exactly this type, where models asked to "strictly avoid using bullet point format" nonetheless produce bullet-pointed responses.

5. Deliberately Incorrect Answers. The regular training paradigm: models are trained exclusively to provide factually accurate, truthful responses — correctness is the primary optimization target in both SFT and RLHF. The inversion: the instruction explicitly demands an answer that the model knows is factually wrong. For example, "Tell me the capital of France, but answer with a different European capital city instead." This forces the model to knowingly produce an output that contradicts both its training objective and its internal knowledge.

6. Instructional Induction. The regular training paradigm: training data typically presents clear, direct instructions that explicitly state what the model should do. The inversion: the model must induce a latent instruction from a provided input-output example pair and then apply that induced pattern to a new input. This is a form of few-shot meta-learning embedded within a single instruction-following task — the model must recognize the transformation pattern from the example and generalize it, rather than receiving explicit natural language directions.

7. Mid-turn Instruction Modification. The regular training paradigm: in multi-turn conversations, instructions are typically consistent across turns or build cumulatively — annotators don't usually provide instructions that directly contradict what was asked earlier in the conversation. The inversion: the instruction changes mid-way through a multi-turn interaction, requiring the model to abandon a previously given directive and adopt a new, potentially contradictory one. This tests whether the model can dynamically update its task representation rather than perseverating on the initial instruction.

8. Counterfactual Answering (with Explicit Constraints). The regular training paradigm: models answer questions based on actual facts about the real world, grounded in their training knowledge. The inversion: the instruction sets up a counterfactual scenario — "Assume X is true (even though it's not)" — and asks the model to reason and answer within that hypothetical world while maintaining internal consistency. This requires the model to temporarily suspend its factual knowledge and operate within an explicitly specified alternative reality.

These eight types collectively cover a broad spectrum of training conventions: factual correctness, formatting norms, linguistic quality standards, code documentation practices, conversational consistency, direct-instruction understanding, and real-world grounding. By inverting each, the benchmark probes whether models can unlearn each category of convention when explicitly told to do so.

The Data Construction Pipeline: Five Sequential Stages

The benchmark of 1012 questions is constructed through a carefully orchestrated five-stage pipeline (illustrated in Figure 2) that combines expert human judgment with large-scale LLM-based generation, followed by rigorous filtering and verification. The process ensures both scale (thousands of candidate questions) and quality (expert-verified final selection).

Stage 1: Observation and Reversal. The construction begins not with question generation but with systematic analysis of existing SFT datasets. The authors — drawing on their direct experience with annotation teams — examined the patterns that annotators consistently follow when producing training data. They identified a set of "canonical response paradigms" such as "follows best practices," "produces well-formatted outputs," "provides factually correct information," and "includes explanatory comments in code." For each identified paradigm, they designed an inversion — a directive that explicitly contradicts the norm. This stage produced the eight-type taxonomy described above. The key methodological insight here is that the taxonomy wasn't derived abstractly but emerged from concrete observation of annotation behavior, which the authors describe as "multiple rounds of discussion and alignment with annotation teams regarding corpus selection, annotation standards, and training objectives."

Stage 2: Seed Data Construction. With the eight types defined, the authors invited "domain experts in LLM training" to manually craft a batch of high-quality seed questions for each type. These seeds serve a dual purpose: they establish exemplars that demonstrate exactly what a good inverse instruction looks like, and they provide the foundation for the prompting templates used in the subsequent large-scale generation stage.

The seed construction process incorporates a multi-dimensional cross-validation mechanism. In addition to the core expert annotation team, reviewers from diverse backgrounds — specifically product, engineering, and operations roles — independently assessed each seed question. The paper specifies a strict inclusion criterion: "unanimous judgments ('qualified' vs. 'unqualified')" were required before a question could enter the seed set. This cross-functional review strategy was designed to ensure consistency across different cognitive perspectives, preventing the seed set from reflecting only the biases of the annotation experts. The requirement of unanimous agreement is an unusually stringent quality bar that eliminates borderline cases.

Stage 3: Large-Scale Data Generation. Building on the manually crafted seeds, the authors applied prompt engineering to design dedicated generation templates for each of the eight instruction types. These templates encode the structural pattern of each inverse instruction type and guide LLMs to produce new questions that conform to that pattern while varying the surface content.

To maximize domain coverage, the generation process incorporated a predefined disciplinary taxonomy spanning mathematics, physics, geography, literature, law, biology, and other fields (ultimately covering 23 distinct domains as shown in Figure 3). The LLMs were explicitly guided to generate domain-specific questions for each type × domain combination.

The generation followed a specific quantitative protocol: for each domain × type combination, 20 candidate questions were generated in the initial stage. Multiple state-of-the-art models were employed in collaborative generation — meaning different models contributed to different parts of the question set, providing diversity in phrasing, complexity, and style. After this initial generation, an automatic filtering mechanism was applied to conduct preliminary quality assurance, checking for:

Length constraints: questions and expected answers must fall within reasonable bounds to be evaluable and to avoid pathological cases.
Semantic similarity detection: near-duplicate questions were identified and removed to prevent redundancy in the benchmark. The paper mentions this as a filter but does not specify the exact similarity threshold or embedding method used.

After this initial filtering, the authors examined coverage gaps — domain × type combinations that were underrepresented in the filtered set — and conducted targeted generation to fill those gaps. They also implemented cross-model verification during this stage, where generated questions were evaluated by different models than the one that produced them to ensure coherence and robustness.

Stage 4: Automatic Filtering. The paper distinguishes between the automatic filtering applied during generation (Stage 3) and a more comprehensive automatic filtering stage applied before human review. While the details of this stage are not extensively elaborated, the paper indicates that it included the length constraints and semantic similarity detection mentioned above, likely with more stringent thresholds applied to the larger candidate pool before humans examined it. The purpose was to reduce the human review burden by eliminating clearly problematic questions algorithmically.

Stage 5: Expert Review and Calibration. All machine-generated questions that passed automatic filtering underwent rigorous expert review — the final and most important quality gate. The review focused on three key aspects:

Type consistency: reviewers verified that each question was precisely aligned with its designated counterintuitive instruction type. A question that was supposed to test "Intentional Textual Flaws" but inadvertently tested something else — or that mixed multiple inverse paradigms — would be rejected or reassigned. This is critical because the benchmark's validity depends on clean type-level separation; if questions are misclassified, type-level analysis becomes meaningless.
Clarity of instruction: reviewers systematically detected and eliminated potential ambiguities in the questions. This included checking for semantic vagueness (is it clear exactly what the model is being asked to do?), unclear references (do pronouns and demonstratives have unambiguous antecedents?), and logical contradictions (does the instruction contradict itself in a way that makes compliance impossible?). The paper emphasizes that the goal is to test whether models can follow clear but counter-conventional instructions, not whether they can resolve ambiguous or contradictory ones.
Scoring rubric calibration: for each question, reviewers designed fine-grained rubrics specifying exactly what constitutes a compliant response and what constitutes a violation. These rubrics were validated through multiple pilot evaluations — meaning they were applied to sample model responses and checked for inter-rater reliability — to ensure both operability (the rubric can be applied consistently) and discriminative power (the rubric distinguishes between genuinely compliant and non-compliant responses, not just between obviously good and obviously bad ones).

After expert review and calibration were complete, the final step was Chinese-English translation of the entire question set. The paper states that the benchmark contains "the same number of instructions in both Chinese and English" (506 each), meaning every question exists in two parallel language versions. The translation process is not detailed but presumably involved professional translation or LLM-assisted translation with human verification, given the overall emphasis on quality control.

The final dataset comprises 1012 questions annotated with metadata: instruction type labels (one of eight categories) and domain labels (one of 23 categories, shown in Figure 3). The domain distribution is highly skewed toward Computer Science, which accounts for 20.2% of the dataset — a notable imbalance that reflects either the difficulty of generating inverse instructions in other domains or the composition of the seed question set. Table 2 provides the per-type statistics: the types range from 82 samples (Counter-Conventional Formatting) to 198 samples (Code without Comments), meaning some types have more than twice as many evaluation items as others. The "Code without Comments" type also exhibits the longest average reference answer length because those instructions typically require both code and functional explanations, while "Mid-turn Instruction Modification" has the longest average question length due to containing multiple segments of conversational text.

The LLM-as-a-Judge Evaluation Framework

To evaluate model responses at scale, the paper adopts the "LLM-as-a-Judge" paradigm — using a strong language model to assess whether another model's output complies with the specified inverse instruction. This avoids the need for human evaluation of every model response (which would be prohibitively expensive at the scale of 15 models × 1012 questions × multiple configurations) while aiming to match human judgment quality.

The evaluation framework's development was itself an optimization process. The authors report that their "initial baseline judge model achieves an accuracy of 88%" when compared against human expert judgments. Through systematic optimization, they improved this to a "final judging accuracy of 98%." The three optimization strategies that produced this improvement are as follows.

Strategy 1: Dedicated Judge Model Selection (Adaptive Model Matrix). Rather than using a single judge model for all instruction types, the authors tested multiple state-of-the-art models on each instruction type separately and selected the model that exhibited the highest scoring accuracy for that specific type. This creates what they call an "adaptive, optimal accuracy judge model matrix" — a mapping from instruction type to judge model where each type gets its best-performing evaluator. The intuition is that different instruction types require different evaluation capabilities: judging whether code contains comments is a different skill than judging whether an answer is deliberately incorrect, and models may have type-specific strengths as judges. The paper does not specify which models were selected for which types or the magnitude of the accuracy differences that motivated this design choice.

Strategy 2: Optimization of Judging Template Structure. The paper observed that "the dependency of different instruction types on context varies significantly, leading to substantial differences in accuracy when using identical templates across the same judge model and question set." In other words, the format of the evaluation prompt — how the instruction, model response, and scoring criteria are presented to the judge model — matters a great deal, and the optimal format differs by instruction type. The authors therefore selected "the most effective template structure for each instruction type," again creating a type-specific mapping rather than a one-size-fits-all evaluation prompt. The paper does not provide examples of these templates or quantify the accuracy improvements from template optimization alone.

Strategy 3: Enhancement of the Judge's System Prompt. To improve the judge model's robustness and its ability to understand complex evaluation criteria, the authors deeply optimized the system prompts used in the evaluation process. Two specific measures were employed:

Detailed scoring logic explanations: for each counterintuitive instruction type, the system prompt was supplemented with more detailed explanations of the scoring logic — essentially teaching the judge model what to look for and how to weigh different aspects of compliance. For example, for "Code without Comments," the prompt might explain that any inline comment, docstring, or explanatory text within code blocks constitutes a violation, regardless of whether the explanation appears helpful or benign.
Exemplar demonstrations: the system prompt incorporates a small set of sample examples related to each kind of question, showing the judge model correct scoring criteria visually. These are essentially few-shot examples within the evaluation prompt, demonstrating what a compliant response looks like, what a non-compliant response looks like, and how each should be scored.

The 98% accuracy figure is reported as agreement with human expert judgments in a validation process where "each question is paired with two different model responses and a ground truth score verified by human experts." The paper does not detail the size of this validation set, the inter-annotator agreement among the human experts, or whether the 98% figure reflects exact score matching or a more lenient criterion.

Design Choices and Their Justifications

Several methodological decisions in the paper's approach are worth examining in detail, as they reveal the reasoning behind the benchmark's construction.

Why eight types? The paper doesn't claim these eight types exhaustively cover all possible inversions of training conventions. Rather, they emerged from the systematic observation of SFT annotation practices described in Stage 1 of the pipeline. The number eight is therefore an empirical outcome of the observation-and-reversal process rather than an a priori design target. The taxonomy covers a deliberately broad range: factual accuracy, formatting, linguistic quality, code documentation, conversational dynamics, instruction interpretation, and counterfactual reasoning — spanning both surface-level conventions (formatting, comments) and deeper cognitive patterns (truthfulness, consistency). This breadth is important because it allows the benchmark to distinguish between models that are brittle in specific ways (e.g., can't violate formatting norms but can provide incorrect answers) versus models that exhibit generalized cognitive inertia.

Why the IQ test analogy? The paper's explicit framing — "We do not assert these instructions are inherently meaningful in a practical sense" — is a methodological choice that anticipates criticism. A natural objection to Inverse IFEval is: "When would anyone actually ask a model to produce intentionally flawed text?" The IQ test analogy preempts this by arguing that diagnostic value doesn't require ecological validity. An analogy helps here: we don't evaluate a student's mathematical ability by asking them to calculate their grocery bill — we give them abstract problems that don't resemble daily life but do reveal whether they understand mathematical principles. Similarly, Inverse IFEval's instructions don't need to be common user requests; they need to reveal whether the model has genuinely learned the abstract capability of "following instructions" versus merely learning to match familiar patterns.

Why human-in-the-loop rather than purely automated generation? The five-stage pipeline invests heavily in human expertise at multiple points: the initial observation of SFT patterns, seed question construction, cross-functional review, and final expert calibration. This stands in contrast to benchmarks that use purely automated generation from LLMs. The justification is validity: without human verification, there is no guarantee that generated questions actually test what they're supposed to test. A purely LLM-generated "Code without Comments" instruction might inadvertently contain ambiguities that make compliance impossible, or might inadvertently test something other than comment inclusion. The expert review stage — with its explicit focus on type consistency, instruction clarity, and rubric calibration — serves as the validity check that automated approaches lack.

Why two language versions? The Chinese-English parallel construction (506 questions each) enables cross-lingual comparison of the same underlying capability. The paper reports in Section 8.2 that "models demonstrate consistent performance across both languages" overall, but notes that GLM-4.5 and Qwen3-235B-A22B-Thinking "perform significantly better in the Chinese version than in the English version" — likely because "the Chinese community contributes these two models," meaning their training data is skewed toward Chinese. This finding demonstrates the benchmark's utility for detecting training data biases: models overfit not just to formatting conventions but also to the language distribution of their training data.

Why the adaptive judge model matrix? The decision to use different judge models for different instruction types — rather than a single unified judge — reflects an important methodological insight: evaluation capability is not uniform across task types. A model that excels at judging formatting compliance might struggle to detect deliberately incorrect factual statements, and vice versa. The adaptive matrix approach optimizes for per-type accuracy at the cost of increased system complexity. The paper reports that this contributed to the 10 percentage point improvement from 88% to 98% judging accuracy, though it doesn't isolate the contribution of this specific strategy versus template optimization or system prompt enhancement.

Why 98% judging accuracy is necessary. The paper treats 98% as a target rather than an incidental achievement. The justification is implicit in the benchmarking philosophy: if the judge makes systematic errors at a rate significantly higher than the performance differences between models, then benchmark scores become unreliable. If two models differ by 5 percentage points but the judge has 10% error, the ranking between those models is essentially random. By pushing judging accuracy to 98%, the authors ensure that measured performance differences reflect genuine differences in instruction-following capability rather than evaluation noise.

The unaddressed question of judging cost. While the paper optimizes for judging accuracy, it does not discuss the computational cost of the adaptive judge model matrix approach. Using different judge models for different instruction types — possibly including very large models like o3-high or Claude-4-Opus as judges — could make evaluation extremely expensive at scale. For a benchmark intended to be widely adopted, the cost of running the judge pipeline is a practical consideration that the paper leaves unexplored.

4. Key Insights and Innovations

Innovation 1: Defining "Counter-intuitive Ability" as a Distinct Evaluation Dimension

The paper's most fundamental conceptual contribution is not the benchmark itself but the diagnostic framing that makes the benchmark necessary: the identification of Counter-intuitive Ability as a distinct dimension of instruction-following capability that existing benchmarks systematically fail to measure.

Prior to this work, the field's evaluation of instruction following operated under an implicit assumption: if a model can follow instructions well, it can follow any instruction well, modulo surface-level difficulty (length, complexity, number of constraints). Benchmarks like IFEval (Zhou et al., 2023) tested whether models could comply with directives that align with training conventions — "write at least 400 words," "include the keyword X," "use at least 3 bullet points" — and interpreted high scores as evidence of strong instruction-following capability. The possibility that this capability might be conditional on instruction type — that models might excel at convention-aligned instructions while collapsing on convention-contradicting ones — was not just unmeasured but conceptually unarticulated.

This paper introduces a sharp distinction between instruction-pattern-matching and genuine instruction-following. The former is what fine-tuned models actually learn: recognize familiar directive patterns in the input and reproduce the corresponding trained behaviors. The latter is the abstract capability to comply with any well-specified instruction, regardless of whether it aligns with training conventions. The paper's core conceptual move is arguing that these are distinct capabilities, that SFT primarily teaches the former, and that existing benchmarks only measure the former — creating a systematic blind spot in the field's understanding of model robustness.

The evidence for this distinction is stark and comes primarily from the ranking inversion documented in Figure 6 and the main results table (Table 3). Qwen3-235B-A22B-Instruct ranks 5th on IFEval but drops to 15th on Inverse IFEval — a ten-position collapse that cannot be explained by differences in task difficulty alone. If Inverse IFEval were merely harder in some uniform way, all models would drop proportionally and rankings would be preserved. Instead, the ranking reorders, indicating that the two benchmarks measure fundamentally different capabilities. Models that rank highly on IFEval are not necessarily good at counter-intuitive instruction following; models that rank poorly on IFEval are not necessarily bad at it. This rank instability is the empirical signature of a distinct evaluation dimension.

This contribution is foundational rather than incremental — it doesn't refine existing evaluation methodology but rather identifies a capability gap that the field didn't know existed. It's analogous to the moment in computer vision when researchers realized that ImageNet-trained models weren't actually recognizing objects but were exploiting dataset-specific texture cues (Geirhos et al., 2019), or in NLP when researchers discovered that models weren't reasoning but were exploiting annotation artifacts (Gururangan et al., 2018). In each case, the discovery wasn't a better metric within an existing framework but a recognition that the framework itself was measuring the wrong thing. Inverse IFEval makes the same kind of move for instruction following.

The "IQ test" analogy in the paper (Section 2.1 and Appendix 7) is not merely rhetorical — it encapsulates the theoretical justification for this new dimension. IQ tests are deliberately unnatural; their diagnostic value comes precisely from being unlike everyday problems. Similarly, Inverse IFEval's value comes from instructions that are unlike training data. This framing anticipates and neutralizes the objection that the benchmark tests unrealistic scenarios: realism is not the goal; diagnostic discrimination is.

Innovation 2: The Taxonomy of Training Convention Inversions as a Systematic Decomposition of Cognitive Inertia

The paper's second conceptual contribution is the eight-type taxonomy of counter-conventional instructions, which decomposes the broad phenomenon of cognitive inertia into specific, separable sub-capabilities. This isn't merely a convenient organizational scheme — it's a diagnostic instrument that reveals which training conventions models are most rigid about and where different models exhibit different weakness profiles.

Prior instruction-following benchmarks either tested atomic constraints (IFEval's "include keyword X," "write N words") without distinguishing between training-aligned and training-contradicting directives, or organized tasks by domain (MMLU) or skill type (reasoning, coding) without reference to training conventions at all. No prior benchmark organized its evaluation around the relationship between the instruction and the training distribution — specifically, whether the instruction aligns with or inverts learned conventions. This is the key structural innovation: the taxonomy is defined by what training pattern is being violated, not by what surface task is being performed.

The taxonomy's power becomes evident in the per-type results (Figure 5, Table 3). If cognitive inertia were a uniform phenomenon — if models were equally rigid about all training conventions — we would expect flat per-type performance (all types equally hard) and consistent model rankings across types. Instead, Figure 5 reveals:

Counterfactual Answering is consistently the easiest type across all models (>65% for every model shown), suggesting that temporarily suspending factual knowledge to reason within a hypothetical is less constrained by SFT conventions than other inversions. This is a non-obvious finding: one might have expected factual correctness to be among the most deeply ingrained conventions given its centrality to RLHF optimization, yet models can override it relatively easily.
Question Correction is consistently the hardest type, with half of the models scoring below 30 and even the top model (o3-high) showing notably lower performance than on other types. This suggests that the training convention of "accept the user's premise and answer the question as asked" is exceptionally difficult to override — models are fundamentally trained to be accommodating rather than critical.
Model-specific weakness profiles emerge: DeepSeek-R1 is notably weak on Intentional Textual Flaws and Counter-Conventional Formatting, while GPT-OSS-120B underperforms on Deliberately Incorrect Answers. These type-level patterns are diagnostic — they tell us not just that a model has cognitive inertia but which specific conventions it is most rigid about. This enables targeted improvement: a model that struggles with Deliberately Incorrect Answers might benefit from training data that explicitly includes examples of intentionally wrong responses (with appropriate caveats), while a model that struggles with formatting inversions needs different interventions.

The taxonomy also reveals an important asymmetry: not all training conventions are equally "sticky." The fact that Code without Comments has the largest number of samples (198, per Table 2) and Counter-Conventional Formatting has the fewest (82) is partly an artifact of generation difficulty, but the performance patterns suggest a deeper principle: conventions that are consistently reinforced across diverse training examples (like "be factually correct," which applies to every knowledge-based response) may actually be easier to override than conventions that are inconsistently or implicitly reinforced (like "accept the user's premise," which is rarely stated explicitly but is pervasive in assistant-style training data). This is a hypothesis the paper doesn't fully explore but that the taxonomy enables others to investigate.

This contribution is incremental in form but fundamental in implication. Taxonomies are standard benchmark-construction practice, and organizing evaluation items by category is not novel. But the specific organizing principle — classification by which training convention is inverted — is theoretically motivated and empirically validated in a way that transforms the taxonomy from an administrative convenience into a diagnostic instrument. It's the difference between organizing a medical test by body part (standard and unsurprising) versus organizing it by underlying disease mechanism (diagnostically powerful). Inverse IFEval does the latter for instruction-following brittleness.

Innovation 3: The Thinking Mechanism as a Mitigator of Cognitive Inertia

The paper's third contribution is the empirical discovery that reasoning mechanisms — chain-of-thought, extended thinking budgets — partially mitigate cognitive inertia, and that this mitigation is detectable through the Inverse IFEval lens in ways that conventional benchmarks would miss.

This finding emerges from multiple convergent lines of evidence. First, the consistent performance gap between thinking and non-thinking variants of the same base model: Qwen3-235B-A22B-Thinking and Qwen3-30B-A3B-Thinking outperform their non-thinking Instruct counterparts across the board (Figure 4a), and the gap is substantially larger on Inverse IFEval than it would be on a conventional benchmark — as evidenced by the dramatic ranking drop of non-thinking models between IFEval and Inverse IFEval (Figure 6). Second, the comparison between Gemini-2.5-Pro (full thinking) and Gemini-2.5-Flash (reduced thinking budget) shows the same pattern: reduced thinking correlates with reduced counter-intuitive instruction following. Third, the AdaCoT experiment (Figure 4b) shows that auto-thinking mode — where the model adaptively decides whether to reason — can actually underperform non-thinking mode on the Chinese version, suggesting that the model's metacognitive assessment of when to think is itself biased by training conventions and may fail to trigger reasoning precisely when it's most needed.

The significance of this finding extends beyond the simple observation that "thinking helps." It suggests a mechanism for why thinking helps and, by extension, a potential mechanism for cognitive inertia itself. The authors' interpretation (Section 3.2.1) is that "thinking enables models to reflect on knowledge acquired during SFT" — essentially, the reasoning process creates a deliberative step between instruction comprehension and response generation where training conventions can be consciously overridden. In non-thinking mode, the model's response is generated more directly from the instruction, and training conventions exert their influence as defaults that the model doesn't pause to question.

This interpretation is consistent with broader findings about reasoning models exhibiting improved robustness and generalization. The paper connects the dots in a specific way: cognitive inertia is a form of reflexive response generation where training patterns are applied automatically, while thinking mechanisms introduce a deliberative override that can suppress those automatic patterns. This framing has practical implications: it suggests that even without retraining, simply running models in thinking mode with sufficient compute budget can partially recover counter-intuitive instruction-following capability — a finding corroborated by the test-time compute scaling results (Figure 15), where Best-of-32 pushes four of five models near or above 90.

However, thinking is not a complete solution. Even the best thinking models (o3-high, Gemini-2.5-Pro) show substantial performance variation across instruction types in Figure 5, and none achieves near-ceiling performance on the hardest types like Question Correction. The thinking mechanism mitigates but does not eliminate cognitive inertia. This qualified finding — thinking helps, but doesn't solve — is more valuable than either extreme claim ("thinking solves everything" or "thinking doesn't matter for instruction following") because it precisely characterizes what additional post-training interventions would need to accomplish.

This contribution is incremental in its mechanism (thinking-vs-non-thinking comparisons are well-established) but novel in its diagnostic application. The paper doesn't invent the idea of comparing thinking and non-thinking models — that's standard practice. What's novel is using that comparison to measure cognitive inertia: the thinking/non-thinking gap on Inverse IFEval quantifies how much of the model's instruction-following failure is due to reflexive training-pattern application versus fundamental inability. A model that performs well in thinking mode but poorly in non-thinking mode "knows" how to follow the instruction but defaults to training conventions without deliberation; a model that performs poorly even in thinking mode fundamentally lacks the capability. This diagnostic decomposition of failure modes — separating reflexive failures from capability failures — is enabled by Inverse IFEval's design and represents a genuine methodological advance.

Innovation 4: Verifier Over-Optimization as a Benchmark Construction Insight (The 98% Judging Accuracy)

While the 98% judging accuracy is presented as a technical achievement in Section 2.5, the underlying insight is more conceptually significant: evaluating counter-conventional instruction following requires a substantially more sophisticated judging framework than evaluating conventional instruction following, and the standard approach of using a single off-the-shelf judge model (which works at ~88% accuracy) is insufficient.

The conceptual move here is recognizing that the judge model itself exhibits cognitive inertia. A standard LLM-as-a-Judge setup — give a strong model the instruction, the response, and a scoring rubric — is prone to the same training-convention biases as the model being evaluated. When asked to judge whether a response violates "no bullet points," a judge model trained on SFT data may be systematically lenient because its own training has taught it that bullet points are good practice. When asked to judge whether an answer is "deliberately incorrect," a judge model may be systematically harsh because its training has taught it that incorrectness is always bad. The initial 88% accuracy reflects not just random noise but systematic bias in the judge's evaluation — it rates convention-violating responses as more compliant than they actually are when the violation aligns with its own training preferences, and rates convention-following responses as less compliant when the convention happens to be one it's been trained to avoid.

The three optimization strategies — adaptive model selection, type-specific templates, and enhanced system prompts with exemplars — can be understood as interventions to suppress the judge model's own cognitive inertia. The adaptive model matrix selects judges that are least biased for each specific task type. The template optimization structures the evaluation prompt to make the scoring criteria unambiguous in ways that overcome the judge's default assumptions. The system prompt enhancement provides explicit counter-examples that teach the judge to recognize convention violations it might otherwise overlook. These aren't just engineering optimizations; they're methodological moves that acknowledge and address a previously unrecognized problem in automated evaluation: that judge models, like the models they evaluate, have training-induced biases that must be actively counteracted.

This insight has implications beyond Inverse IFEval. Any benchmark that tests model behavior on out-of-distribution tasks faces a judging problem: the judge model's training distribution influences what it considers "good" behavior. If the task being tested is OOD relative to the judge's training, the judge may be an unreliable evaluator. The paper's solution — adaptive judge selection and type-specific calibration — provides a template for addressing this problem in future benchmarks.

The 10-percentage-point improvement from 88% to 98% is not merely a reliability gain; it's the difference between a benchmark that can detect genuine performance differences between models and one where evaluation noise dominates. At 88% accuracy, the judge's error rate (12%) is comparable to or larger than the performance gaps between many models in Table 3, making rankings unreliable. At 98%, the noise floor is low enough that measured differences reflect genuine capability differences. This is the benchmarking equivalent of improving a telescope's resolution from blurry to sharp — the instrument itself becomes capable of seeing phenomena that were previously washed out by noise.

This contribution is methodological rather than empirical — it advances the practice of benchmark construction rather than the understanding of model capabilities — but it's fundamental to the benchmark's validity. Without it, the paper's core claims about cognitive inertia would rest on a judging framework that itself exhibits the very biases being measured.

5. Experimental Analysis

Evaluation Methodology

Dataset. Inverse IFEval, a custom-constructed benchmark of 1012 high-quality questions (506 Chinese, 506 English) spanning 23 knowledge domains and eight adversarial instruction types. The dataset was built through a five-stage human-in-the-loop pipeline (Section 2.2) and is publicly available on Hugging Face. No train/dev/test split exists — the entire dataset is used for evaluation only, as this is a benchmark, not a training corpus.
Base model(s). Fifteen models spanning both closed-source and open-source families are evaluated (Section 3, Table 3). Closed-source: o3-high, o4-mini, o3-mini, GPT-5-high, GPT-4.1 (OpenAI); Gemini-2.5-Pro, Gemini-2.5-Flash (Google); Claude-4-Opus, Claude-4-Sonnet (Anthropic); Doubao-Seed-1.6-thinking (ByteDance); StepFun-R1-V-Mini. Open-source: GPT-OSS-120B, Qwen3 series (235B-A22B and 30B-A3B, in both Thinking and Instruct variants), GLM-4.5, Kimi-K2, DeepSeek-R1, DeepSeek-V3, DeepSeek-V3.1. The selection spans a wide capability range — from frontier reasoning models (o3-high) to fine-tuned non-reasoning models (Qwen3-Instruct variants) — specifically chosen to test whether cognitive inertia manifests differently across model families, scales, and training paradigms. The inclusion of "Flash" variants (reduced thinking budget) and both thinking/non-thinking versions of the same base model (Qwen3) enables direct measurement of how reasoning mechanisms affect counter-intuitive instruction following.
Metrics. The primary metric is instruction-following accuracy — the percentage of questions for which a model's response is judged compliant with the specified inverse instruction. Scoring is performed automatically via the optimized LLM-as-a-Judge framework (Section 2.5), which achieved 98% accuracy against human expert judgments. Results are reported in Table 3 as overall accuracy (averaged across all eight instruction types) for both English and Chinese versions separately, as well as per-type accuracy for selected models (Figure 5). The Best-of-N metric (Section 8.1, Figure 15) reports the maximum score achieved among N independent samples.
Baselines. The paper does not define explicit baseline methods in the traditional sense (since it's evaluating models, not proposing new training techniques). However, there are two implicit comparison frameworks. First, IFEval performance (Zhou et al., 2023) serves as the convention-aligned baseline — models' rankings on IFEval are compared against their rankings on Inverse IFEval (Figure 6) to measure rank instability and thus demonstrate that the two benchmarks measure distinct capabilities. Second, non-thinking mode serves as a baseline against thinking mode within the same model family (Figure 4a) to isolate the effect of reasoning mechanisms on counter-intuitive instruction following. The auto-thinking mode (AdaCoT, Lou et al., 2025) provides an additional baseline that tests whether adaptive reasoning-triggering can match explicit thinking (Figure 4b).
Generation budget / compute accounting. The paper does not standardize generation compute across models — each model is evaluated with its default generation parameters. For the Best-of-N analysis (Section 8.1, Figure 15), compute is measured in number of independent samples: N = 1, 16, and 32, where the best-scoring sample among N is selected. This is a standard compute scaling framework where "more compute" means "more parallel samples," but no per-token or per-FLOP accounting is provided. The thinking/non-thinking comparison (Figure 4a) implicitly involves different compute budgets since thinking models generate more tokens, but this cost is not quantified or controlled for — the comparison measures capability differences, not compute-normalized efficiency.
Cross-validation / statistical protocol. The paper does not report confidence intervals, statistical significance tests, or cross-validation protocols for the benchmark results. The evaluation framework's reliability is established through the 98% judge accuracy validation (Section 2.5), but the benchmark scores themselves (Table 3) are presented as point estimates without uncertainty quantification. The dataset construction process does incorporate cross-validation mechanisms during seed data construction (multi-dimensional cross-functional review requiring unanimous agreement) and expert review (multi-pilot rubric validation), but these are quality control measures for the benchmark itself, not statistical protocols for the evaluation results.

Main Quantitative Results

Overall Benchmark Performance (Table 3)

The central results table (Table 3) presents accuracy scores for all 15 models on both the English and Chinese versions of Inverse IFEval, broken down by the eight instruction types. The headline findings are as follows.

o3-high achieves the best overall performance, though the paper does not report the exact numerical average across all eight types in the main text — it states that o3-high "achieves the best performance on the Inverse IFEval, with o3-mini and GPT-5-high following closely behind" (Section 3.1). The per-type values in Table 3 provide the granular data, but exact overall averages require computation from the table values.

Fine-tuned non-thinking models perform dramatically worse than their thinking counterparts. The Qwen3 family provides the clearest comparison: Qwen3-235B-A22B-Instruct (non-thinking) and Qwen3-30B-A3B-Instruct both show "poor" performance, while Qwen3-235B-A22B-Thinking and Qwen3-30B-A3B-Thinking achieve substantially higher scores (Section 3.1, Table 3). The paper does not report exact gap magnitudes but describes the Instruct variants' performance as "poor, indicating that the dataset effectively meets its intended purpose" — meaning the benchmark successfully discriminates between models that can and cannot override training conventions. The "Flash" series models (Gemini-2.5-Flash) with reduced thinking budgets show lower performance than their full-thinking counterparts (Gemini-2.5-Pro), reinforcing the thinking-dependence finding.

Larger models within the same family perform better. The Qwen3 series demonstrates the parameter-scaling relationship: Qwen3-235B-A22B-Thinking outperforms Qwen3-30B-A3B-Thinking (Section 3.1, Table 3), consistent with the general finding that scale improves instruction-following capability. However, the paper does not quantify whether this scaling benefit is proportionally larger on Inverse IFEval than on conventional benchmarks — a comparison that would reveal whether scale specifically helps with cognitive inertia or provides uniform improvement.

The "Flash" reduced-thinking models underperform full-thinking variants. Gemini-2.5-Flash shows lower performance than Gemini-2.5-Pro (Section 3.1, Table 3), and this gap is attributed to the reduced thinking budget rather than architectural differences — the Flash designation specifically indicates a model optimized for speed with less compute allocated to reasoning.

Thinking vs. Non-Thinking Mechanism Analysis (Figure 4)

Figure 4 provides dedicated analysis of how reasoning mechanisms affect Inverse IFEval performance, with two complementary comparisons.

Figure 4a: Qwen3 thinking vs. non-thinking comparison. The paper reports a "consistent performance drop in the non-thinking mode compared to the thinking mode" across the Qwen3 family when evaluating average overall scores on both English and Chinese versions (Section 3.2.1). The authors attribute this drop to "a preference for common instructions induced during the SFT phase" — essentially, non-thinking models apply training conventions reflexively, while thinking models can deliberate and override those conventions. The consistency of this finding across model sizes (both 30B and 235B variants) suggests the effect is robust to scale.

Figure 4b: AdaCoT auto-thinking mode evaluation. The Doubao-Seed-1.6-thinking model is evaluated in three configurations: thinking mode, non-thinking mode, and auto mode (where the model adaptively decides whether to engage chain-of-thought reasoning, using the AdaCoT method). The results (Section 3.2.1) show that "thinking mode achieves the optimal performance, whereas the non-thinking mode yields the poorest outcomes" — consistent with the Qwen3 finding. However, a critical and non-obvious finding emerges: "in the Chinese version, the auto-thinking mode performs even worse than the non-thinking mode." This is a notable negative result. It suggests that the model's metacognitive assessment of when to think — its adaptive reasoning trigger — is itself biased by training conventions and may systematically fail to engage thinking precisely when it's most needed (i.e., on counter-intuitive Chinese instructions). The authors interpret this as evidence that "the auto-thinking mode still needs further optimization to better suit the Chinese language context."

Per-Instruction-Type Analysis (Figure 5)

Figure 5 presents radar charts for eight selected models (o3-high, Gemini-2.5-Pro, Claude-4-Opus-Thinking, Doubao-Seed-1.6-thinking, Qwen3-235B-A22B-Thinking, GLM-4.5, GPT-OSS-120B, DeepSeek-R1) showing their accuracy broken down by the eight instruction types on the English version.

Counterfactual Answering is consistently the easiest type. Across all eight models shown, Counterfactual Answering (CA) scores exceed 65, with most models achieving substantially higher values (Section 3.2.2). This is a non-obvious finding: one might expect that overriding factual knowledge — arguably the most fundamental training objective — would be exceptionally difficult, yet models manage it relatively well. The paper does not provide a mechanistic explanation for this, but it suggests that counterfactual reasoning may be represented differently in model training than formatting or stylistic conventions.

Question Correction is consistently the hardest type. The paper reports that "half of the models score below 30" on Question Correction (QC) in Figure 5 (Section 3.2.2). This type requires models to identify errors in the user's question and correct them — essentially, to be critical of the user rather than accommodating. The difficulty of this type reveals that the training convention of "accept the user's framing and answer helpfully" is among the most deeply ingrained behaviors.

Model-specific weakness profiles emerge. DeepSeek-R1 is "notably weaker on Intentional Textual Flaws and Counter-Conventional Formatting" (Section 3.2.2), while GPT-OSS-120B "underperforms on Deliberately Incorrect Answers." These type-level patterns are diagnostic — they identify specific training conventions that particular models are most rigid about, enabling targeted improvement strategies. The paper frames these patterns as providing "insights for subsequent optimization."

o3-high shows the most balanced profile but still exhibits substantial variation across types. Even the best model doesn't achieve uniform high performance — the gap between its strongest and weakest types in Figure 5(a) demonstrates that cognitive inertia affects all models, just to different degrees and in different patterns.

IFEval vs. Inverse IFEval Rank Comparison (Figure 6)

Figure 6 provides the most direct evidence that Inverse IFEval measures a fundamentally different capability than conventional instruction-following benchmarks. The figure shows model rankings on IFEval compared to Inverse IFEval, split into models whose rankings improved (Figure 6a) and models whose rankings declined (Figure 6b).

Top models are stable: o3-high, Gemini-2.5-Pro, and o4-mini-high "consistently achieve top ranks in both benchmarks" (Section 3.2.3). This stability at the top of the ranking suggests that frontier reasoning models possess genuine instruction-following flexibility — they can handle both convention-aligned and convention-contradicting instructions, though not equally well.

Non-thinking models show dramatic rank declines. The most striking example: Qwen3-235B-A22B-Instruct ranks 5th on IFEval but drops to 15th on Inverse IFEval (Section 3.2.3, Figure 6b) — a ten-position collapse. This is the empirical signature of cognitive inertia: a model that excels at convention-aligned instruction following (5th place) becomes essentially the worst model in the set when instructions contradict training conventions (15th place). The paper emphasizes that "the remaining models exhibit substantial rank variations between the two benchmarks, suggesting instability under non-conventional instructions."

Non-thinking models consistently rank much lower on Inverse IFEval than on IFEval — this pattern is sufficiently strong that the paper calls it out explicitly (Section 3.2.3). The ranking inversion is not random noise; it's systematic and correlates with whether the model has a reasoning mechanism. This is the paper's central empirical claim: instruction-following capability as measured by conventional benchmarks does not generalize to counter-conventional instructions, and the discrepancy is large enough to fundamentally reorder model rankings.

Test-Time Compute Scaling (Figure 15)

Figure 15 (Section 8.1) examines whether increasing inference compute through Best-of-N sampling can improve Inverse IFEval performance. The paper evaluates Best-of-1, Best-of-16, and Best-of-32, computing the "average overall score of both English and Chinese versions" for five models (the specific models are not named in Section 8.1 but the figure shows five lines).

Best-of-N scaling provides substantial gains. As N increases, "the models achieve consistently higher scores" (Section 8.1). The magnitude is significant: "when N=32, four out of the five models approach or surpass a score of 90." This indicates that the benchmark items are within the models' capability — with sufficient sampling, models can produce compliant responses — but that single-sample generation often defaults to training-conventional behavior.

The paper interprets this as evidence that post-training can likely close the gap. The authors state that the Best-of-N results "further indicate that with appropriate post-training, substantial performance gains on our benchmark are likely achievable" (Section 8.1). This is a forward-looking claim: the fact that correct responses exist in the model's output distribution (they can be found through rejection sampling) suggests that the capability is latent and could be surfaced through targeted training rather than expensive test-time search.

The scaling curves are not extrapolated. The paper only tests up to N=32, providing three data points per model. The shape of the scaling relationship (linear, logarithmic, saturating) cannot be determined from this data, and the paper doesn't attempt to predict performance at higher N values.

Cross-Lingual Performance Patterns (Section 8.2)

The parallel Chinese-English construction enables comparison of how language affects counter-intuitive instruction following.

Overall consistency across languages. The paper reports that "models demonstrate consistent performance across both languages" and that "those who perform well in English tend to show similar success in Chinese, with their scores in both versions typically being close" (Section 8.2). This suggests that cognitive inertia is primarily a language-agnostic phenomenon — the training conventions being overridden are conceptual patterns, not surface-level linguistic patterns.

Language-specific exceptions reveal training data biases. Two models deviate from this pattern: "GLM-4.5 and Qwen3-235B-A22B-Thinking perform significantly better in the Chinese version than in the English version" (Section 8.2). The paper attributes this to the fact that "the Chinese community contributes these two models" — meaning their training data is skewed toward Chinese, and this skew affects their ability to follow counter-conventional instructions in English. This is a noteworthy finding because it demonstrates that cognitive inertia has a language-specific component: models overfit not just to formatting and stylistic conventions but also to the language distribution of their training data, making them more flexible in their dominant training language even on tasks that aren't inherently language-specific.

Ablation Studies and Robustness Checks

The paper's ablation-like analyses are embedded throughout the experimental design rather than presented in a dedicated ablation section. The key controlled comparisons are as follows.

Thinking vs. non-thinking within the same model family (Qwen3, Figure 4a). This is the paper's central controlled experiment: comparing thinking and non-thinking variants of the same base model to isolate the effect of the reasoning mechanism while holding architecture, training data, and scale constant. The consistent performance drop in non-thinking mode (Section 3.2.1) demonstrates that the reasoning mechanism itself — not model scale, architecture, or training data quality — is responsible for improved counter-intuitive instruction following. This finding is robust across two model sizes (30B and 235B), providing within-family replication.

Full-thinking vs. reduced-thinking budget (Gemini-2.5-Pro vs. Flash, Table 3). By comparing Gemini-2.5-Pro (full thinking budget) against Gemini-2.5-Flash (reduced thinking budget, otherwise similar architecture), the paper isolates the effect of thinking depth or duration as distinct from the presence/absence of a thinking mechanism. The performance gap between these two variants (Section 3.1) shows that more thinking compute produces better counter-intuitive instruction following — a dose-response relationship that strengthens the causal interpretation.

Explicit thinking vs. adaptive thinking triggering (AdaCoT, Figure 4b). Comparing the auto-thinking mode against both explicit thinking and non-thinking modes reveals that adaptive reasoning triggering — where the model decides when to think — can underperform even non-thinking mode in some contexts (specifically, Chinese instructions). This is a critical negative result: the model's metacognitive judgment about when deliberation is needed is itself biased by training, and that bias can lead to worse performance than never thinking at all. This finding suggests that simple "always think" is more robust than "think when needed" for counter-conventional instructions, which has practical implications for deployment strategies.

Single-sample vs. Best-of-N sampling (Figure 15). The comparison of N=1, 16, and 32 reveals how much of the performance gap is due to the model never producing a compliant response versus sometimes producing one but not reliably. The finding that Best-of-32 pushes four of five models near or above 90 suggests that the models' failure mode is primarily one of preference (they prefer training-conventional outputs) rather than capability (they are incapable of producing compliant outputs). This is a crucial diagnostic distinction: preference failures can potentially be addressed through better alignment or prompting, while capability failures would require fundamentally different training.

Cross-lingual robustness (Section 8.2, Table 3). The parallel evaluation in English and Chinese tests whether cognitive inertia is language-specific or language-general. The finding of overall consistency with specific exceptions (GLM-4.5, Qwen3-235B-A22B-Thinking better in Chinese) suggests that while the underlying phenomenon is language-agnostic, training data language distribution modulates its severity. This is effectively an ablation of language as a factor — if cognitive inertia were purely about surface-level linguistic patterns, we would expect larger and more systematic cross-lingual differences than what is observed.

Judge accuracy optimization (Section 2.5). The improvement from 88% to 98% judging accuracy through three strategies (adaptive model selection, template optimization, system prompt enhancement) is itself a multi-factor ablation, though the paper does not report the individual contribution of each strategy. The 10-percentage-point improvement demonstrates that standard LLM-as-a-Judge approaches are insufficient for evaluating counter-conventional instruction following — a finding with implications for benchmark design beyond this paper.

Critical Assessment

How well do the experiments support the central claim that models exhibit cognitive inertia?

The paper's core claim — that LLMs exhibit cognitive inertia, failing to follow instructions that conflict with training conventions — is strongly supported by the evidence, but the characterization of why this happens is more speculative than the experimental design can fully validate.

The strongest evidence comes from the rank inversion between IFEval and Inverse IFEval (Figure 6). The fact that Qwen3-235B-A22B-Instruct drops from 5th to 15th place cannot be explained by Inverse IFEval simply being "harder" — difficulty alone would produce proportional drops across models, preserving rankings. The rank reordering is the empirical signature of a different underlying capability being measured. This is methodologically clean evidence that Inverse IFEval captures something IFEval does not.

The thinking/non-thinking gap (Figure 4) provides additional convergent evidence. If cognitive inertia were not a real phenomenon, there would be no systematic reason for thinking mechanisms to disproportionately benefit Inverse IFEval performance relative to IFEval performance — yet this is exactly what the data show. The within-family comparison (Qwen3 thinking vs. non-thinking) is particularly strong because it controls for architecture, scale, and training data, isolating the reasoning mechanism as the causal factor.

However, the paper's mechanistic interpretation — that this is specifically about "cognitive inertia" from SFT conventions — is supported more by the design logic than by direct experimental manipulation. The paper never runs a controlled experiment where the same model is fine-tuned on different data distributions to show that different training conventions produce different patterns of rigidity. The evidence is entirely correlational: the benchmark was designed to invert SFT conventions, and models fail on it. But we cannot rule out alternative explanations: perhaps the failures arise from general instruction complexity rather than specifically from convention conflict; perhaps models fail because the instructions are simply unusual or poorly represented in pretraining, not because of SFT specifically; perhaps the failures reflect fundamental limitations in language understanding rather than training-induced biases. The paper's argument for the SFT-specific mechanism rests on the plausibility of the connection between annotation practices and model behavior, not on experimental isolation of that mechanism.

How well do the experiments support the claim that thinking mechanisms mitigate cognitive inertia?

This claim is well-supported by the within-family comparisons (Qwen3 variants, Gemini variants, AdaCoT), which provide the necessary control to attribute differences to the thinking mechanism rather than confounds. The dose-response relationship — more thinking budget → better performance — strengthens the causal interpretation.

However, the magnitude of the mitigation is not fully characterized. The paper shows that thinking helps, but doesn't quantify how much of the gap between Inverse IFEval and IFEval performance is closed by thinking. We know thinking models outperform non-thinking models on Inverse IFEval, but we don't know whether thinking models' Inverse IFEval scores approach their IFEval scores (suggesting thinking essentially solves cognitive inertia for those models) or remain substantially lower (suggesting thinking only partially mitigates). The per-type analysis in Figure 5 suggests partial mitigation — even o3-high shows substantial type-level variation — but a direct comparison of thinking models' IFEval-vs-Inverse IFEval gaps versus non-thinking models' gaps would be more diagnostic and is not provided.

The AdaCoT negative result (auto-thinking underperforming non-thinking on Chinese, Figure 4b) is intriguing but rests on a single model in a single configuration (Doubao-Seed-1.6-thinking). Replication across models would be needed to establish whether this is a general property of adaptive reasoning or a model-specific quirk. Without replication, the finding is suggestive but not conclusive.

How well do the experiments support the claim that Inverse IFEval measures a distinct evaluation dimension?

This claim is strongly supported by the ranking inversion evidence (Figure 6) and the per-type diagnostic patterns (Figure 5). The fact that models' relative standing changes dramatically between IFEval and Inverse IFEval — and that this change is systematic (non-thinking models drop, top reasoning models stay stable) — provides convergent validity that the benchmark captures something real and distinct.

However, there is a missing comparison that would substantially strengthen this claim: the paper never reports models' IFEval scores directly or provides a scatter plot of IFEval vs. Inverse IFEval performance. Without seeing the actual scores on both benchmarks, readers cannot assess the magnitude of the performance drop for individual models or determine whether the two benchmarks are orthogonal (near-zero correlation) or simply differentially difficult (strong correlation but different intercepts and slopes). The ranking data alone doesn't distinguish between these possibilities.

Additionally, the paper doesn't establish discriminant validity — does Inverse IFEval correlate with other established benchmarks that it shouldn't correlate with, or fail to correlate with benchmarks it should? A multitrait-multimethod analysis would be ideal but is beyond the scope of an introductory benchmark paper. At minimum, reporting actual IFEval scores alongside Inverse IFEval scores would let readers compute the correlation themselves.

What genuine weaknesses exist in the experimental design?

No confidence intervals or statistical testing. Table 3 reports point estimates for 15 models across 8 types in 2 languages (240+ data points) without any measure of uncertainty. Given the test set size (1012 questions, but only ~126 per type on average), sampling error could be non-trivial. The paper cannot distinguish a genuine 2-point performance difference from noise, making fine-grained model comparisons unreliable. This is a significant omission for a benchmark paper whose purpose is to enable precise model comparisons.

The 98% judging accuracy claim lacks detail. The paper reports this figure (Section 2.5) without specifying the size of the validation set, the inter-annotator agreement among the human experts providing ground truth, whether accuracy is exact-score matching or within-tolerance, and whether it's computed per-type or aggregated. Without these details, readers cannot assess the reliability of the claim or determine whether 98% accuracy holds uniformly across all instruction types (it almost certainly doesn't — some types are harder to judge than others).

No human performance baseline. The paper never reports how well humans perform on Inverse IFEval. This matters because if humans also struggle with these instructions — if, for example, people find it difficult to produce intentionally flawed text without accidentally slipping into correct patterns — then the benchmark may be measuring a genuinely difficult cognitive task rather than specifically LLMs' training-induced rigidity. A human baseline would contextualize model performance and help distinguish "this is hard for LLMs because of SFT" from "this is hard for any intelligent agent."

The domain distribution is highly skewed. Figure 3 shows that Computer Science accounts for 20.2% of the dataset, with the remaining ~80% distributed across 22 other domains. Per-type sample sizes (Table 2) vary from 82 (Counter-Conventional Formatting) to 198 (Code without Comments) — a >2× range. These imbalances mean that overall accuracy scores are weighted toward certain domains and types, and domain-specific or type-specific conclusions based on small-N subsets may be unreliable.

Single benchmark, no external validation. All claims about cognitive inertia rest on a single instrument. There is no correlation analysis with other potential measures of training-convention rigidity (e.g., performance on tasks requiring violation of Gricean maxims, susceptibility to sycophancy, resistance to prompt injection). Without external validation, the interpretation that Inverse IFEval specifically measures "SFT convention overfitting" rather than some broader construct (like general instruction-following difficulty or unusual-task flexibility) remains an inference from design rather than an empirically established fact.

The test-time compute scaling experiment (Figure 15) is underpowered. Only five models are tested (and not explicitly named in Section 8.1), only three N values are used (1, 16, 32), and only average scores across both languages are reported. The shape of the scaling curve cannot be determined from three points, and the claim that Best-of-32 "approaches or surpasses a score of 90" for four of five models cannot be verified without knowing which models and what their exact scores are.

The cross-lingual comparison is underanalyzed. The paper notes that GLM-4.5 and Qwen3-235B-A22B-Thinking perform better in Chinese (Section 8.2) and attributes this to training data language distribution, but provides no analysis of whether this language effect is uniform across instruction types or concentrated in particular types. It's possible that certain types of cognitive inertia (e.g., formatting conventions) are language-agnostic while others (e.g., factual correctness norms) are affected by language-specific training data differences — but the paper doesn't investigate this.

What experiments would have strengthened the paper but were not run?

Direct manipulation of training data. The strongest possible evidence for the SFT-convention mechanism would be an experiment where the same base model is fine-tuned on two datasets — one with standard conventions, one where those conventions are deliberately violated — and then tested on Inverse IFEval. If the model trained on convention-violating data performs better, the causal link between SFT conventions and cognitive inertia would be experimentally established. This experiment is expensive but feasible, and its absence means the paper's central mechanistic claim remains correlational.

Comparison of pretrained-only vs. instruction-tuned models. The paper attributes cognitive inertia to SFT specifically, but evaluates only post-trained models. Comparing a pretrained base model (no SFT) against its instruction-tuned counterpart on Inverse IFEval would isolate the contribution of SFT to the rigidity. If the base model performs better, SFT is directly implicated. If the base model performs similarly or worse, the rigidity may arise from pretraining conventions rather than SFT.

Correlation analysis with IFEval scores. Reporting the IFEval scores that produce the rankings in Figure 6 would allow computation of the correlation between IFEval and Inverse IFEval performance, which would quantify how much these benchmarks measure the same versus different constructs. A low correlation would strongly support the distinct-dimension claim; a moderate-to-high correlation with rank shifts driven by a few outliers would weaken it.

Human evaluation to complement LLM-as-a-Judge. Even with 98% claimed accuracy, human evaluation on a subset of model responses would provide an important validity check, particularly for edge cases where the judge model might share the cognitive biases of the models being evaluated. The paper validates the judge against human judgments during development but doesn't report ongoing human validation of final benchmark results.

Fine-grained error analysis with quantitative breakdowns. The error analysis in Appendix 9 (Figures 16-19) is purely qualitative — selected examples of model failures. A quantitative breakdown of types of errors would be more diagnostic. For instance, on "Code without Comments," what fraction of failures involve the model adding comments anyway versus providing no code at all versus providing code with comments only in certain locations? This would reveal whether failures are due to specific sub-conventions being harder to override or due to general instruction misunderstanding.

Evaluation of models at different temperatures. The Best-of-N scaling experiment (Figure 15) samples at the default temperature. Varying sampling temperature would reveal whether cognitive inertia is a deterministic phenomenon (models always default to training conventions) or a probabilistic one (models sometimes sample convention-violating outputs at higher temperatures), which has different implications for mitigation strategies.

Overall, the experiments convincingly demonstrate that something real and important is being measured — models genuinely struggle with counter-conventional instructions, and this struggle is not captured by existing benchmarks. The paper succeeds in its primary purpose: establishing a diagnostic instrument and demonstrating that it reveals capability gaps the field was not measuring. However, the specific attribution of these failures to SFT-induced cognitive inertia rests more on the benchmark's design logic than on direct experimental manipulation, and several important validation steps (human baselines, confidence intervals, discriminant validity analysis) remain incomplete. The paper provides a strong foundation but leaves substantial work for future studies to fully characterize the phenomenon it identifies.

6. Limitations and Trade-offs

Limitation 1: No Evidence That Failures Are Specifically Caused by SFT Rather Than Pretraining or General Unfamiliarity

The assumption or constraint. The paper's central mechanistic claim is that models fail on Inverse IFEval because supervised fine-tuning (SFT) on idealized annotation data creates "cognitive inertia" — a rigid adherence to training conventions that persists even when explicitly countermanded. The entire benchmark design is predicated on this causal story: each of the eight instruction types inverts a convention that the authors observed in SFT annotation practices, and model failures are interpreted as evidence that SFT specifically is the source of the rigidity.

However, the paper never experimentally isolates SFT as the causal mechanism. It evaluates only post-trained models (which have undergone both pretraining and SFT) and never compares pretrained-only models against their instruction-tuned counterparts on Inverse IFEval. It also never manipulates the SFT data distribution to show that different annotation conventions produce different patterns of rigidity — a direct test of the causal claim.

The consequence. Without experimental isolation, alternative explanations for model failures remain equally plausible:

Pretraining conventions may be the primary source of rigidity. LLMs are pretrained on vast corpora of human-written text that overwhelmingly follows standard formatting, correctness, and stylistic norms. The conventions that Inverse IFEval inverts — bullet points, grammatical correctness, code comments, factual accuracy — are pervasive in internet text, not just in SFT data. Models may have internalized these conventions during pretraining, with SFT merely reinforcing rather than creating them.
The failures may reflect general unfamiliarity rather than convention conflict. Inverse IFEval instructions are deliberately unusual — "produce an unstructured wall of text with no paragraph breaks" is not something models encounter frequently in any training data, SFT or otherwise. Models may fail simply because they have little experience with such directives, not because they are specifically "stuck" on SFT conventions. This would make the benchmark a measure of out-of-distribution generalization rather than SFT-induced rigidity — still valuable, but a different construct.
The failures may reflect fundamental limitations in instruction understanding. Some of the instruction types, particularly Question Correction (which requires identifying errors in the user's question) and Instructional Induction (which requires inferring latent patterns from examples), are cognitively demanding tasks that might challenge any language system regardless of training history. Human performance on these tasks is unknown — and if humans also struggle, the failures may reflect task difficulty rather than training-induced bias.

This causal ambiguity matters for remediation. If rigidity comes from SFT, the solution is to modify SFT data or training procedures. If it comes from pretraining, the solution is fundamentally different (and much harder). If it comes from general unfamiliarity, the solution is to include more diverse and unusual instructions in training. If it comes from fundamental capability limits, no amount of data modification will help. The paper's attribution to SFT — reflected in its framing, its abstract ("struggling to follow instructions that conflict with the standardized patterns learned during supervised fine-tuning"), and its conclusion ("mitigate cognitive inertia, reduce overfitting to narrow patterns") — directs remediation efforts toward SFT specifically, but this is a hypothesis rather than an established finding.

What evidence exists in the paper. The paper provides no experiment that isolates SFT as the causal mechanism. The evidence is entirely design-based: the authors describe observing SFT annotation conventions (Section 2.1, Section 2.2 Stage 1), they invert those conventions to create the eight instruction types, and they observe that models fail. This establishes correlation (models fail on instructions that invert SFT conventions) but not causation (SFT specifically causes the rigidity). The strongest indirect evidence is that non-thinking models — which the authors argue apply SFT conventions more reflexively — perform worse than thinking models (Figure 4a). However, thinking models also have more compute and generate more tokens, so the performance gap could reflect general compute scaling rather than specifically overcoming SFT-induced rigidity.

The paper's discussion of SFT as the mechanism appears primarily in Section 1, Section 2.1, and the Conclusion, but these are framing statements rather than experimentally tested claims. The introduction states that models "struggle when tasked with instructions that explicitly deviate from their training norms" and links this to SFT annotation practices, but the experiments never manipulate training norms to test this link.

Mitigation status. The paper does not acknowledge this limitation. It consistently frames the findings in terms of SFT-induced cognitive inertia without noting alternative explanations or calling for experiments to isolate the mechanism. The conclusion recommends "developing methods that mitigate cognitive inertia, reduce overfitting to narrow patterns," implicitly directing future work toward SFT-focused interventions without establishing that SFT is the primary source of the problem. A pretrained-only baseline — which would be relatively straightforward to include — is absent from the experimental design.

Limitation 2: The Cost of Computing Difficulty Estimates Is Not Accounted for in Any Headline Efficiency Claim

The assumption or constraint. The paper makes no claims about efficiency or compute-optimality — unlike the test-time compute paper, Inverse IFEval does not propose a method whose practical deployment depends on compute cost tradeoffs. However, the benchmark's construction and evaluation pipeline involve substantial compute costs that are neither quantified nor discussed, and these costs affect the benchmark's practical adoptability as a diagnostic tool.

Specifically, the paper's optimized LLM-as-a-Judge framework requires running multiple judge models (the "adaptive, optimal accuracy judge model matrix" that selects the best judge per instruction type) with enhanced system prompts and exemplar demonstrations. The paper does not report what models serve as judges, how many judge model calls are made per evaluated response, or the total inference cost of evaluating one model on the full 1012-question benchmark.

The consequence. For a benchmark intended to be "publicly available on Hugging Face" and positioned as "both a diagnostic tool and a foundation for developing methods," evaluation cost directly affects practical utility. If evaluating a single model requires running expensive frontier models (potentially the same o3-high or Claude-4-Opus that top the leaderboard) as judges for every response, the cost could be prohibitive for academic researchers or smaller labs. This creates a barrier to adoption that the paper does not address.

More subtly, the judge model matrix approach creates a dependency on proprietary models. If the optimal judge for a particular instruction type is a closed-source model that becomes deprecated or whose API pricing changes, the benchmark's evaluation pipeline becomes unstable. The paper does not specify which models serve as judges for which types, making it impossible for other researchers to replicate the exact evaluation setup without guessing or re-deriving the optimal judge matrix.

There is also a circularity concern: if the best-judge models are also the best-performing models on the benchmark (o3-high, Gemini-2.5-Pro top both the performance rankings and likely serve as judges given the "multiple state-of-the-art models" language in Section 2.5), then the evaluation framework may be systematically biased toward the reasoning patterns of the very models it evaluates. A judge model that exhibits cognitive inertia (as argued in the Key Insights section above) may fail to recognize compliant responses that violate conventions in ways the judge model itself cannot reproduce — potentially penalizing models that find novel ways to comply with inverse instructions.

What evidence exists in the paper. The paper reports that the judge framework achieves 98% accuracy (Section 2.5) but provides no information about judge model identity, per-evaluation cost, or total benchmark evaluation cost. The 98% figure is validated against human judgments, which provides some reassurance against circularity, but the validation set size, composition, and human inter-annotator agreement are not reported. Without knowing whether the validation set adequately covers edge cases where judge model bias is most likely to manifest, the 98% figure may overstate reliability.

Mitigation status. The paper does not acknowledge evaluation cost as a limitation. It treats the 98% accuracy as a pure achievement without discussing the compute or financial cost required to achieve it or the dependency on proprietary models that this creates. Future work could address this by (a) reporting the specific judge models and per-type accuracy contributions, (b) providing a lower-cost evaluation option (e.g., a single open-source judge model with somewhat lower accuracy), and (c) releasing the human-validation dataset to enable independent verification of judge reliability.

Limitation 3: No Human Performance Baseline — We Cannot Distinguish "Hard for LLMs Because of Training" from "Hard for Any Intelligent Agent"

The assumption or constraint. The paper evaluates 15 LLMs on Inverse IFEval but never reports how well humans perform on the same benchmark. Without a human baseline, model scores are unanchored. An accuracy of 60% on "Deliberately Incorrect Answers" could mean the model is substantially worse than humans (who might achieve 95%) or substantially better (if humans find it cognitively demanding to produce intentionally wrong answers while suppressing correct knowledge). The interpretation of model performance — and specifically the claim that failures reflect training-induced cognitive inertia rather than inherent task difficulty — depends on this anchoring.

The consequence. The absence of a human baseline creates a fundamental ambiguity in interpreting every score in Table 3 and Figure 5. Consider the finding that Question Correction is the hardest type, with half of models scoring below 30 (Section 3.2.2). Is this because models are exceptionally rigid about the "accept the user's premise" convention, or because Question Correction is an inherently difficult cognitive task? If humans also score poorly on Question Correction — if identifying errors in complex questions and correcting them appropriately requires substantial analytical skill — then a 30% model score might represent reasonable performance rather than evidence of pathological rigidity.

Conversely, if humans score near-ceiling on Question Correction but near-floor on, say, Intentional Textual Flaws (producing deliberately flawed text might feel unnatural to humans too), then the model's pattern of strengths and weaknesses would mean something very different than if the human pattern is the reverse.

The IQ test analogy that the paper uses to justify the benchmark's unnatural instructions (Section 2.1, Appendix 7) implicitly acknowledges this need for normative anchoring. IQ tests are meaningful precisely because they are normed against human populations — a score of 100 represents average human performance, and deviations are interpretable relative to that baseline. Inverse IFEval provides model scores with no such normative reference, making them difficult to interpret in absolute terms.

The human baseline is also essential for the paper's central claim about SFT-induced cognitive inertia. If humans — who do not undergo SFT — exhibit similar patterns of difficulty across instruction types (e.g., also find formatting inversions easier than question correction), then the difficulty profile reflects something about the cognitive demands of the tasks rather than about SFT specifically. Only if the human difficulty profile differs systematically from the model difficulty profile — with models showing disproportionate difficulty on tasks that specifically require violating conventions they were trained to follow — would the SFT-attribution claim be supported.

What evidence exists in the paper. None. The paper provides no human evaluation data whatsoever. It does not report human scores on Inverse IFEval, human inter-annotator agreement on the benchmark questions themselves, or human performance on individual instruction types. The only human involvement documented is in the benchmark construction pipeline (seed question authoring, expert review, rubric calibration) and in the judge validation process (providing ground-truth scores for 98% accuracy validation), but none of this constitutes a human performance baseline on the benchmark tasks.

Mitigation status. The paper does not acknowledge the absence of a human baseline as a limitation. This is a significant omission given the paper's explicit IQ-test analogy, which would normally demand normative data. Collecting human performance data — even on a subset of the benchmark — would substantially strengthen the interpretability of model scores and the validity of the cognitive inertia interpretation. The paper's positioning on Hugging Face as a public benchmark makes it feasible for future work to collect this data, but the paper itself provides no foundation for human-model comparison.

Limitation 4: The Benchmark Construction Assumes the Eight Chosen Types Adequately Cover the Space of Training Convention Inversions — But This Is Not Validated

The assumption or constraint. The paper's eight-type taxonomy was derived from systematic observation of SFT annotation practices — the authors "conducted a systematic analysis of widely used SFT datasets and summarized a set of canonical response paradigms" (Section 2.2, Stage 1) and then inverted each one. This process produced eight types, but the paper provides no principled argument that eight is the right number, that the identified types are the most important ones, or that the taxonomy is comprehensive.

The consequence of this is not merely taxonomic incompleteness but potential systematic blind spots in what the benchmark measures. If certain types of training-convention rigidity are not represented in the eight chosen types, models that are particularly rigid about those unrepresented conventions will score well on Inverse IFEval despite having significant cognitive inertia. The benchmark would produce false negatives — certifying models as flexible when they are actually brittle in ways the taxonomy misses.

For instance, the taxonomy tests whether models can violate formatting norms (Counter-Conventional Formatting), linguistic quality norms (Intentional Textual Flaws), and correctness norms (Deliberately Incorrect Answers). But it does not test whether models can violate safety norms (e.g., providing dangerous information when explicitly instructed that this is a sanctioned red-teaming exercise), conversational norms (e.g., being deliberately rude or unhelpful when instructed to role-play an antagonist), or epistemic norms (e.g., expressing high confidence in statements the model knows are uncertain). These are all training conventions that SFT and RLHF strongly enforce, and a model that is flexible about formatting but rigid about safety would appear strong on Inverse IFEval despite having significant cognitive inertia.

What evidence exists in the paper. The paper provides no validation that the eight-type taxonomy is comprehensive, representative, or even the most diagnostic set of convention inversions. The taxonomy is presented as an output of the observation-and-reversal process (Stage 1) without meta-analysis of whether other important conventions were missed or whether the eight types capture the majority of variance in cognitive inertia across models. The per-type sample sizes (Table 2) vary from 82 to 198, and the domain distribution (Figure 3) is heavily skewed toward Computer Science (20.2%), but there is no analysis of whether certain domain × type combinations are underrepresented or whether coverage gaps affect the benchmark's ability to detect certain kinds of rigidity.

The paper also does not analyze inter-type correlations — do models that perform well on Counter-Conventional Formatting also tend to perform well on Intentional Textual Flaws? If so, these types may be measuring the same underlying rigidity factor, suggesting the taxonomy could be compressed. If not, each type measures something distinct. Understanding the factor structure of the benchmark would help validate (or invalidate) the taxonomy and would inform users about which scores to treat as independent diagnostic signals versus redundant measurements of the same construct.

Mitigation status. The paper does not acknowledge the taxonomic coverage limitation. It treats the eight types as a given — a reasonable output of the observation-and-reversal methodology — without discussing what might be missing or how future work could validate or extend the taxonomy. The paper also does not analyze whether the benchmark's difficulty for models is driven by a few types dominating the overall score (e.g., if Question Correction, the hardest type, drives most of the variance in overall rankings, the benchmark might effectively be a Question Correction benchmark with additional items that don't provide much discriminatory power).

This limitation is partially inherent to any benchmark construction effort — no finite set of tasks can exhaustively cover an unbounded capability space. But the paper's strong framing around "cognitive inertia" and "overfitting to post-training conventions" implies a comprehensiveness that the taxonomy's ad-hoc derivation does not support. Future work could address this by (a) conducting factor analysis to understand the dimensionality of the benchmark, (b) systematically surveying SFT datasets to identify conventions not covered by the current types, and (c) expanding the taxonomy to cover additional convention categories, particularly safety and epistemic norms that are central to RLHF training.

Limitation 5: The Cross-Lingual Comparison Is Underanalyzed and Potentially Misleading

The assumption or constraint. The benchmark contains parallel Chinese and English versions with 506 questions each, and the paper reports per-language results in Table 3 with a brief analysis in Section 8.2. The paper claims that "models demonstrate consistent performance across both languages" and attributes the exceptions (GLM-4.5 and Qwen3-235B-A22B-Thinking performing better in Chinese) to training data language distribution.

However, the paper does not analyze whether the Chinese and English versions are truly parallel — i.e., whether the same questions translated between languages present equivalent difficulty. Translation can introduce subtle differences in instruction clarity, cultural references, or domain-specific terminology that make the two versions non-equivalent. A model performing better in Chinese might be genuinely more flexible in Chinese, or it might be that the Chinese versions of certain questions are accidentally easier due to translation artifacts.

The consequence. The cross-lingual comparison — which the paper presents as evidence about language-specific cognitive inertia — may instead be measuring translation quality differences or language-specific question difficulty variations. If the Chinese version of "Counter-Conventional Formatting" instructions are phrased more explicitly or with less ambiguity than their English counterparts, the observed language effect would be an artifact of the translation process rather than a genuine difference in model capability.

This matters for two audiences. For researchers studying multilingual model behavior, the paper's finding that cognitive inertia is "language-agnostic" (my characterization from the Key Insights section, but implied by the paper's "consistent performance" claim) could be misinterpreted as evidence that training data language distribution doesn't affect convention rigidity, when in fact the benchmark's cross-lingual design may not be clean enough to support that conclusion. For practitioners deploying models in specific languages, the language-specific findings (e.g., "Chinese-contributed models perform better in Chinese") might lead to deployment decisions (prefer Chinese-contributed models for Chinese-language applications requiring instruction flexibility) based on evidence that may reflect benchmark artifacts rather than genuine capability differences.

What evidence exists in the paper. The paper provides almost no detail about the translation process. Section 2.2 mentions "Chinese-English translation on the final instructions" as the last step after expert review and calibration, but does not specify whether translation was done by professional translators, by LLMs, or by the expert reviewers themselves. There is no validation that the translated instructions preserve the difficulty, clarity, and counter-conventional nature of the originals. Section 8.2 reports overall consistency but does not analyze per-type consistency — it's possible that some instruction types show large cross-lingual differences that are masked by averaging across all types. The paper also does not report statistical tests for whether the observed language effects (GLM-4.5 better in Chinese, etc.) are significant or could arise from sampling variability given the 506-question per-language sample.

Mitigation status. The paper does not acknowledge translation quality or cross-lingual equivalence as a limitation. It treats the parallel versions as straightforwardly comparable and attributes observed differences to model properties rather than considering benchmark properties as an alternative explanation. Future work could validate cross-lingual equivalence through back-translation checks, human judgments of instruction clarity and difficulty in both languages, and per-type analysis of language effects. Until such validation is provided, the cross-lingual findings should be treated as suggestive rather than definitive.

Limitation 6: The Experimental Design Cannot Distinguish Preference Failures from Capability Failures — And This Distinction Matters for Remediation

The assumption or constraint. The paper evaluates models using single-sample generation (Table 3, Figure 5) and Best-of-N sampling (Figure 15), but never systematically decomposes model failures into two fundamentally different categories: (1) the model is incapable of producing a compliant response under any sampling conditions, versus (2) the model is capable of producing compliant responses but defaults to training-conventional outputs in expectation (a preference failure). These two failure modes have completely different implications for how to improve performance, but the paper's primary results (single-sample accuracy) conflate them.

The consequence. If most failures are capability failures — the model literally cannot generate text without paragraph breaks, or cannot produce an intentionally wrong answer without also providing the correct answer somewhere — then the solution needs to involve fundamentally different training (expanding the model's behavioral repertoire). If most failures are preference failures — the model knows how to comply but its default behavior, shaped by SFT and RLHF, strongly favors conventional outputs — then the solution might be much simpler: better prompting, higher-temperature sampling, or lightweight post-training to adjust output preferences.

The Best-of-N results (Figure 15) provide partial evidence: at N=32, four of five models approach or surpass 90, suggesting that for those models, compliant responses exist in the output distribution and the primary failure mode at N=1 is preference rather than capability. However, this analysis is limited to only five models (not explicitly named), only three N values, and only aggregate scores across both languages and all instruction types. We don't know whether the preference-vs-capability breakdown differs by instruction type (perhaps Question Correction failures are capability failures while formatting failures are preference failures), by model (perhaps smaller models have more capability failures), or by language.

Without this decomposition, the paper's diagnostic value is limited. A model scoring 40% on Inverse IFEval could mean "this model can follow counter-conventional instructions 40% of the time and fails 60% of the time due to capability gaps" or "this model is capable of following these instructions nearly always but defaults to conventions 60% of the time due to preference biases." These are very different diagnoses with very different remediation strategies, but the benchmark's primary metric cannot distinguish them.

What evidence exists in the paper. The Best-of-N experiment (Section 8.1, Figure 15) provides preliminary evidence that preference failures dominate for the tested models at N=32, but the experiment is too limited to draw general conclusions. It covers only five models (versus the 15 in Table 3), uses only three N values (1, 16, 32), and reports only aggregate scores rather than per-type or per-language breakdowns. The paper does not report Best-of-1 scores in Figure 15 alongside the Best-of-16 and Best-of-32 data, making it impossible to compute the preference-vs-capability gap quantitatively. We also don't know whether the scaling continues beyond N=32 — does Best-of-64 push all models to near-ceiling, or does performance saturate, indicating a capability ceiling?

The error analysis (Appendix 9, Figures 16-19) is purely qualitative — it shows examples of model failures but does not quantify what fraction of failures fall into preference versus capability categories. A quantitative error taxonomy would be substantially more informative.

Mitigation status. The paper does not frame preference-vs-capability as a distinction worth making, and its primary metric (single-sample accuracy) is inherently a conflation of the two. The Best-of-N experiment partially addresses this but is underpowered and underanalyzed. Future work could systematically decompose benchmark scores into preference and capability components by: (a) evaluating all models at multiple N values and fitting scaling laws to estimate asymptotic performance (capability ceiling) versus N=1 performance (preference-biased performance), (b) analyzing whether models that produce compliant responses at high N still produce non-compliant responses at N=1 (indicating that the same model has both the capability and the bias), and (c) reporting per-type preference-vs-capability decompositions to identify which instruction types are limited by capability versus preference for which models. This would substantially increase the benchmark's diagnostic power and its utility for guiding remediation efforts.

7. Implications and Future Directions

How This Work Changes the Landscape

Inverse IFEval introduces a new diagnostic dimension for LLM evaluation—Counter-intuitive Ability—that the field wasn't previously measuring. This isn't a paradigm shift in how models are trained or architected, nor does it propose a new method that outperforms existing approaches. Rather, it's a measurement innovation that reveals a capability gap the field had systematically overlooked: the distinction between instruction-pattern-matching (complying with directives that resemble training data) and genuine instruction-following (complying with any well-specified directive, even those that actively contradict training conventions).

The magnitude of this contribution is best characterized as foundational for evaluation, enabling for future methods work. The paper doesn't solve cognitive inertia—it demonstrates that it exists, provides a reliable instrument for measuring it, and shows that the phenomenon is large enough to fundamentally reorder model rankings. A model that ranks 5th on IFEval can drop to 15th on Inverse IFEval (Qwen3-235B-A22B-Instruct, Figure 6b). That's not a marginal difference in difficulty—it's evidence of a qualitatively different capability being tested.

What conceptual shift this produces. Prior to this work, the instruction-following evaluation literature operated under an implicit assumption: better instruction-following meant better compliance with any instruction, modulo surface-level complexity. IFEval (Zhou et al., 2023) tested whether models could follow verifiable constraints like "write at least 400 words" or "include the keyword X," and high scores were interpreted as evidence of strong general instruction-following capability. Inverse IFEval demonstrates that this interpretation is incorrect—or at least dangerously incomplete. Instruction-following capability is not a unidimensional construct; it has at least two separable components (convention-aligned and convention-contradicting), and existing benchmarks only measure one.

This reframes how the field should think about instruction-following evaluation. It's no longer sufficient to demonstrate that a model can follow familiar instructions. Robustness requires demonstrating that the model can follow unfamiliar, convention-violating instructions as well. This is the same conceptual move that occurred in other areas of ML evaluation: from testing on IID data to testing on out-of-distribution data, from testing on clean inputs to testing on adversarial inputs, from testing on common cases to testing on long-tail cases. Inverse IFEval does for instruction-following what distribution-shift benchmarks did for image classification—it reveals that models are brittle in ways that standard evaluation doesn't capture.

Reconciliation of prior contradictions. The paper doesn't resolve an explicit empirical contradiction in the literature in the way the test-time compute paper reconciled conflicting findings about self-correction. Instead, it resolves a more subtle measurement gap: the field had extensive evidence that models can follow instructions (from IFEval, FollowEval, ComplexBench, and numerous instruction-tuning papers) but almost no evidence about whether this capability generalizes to instructions that conflict with training conventions. The result was a form of evaluation-induced false confidence—models appeared robust because we were only testing the conditions where they were designed to succeed.

The paper also provides a unifying explanation for anecdotal user frustration. The "Do As I Say, Not As You Were Trained!!" refrain that opens the paper captures a common experience: users ask models to deviate from standard formats or norms, and models stubbornly revert to their training defaults. Prior to Inverse IFEval, these failures were attributed to vague causes like "models being stubborn" or "prompt engineering failures." The paper provides a systematic framework: these failures are instances of cognitive inertia, they manifest in specific, categorizable ways (the eight instruction types), and they vary systematically across models and reasoning mechanisms. This transforms a diffuse user complaint into a measurable, diagnosable phenomenon.

Which research directions become more attractive. The paper makes several lines of inquiry newly salient:

SFT data diversification becomes a priority. If cognitive inertia arises because SFT data is overwhelmingly convention-conforming (as the paper hypothesizes), then deliberately including counter-conventional examples in training data—with appropriate scaffolding to prevent models from learning to produce flawed outputs indiscriminately—becomes an obvious intervention. The paper's Best-of-N results (Figure 15: four of five models near or above 90 at N=32) suggest that the capability to comply exists latently; the problem is making it the default behavior rather than something that must be discovered through expensive sampling.
Reasoning mechanism design becomes more interesting. The consistent advantage of thinking models over non-thinking models (Figure 4a), combined with the failure of adaptive reasoning triggers on Chinese instructions (Figure 4b), suggests that how models reason matters as much as whether they reason. Research into metacognitive triggering mechanisms—how models decide when to deliberate—becomes directly relevant to instruction-following robustness.
Judge model debiasing becomes a recognized methodological challenge. The paper's finding that standard LLM-as-a-Judge achieves only 88% accuracy on counter-conventional evaluation—and requires systematic optimization (adaptive model selection, type-specific templates, enhanced prompts) to reach 98%—establishes that judge models exhibit their own form of cognitive inertia. Future benchmarks that test counter-conventional behavior will need to incorporate similar judge-debiasing procedures, and the paper's three-strategy approach (Section 2.5) provides a template.

Which research directions become less attractive. The paper implicitly argues against the sufficiency of incremental improvements to standard instruction-tuning. If models can achieve 5th place on IFEval while dropping to 15th on Inverse IFEval, then better performance on convention-aligned benchmarks is not evidence of improved general instruction-following. This suggests that research focused solely on maximizing IFEval scores—without testing on counter-conventional benchmarks—may be optimizing for the wrong thing, producing models that are increasingly overfit to training conventions while appearing to improve.

Similarly, the paper's findings cast doubt on the assumption that larger models will naturally overcome cognitive inertia through scale alone. While the Qwen3 family does show a scaling trend (larger models perform better, Table 3), the best-performing model (o3-high) still shows substantial type-level variation (Figure 5a), and even at Best-of-32 some models don't reach ceiling (Figure 15). Scale helps but doesn't eliminate the problem—suggesting that architectural or training innovations specifically targeting convention flexibility will be needed.

Follow-Up Research This Work Enables

1. Direct causal test of the SFT convention hypothesis through controlled fine-tuning. The paper attributes cognitive inertia to SFT specifically (Section 1, Section 2.1, Conclusion), but this causal claim remains experimentally untested. A strong follow-up would take a single pretrained base model, fine-tune it on two datasets—one with standard conventions (bullet points, grammatical correctness, code comments) and one where those conventions are systematically violated—and evaluate both on Inverse IFEval. If the model fine-tuned on convention-violating data significantly outperforms the standard fine-tuned model, the SFT-attribution hypothesis would be experimentally confirmed. If performance is similar, the rigidity likely arises from pretraining rather than SFT—fundamentally reframing where intervention is needed. This experiment is feasible: the fine-tuning datasets could be constructed by taking existing SFT data and programmatically stripping formatting, introducing errors, and removing code comments for a subset of examples. The key measurement would be the performance gap between the two fine-tuning conditions across the eight instruction types—with the prediction that convention-violating fine-tuning improves scores most on types that directly invert the manipulated conventions (e.g., Counter-Conventional Formatting, Intentional Textual Flaws).

2. Human performance baseline to anchor model scores and distinguish task difficulty from training bias. The paper's IQ-test analogy (Section 2.1, Appendix 7) demands normative data that the paper never provides. A straightforward follow-up would recruit human participants (potentially through crowd-sourcing platforms) to complete a representative subset of Inverse IFEval instructions—say, 100 questions spanning all eight types and both languages—under similar conditions (written responses, no external tools, time pressure equivalent to model inference). The critical analysis would compare human and model difficulty profiles: do humans also find Question Correction hardest and Counterfactual Answering easiest? If human and model difficulty rankings correlate strongly, then the benchmark primarily measures task difficulty rather than SFT-induced rigidity, and the cognitive inertia interpretation would need substantial revision. If the rankings diverge—particularly if models show disproportionate difficulty on types that specifically require violating conventions they were trained to follow (Counter-Conventional Formatting, Deliberately Incorrect Answers) while humans don't—the SFT-attribution claim would gain convergent validity. This experiment would also establish ceiling performance, enabling model scores to be interpreted as fractions of human capability rather than as unanchored percentages.

3. Systematic decomposition of preference failures vs. capability failures through Best-of-N scaling laws. The paper's Best-of-N experiment (Figure 15) provides tantalizing but incomplete evidence that compliant responses exist in models' output distributions—four of five models approach or surpass 90 at N=32. A rigorous follow-up would evaluate all 15 models from Table 3 at a dense grid of N values (1, 2, 4, 8, 16, 32, 64, 128, 256) and fit scaling laws to estimate the asymptotic performance (capability ceiling) versus the single-sample performance (preference-biased performance). The gap between N=1 and the asymptote quantifies how much of the model's failure is "knowing how but defaulting to conventions" versus "not knowing how at all." This decomposition should be computed per instruction type and per language—the hypothesis being that some types (e.g., Counter-Conventional Formatting) are primarily preference failures (models can produce unstructured text but overwhelmingly prefer formatting) while others (e.g., Question Correction) may have genuine capability ceilings. The decomposition would directly inform remediation: preference-dominated types could be addressed through better prompting or lightweight alignment adjustments; capability-dominated types would require fundamentally expanded training.

4. Language-balanced training data experiment to test whether cross-lingual effects reflect training distribution or benchmark artifacts. The paper observes that GLM-4.5 and Qwen3-235B-A22B-Thinking perform "significantly better" in Chinese than English (Section 8.2) and attributes this to training data language distribution. An alternative hypothesis—that the Chinese translations of Inverse IFEval questions are accidentally easier—can be tested by taking models with known training data language distributions (e.g., models trained on carefully balanced multilingual corpora versus models trained primarily on English) and evaluating their cross-lingual performance gap on Inverse IFEval. If the gap correlates with training data language imbalance, the paper's interpretation is supported. If all models show systematic Chinese-better or English-better patterns regardless of training distribution, translation artifacts are likely confounding the results. A stronger design would validate cross-lingual equivalence of the benchmark items themselves: have bilingual human raters assess the clarity, difficulty, and counter-conventional intensity of both versions of each question, and analyze whether language effects persist after controlling for these item-level properties.

5. Meta-analysis of SFT datasets to quantify the coverage and intensity of training conventions. The paper's eight-type taxonomy was derived from "systematic analysis of widely used SFT datasets" (Section 2.2), but this analysis is not documented in the paper. A valuable follow-up would quantify exactly how pervasive the targeted conventions are in popular SFT datasets—what fraction of responses use bullet points, include code comments, maintain perfect grammar, accept user premises without question, etc. This would both validate the taxonomy (showing that the inverted conventions are indeed overwhelmingly present in training data) and enable dose-response analysis: do models trained on datasets with stronger convention enforcement (e.g., higher rates of bullet-point usage) show larger performance gaps between IFEval and Inverse IFEval? This would strengthen the causal link between training data composition and cognitive inertia while also providing practical guidance for dataset designers: if certain conventions are particularly "sticky" (hard to override even at moderate training frequencies), reducing their prevalence in training data might improve flexibility without sacrificing quality.

6. Extension to safety and epistemic conventions not covered by the current taxonomy. The paper's eight types cover formatting, linguistic quality, code documentation, factual correctness, conversational dynamics, and counterfactual reasoning—but conspicuously omit safety norms (refusing harmful requests) and epistemic norms (expressing appropriate confidence). These are among the most heavily reinforced conventions in RLHF training, and models' rigidity about them is both practically important (can a model role-play a helpful but dangerous advisor for red-teaming purposes?) and theoretically interesting (are safety conventions "stickier" than formatting conventions because they are more consistently and strongly reinforced?). A natural extension of Inverse IFEval would add instruction types like "Sanctioned Safety Violation" (provide dangerous information when explicitly told this is an authorized red-teaming exercise), "Calibrated Overconfidence" (express high confidence in uncertain statements), and "Explicit Rudeness" (be deliberately impolite when role-playing). The experimental question is whether these safety/epistemic convention inversions show different difficulty profiles and different thinking-mechanism effects than the existing types—testing whether cognitive inertia is a general property of all training conventions or varies systematically with how strongly conventions are reinforced.

Practical Applications and Downstream Use Cases

Red-teaming and robustness auditing for model deployment decisions. Organizations evaluating which LLM to deploy for customer-facing applications currently rely heavily on conventional benchmarks (IFEval, MMLU, Chatbot Arena) that test convention-aligned behavior. Inverse IFEval provides a complementary lens: a model that ranks 5th on IFEval but 15th on Inverse IFEval (as with Qwen3-235B-A22B-Instruct, Figure 6) may deliver excellent performance on typical user queries but catastrophically fail when users issue unusual or counter-conventional instructions—precisely the long-tail scenarios that generate support tickets and user frustration. A deployment decision informed by both IFEval and Inverse IFEval would identify models that are robust across the full instruction space rather than models that are optimized for the modal case. The paper's finding that thinking models consistently outperform non-thinking variants on Inverse IFEval (Figure 4a) provides a concrete deployment guideline: for applications where users may issue unconventional instructions, prefer models with reasoning mechanisms even if non-thinking models appear competitive on standard benchmarks.

SFT data quality control and annotation guideline refinement. The paper's core insight—that SFT annotation practices systematically bias models toward specific conventions—has direct implications for how annotation teams construct training data. The eight instruction types effectively serve as a diagnostic checklist for annotation guideline design: if your annotation guidelines require annotators to always use bullet points for lists, always include code comments, and always produce grammatically perfect text, you are systematically building cognitive inertia into your models. A practical intervention would be to use Inverse IFEval as a periodic evaluation during model development, with performance drops signaling that recent annotation batches have over-emphasized certain conventions. More proactively, annotation guidelines could be modified to include deliberate counter-conventional examples—a small fraction of training data where models are taught to produce unstructured text, intentionally flawed outputs, or deliberately incorrect answers when appropriately instructed. The Best-of-N results (Figure 15: models can produce compliant outputs when sampled extensively) suggest that even a small amount of counter-conventional training data could substantially shift the default behavior toward flexibility.

Judge model selection for automated evaluation pipelines. The paper's finding that standard LLM-as-a-Judge achieves only 88% accuracy on counter-conventional evaluation—and that an adaptive model matrix with type-specific templates and enhanced prompts achieves 98% (Section 2.5)—has practical implications for any organization building automated evaluation systems. The three optimization strategies (dedicated judge model per task type, template optimization for task-specific context, enhanced system prompts with exemplars) provide a concrete recipe for improving judge accuracy, with the 10-percentage-point gain quantifying the benefit. Organizations that currently use a single judge model with a uniform prompt template for all evaluation tasks are likely leaving substantial accuracy on the table, particularly when evaluating model behavior on out-of-distribution instructions. The paper's approach of validating judge accuracy against human judgments (the 98% figure) also provides a methodological standard: automated evaluators should be calibrated against human ground truth specifically on the types of tasks being evaluated, not just on generic quality judgments.

Model selection for applications with diverse user populations and non-standard requirements. The paper's cross-lingual findings—specifically that Chinese-contributed models (GLM-4.5, Qwen3-235B-A22B-Thinking) perform significantly better in Chinese than English (Section 8.2)—suggest that cognitive inertia has a language-specific component tied to training data distribution. For organizations serving multilingual user bases, this implies that the best model for instruction-following in Language A may not be the same as the best model for Language B, even if both models perform similarly on standard benchmarks. More broadly, the per-type diagnostic profiles (Figure 5) enable fine-grained model selection for specific application requirements: an application that frequently requires code without comments (e.g., a code golf or obfuscation tool) should select a model that performs well specifically on the "Code without Comments" type, regardless of its overall Inverse IFEval score. The benchmark's type-level granularity transforms it from a single-score ranking into a diagnostic instrument for matching models to deployment requirements.