EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A

🎯 Pitch

A 4B parameter model outperforms GPT-5.2 and Claude Opus at detecting when executives dodge questions—achieving 84.9% Macro-F1. The secret is training on labels generated by multi-model consensus, which boosts performance by 4.3 points over single-LLM annotation and resolves the infamous subjectivity problem that has kept evasion detection without a benchmark until now.

1. Executive Summary

This paper introduces EvasionBench, a large-scale benchmark for detecting evasive managerial responses in earnings call Q&A sessions, along with Eva-4B, a 4-billion-parameter classifier fine-tuned from Qwen3-4B that achieves 84.9% Macro-F1 on this task—outperforming frontier models including Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash. The annotation framework employs a Multi-Model Consensus (MMC) framework that combines dual frontier LLM labeling with a three-judge majority voting mechanism for ambiguous cases (resolving 16.1% of samples where primary annotators disagree), achieving a Cohen's Kappa of 0.835 against human annotators on a three-level evasion taxonomy—direct, intermediate, and fully evasive. Ablation studies demonstrate that MMC labeling yields a +4.3 percentage point Macro-F1 improvement over single-model annotation and produces dramatically cleaner training signals (final loss of 0.007 versus 0.56), establishing that consensus-based LLM annotation is essential for generating reliable evasion labels only when the underlying models exhibit systematically different labeling tendencies that single-model approaches cannot resolve.

2. Context and Motivation

The Core Problem: Evasion Detection Lacks a Canonical Benchmark

The fundamental problem this paper addresses is deceptively straightforward: despite decades of research across multiple disciplines demonstrating that evasive communication is pervasive, consequential, and theoretically well-motivated, the NLP community lacks any large-scale benchmark for detecting it. The authors frame this absence as a critical gap that has prevented systematic progress on a discourse-level phenomenon fundamentally different from the classification tasks that dominate NLP benchmarks.

This gap is not an accident of neglect. The paper identifies two structural challenges that have historically made evasion detection resistant to the standard benchmarking approach that has been so successful for tasks like sentiment analysis or question answering:

Challenge 1: Evasiveness is inherently graded, not binary. Unlike sentiment (positive/negative) or stance detection (favor/against/neutral), whether a response is "evasive" exists on a continuous spectrum. An executive might partially address a question, answer a related-but-different question, provide context without the specific answer requested, or completely ignore the question—and the boundaries between these categories are fuzzy. The paper grounds this observation in Gricean pragmatics (Grice, 1975), specifically the Maxim of Relation: a fully cooperative response adheres to the maxim, a partially evasive one partially violates it, and a fully evasive one completely violates it. This is not a matter of identifying surface-level linguistic features (like negative sentiment words) but of assessing discourse-level coherence between what was asked and what was answered—a fundamentally harder problem.

Challenge 2: Expert annotation is prohibitively expensive at scale. Detecting evasion requires understanding both the question's intent and whether the response satisfies it. In financial contexts, this demands domain expertise—understanding accounting terminology, competitive dynamics, disclosure norms, and what constitutes a "real" answer to a specific type of analyst question. Human expert annotation at the scale required for training modern neural classifiers (tens of thousands of examples) is financially prohibitive. This cost barrier has constrained prior datasets to at most a few thousand examples (Gow et al., 2021's ~2.1K; Nuaimi et al., 2025's 12K), which is insufficient for training robust models and leaves the research community without a shared evaluation framework.

These two challenges interact in a way that compounds the difficulty: the graded nature of evasion means annotators need domain expertise to make fine-grained judgments, but domain expertise is precisely what makes annotation expensive, which limits scale, which means no benchmark, which means no systematic progress.

Why This Problem Matters: Real-World Stakes and Theoretical Significance

The importance of evasion detection operates on two distinct levels: concrete real-world consequences and fundamental theoretical questions about language understanding.

Real-World Impact Across Multiple Domains

The paper marshals evidence from multiple fields to establish that evasion is not merely a linguistic curiosity—it has measurable, consequential effects:

In financial markets: The paper cites evidence that evasive management communication predicts subsequent earnings misses and stock underperformance (Barcellos, 2025; Larcker and Zakolyukina, 2012). Gow et al. (2021) find that approximately 11% of analyst questions receive non-answers, and that product-related questions in competitive environments are systematically evaded. This is not random—managers are strategically choosing what to disclose, and investors interpret silence negatively (Hollander et al., 2010). An automated evasion detection system operating at scale could serve as an early warning mechanism for investors, flagging patterns of non-responsiveness that might precede negative financial events.

In politics: Bull and Mayer (1993) identified 43 distinct techniques politicians use to avoid answering questions. Evasive political communication undermines democratic accountability: if interviewers cannot extract commitments from politicians, and audiences cannot reliably detect when questions are being evaded, the informational function of political interviews breaks down. Clayman (2001) provided a foundational distinction between overt evasion (explicit refusal—relatively easy to detect) and covert evasion (appearing to answer while sidestepping—much harder, and the primary challenge this paper addresses).

In legal contexts: Bachenko et al. (2008) and Vrij (2008) have studied deception and evasion in legal testimony, where truth-finding is the explicit goal. An important distinction the paper draws: evasion differs from deception. Deception involves making false statements—an assertion-level phenomenon. Evasion involves strategic non-responsiveness where speakers appear cooperative but avoid addressing the question core (Pérez-Rosas and Mihalcea, 2015). Detecting evasion thus requires assessing what was not said relative to what was asked, not what was said relative to ground truth. This makes existing deception detection approaches (e.g., Newman et al., 2003's LIWC-based linguistic markers achieving 61-67% accuracy) insufficient for evasion detection.

Theoretical Significance: Testing Ground for Discourse Understanding

Beyond practical applications, evasion detection serves as an acid test for whether NLP systems can perform discourse-level pragmatic reasoning—understanding not just the literal content of utterances but their communicative function within a conversational context.

The paper explicitly frames evasion through the lens of Gricean pragmatics (Grice, 1975). Grice proposed that cooperative communication is governed by maxims (Quantity, Quality, Relation, Manner). Evasion represents a violation—often covert—of the Maxim of Relation: "Be relevant." Unlike sentiment analysis, which operates largely at the lexical and sentence level, evasion detection requires:

Parsing the question's core communicative intent
Extracting the response's actual informational content
Evaluating whether (2) is relevant to (1)
Recognizing when surface relevance (related vocabulary, adjacent topics) masks a failure to address the specific ask

This is exactly the kind of structured reasoning that LLMs are hypothesized to be capable of but that standard NLP benchmarks do not systematically evaluate. The paper positions earnings calls as an ideal testbed precisely because they provide naturally occurring, high-stakes, adversarial Q&A with clearly defined speaker roles and well-structured question types that enable operationalization of "directness."

Where Prior Approaches Fall Short

The paper identifies three specific limitations in existing work, which are usefully understood through the lens of Table 1 in the paper (reproduced in Section 2.5):

Limitation 1: Scale is Insufficient for Modern Methods

The largest prior dataset specifically focused on evasion in earnings calls contains at most 12K labeled examples (Nuaimi et al., 2025). Gow et al. (2021), a foundational empirical study, analyzed only 2.1K Q&A pairs for non-answering behavior. These scales are adequate for regression-style analysis in accounting research (the domain where most of this work originates) but are fundamentally insufficient for training and evaluating modern neural classifiers, which typically require tens of thousands of examples to converge reliably and to capture the subtle linguistic patterns involved in intermediate forms of evasion.

This scale limitation has a cascading effect: without a large training set, researchers cannot train specialized evasion detection models, which means evaluation must rely on off-the-shelf models prompted zero-shot, which means performance is bounded by whatever evasion detection capability those models incidentally possess, which means the field cannot make cumulative progress through improved modeling techniques applied to a shared benchmark.

Limitation 2: Annotation Quality Suffers Without Consensus Mechanisms

Nuaimi et al. (2025) propose a seven-type psychological taxonomy for evasive answers, representing the most fine-grained prior annotation effort. However, the paper identifies a critical weakness: "their dataset relies solely on single-model LLM annotation without human validation, raising concerns about label reliability and potential model-specific biases." This is not merely a methodological footnote—it is a fundamental threat to validity.

The paper provides direct evidence for why single-model annotation is problematic through Figure 2. When the authors compared the labeling tendencies of three frontier LLMs (Claude Opus 4.5, Gemini 3 Flash, GPT-5.2) on the same samples, they found systematically different distributions: Opus 4.5 assigned "direct" labels to 53.3% of disagreement samples, Gemini 3 Flash assigned "fully evasive" to 23.5%, and GPT-5.2 assigned "intermediate" to 56.7%. These are not random variations around a stable consensus—they represent fundamentally different interpretations of what counts as evasive. A dataset labeled solely by Opus would underestimate evasion prevalence; one labeled solely by Gemini would overestimate it; one labeled solely by GPT-5.2 would produce an ambiguous intermediate-heavy distribution.

This finding is particularly significant because it demonstrates that annotation bias is not noise that can be overcome with scale—it is systematic, model-specific, and would be baked into any single-model-labeled dataset regardless of size. A 100K-sample dataset labeled entirely by Opus would still systematically undercount fully evasive responses. The MMC framework is thus not merely a cost-saving measure; it is a necessary methodological innovation for producing labels that approximate an unbiased consensus.

Limitation 3: No Prior Work Provides Both Training Data and Rigorous Evaluation

Table 1 makes explicit what is implicit throughout the literature review: prior datasets offer either scale without human validation (Nuaimi et al., 2025: 12K samples but LLM-labeled only) or human validation without sufficient scale for training (Gow et al., 2021: rigorously validated but only 2.1K samples, two label categories). The field has no single resource that provides:

Sufficient scale (tens of thousands of examples) for model training
Human validation to establish label quality and benchmark performance
A taxonomy granular enough to capture meaningful distinctions (three levels rather than binary)
Broad coverage across companies, time periods, and speaker pairs

This is the gap EvasionBench explicitly fills.

How This Paper Positions Itself

The paper positions EvasionBench not as a single contribution but as an infrastructure contribution that enables a line of research previously impossible. The framing is deliberate: the benchmark is the primary artifact, and Eva-4B is a demonstration that a relatively small model, trained on high-quality consensus data, can achieve competitive or superior performance to frontier models prompted zero-shot. This inverts the typical LLM narrative—the value is in the data quality and the annotation methodology, not giant model scale.

Several specific positioning choices are worth noting:

Domain-specific but methodology-transferable. The paper is explicit that earnings calls are the domain of study but that "both our annotation framework and taxonomy could transfer to other adversarial Q&A settings (e.g., political interviews, legal depositions), though cross-domain validation remains future work." This is a careful hedge—they are not claiming that Eva-4B trained on earnings calls will generalize to political interviews, but that the annotation methodology (MMC with three-judge arbitration) and taxonomy design principles (empirically validated three-level scale grounded in Gricean pragmatics) are domain-agnostic.

Positioned against sentiment analysis as a contrast case. The paper repeatedly invokes sentiment analysis not as a competitor but as a point of contrast. Sentiment has canonical benchmarks (SST, Financial PhraseBank) that enabled systematic progress; evasion detection lacks this. The implicit argument is: the NLP community knows how to make progress on a task when it has a shared benchmark; EvasionBench provides the missing benchmark for evasion detection.

LLMs as annotation tools, not just evaluation targets. A key positioning move is that LLMs serve a dual role: they are the objects of evaluation (Table 5 ranks them alongside Eva-4B) AND the instruments of annotation (Claude Opus 4.5 and Gemini 3 Flash label the training data). This creates an interesting dynamic where a model could theoretically evaluate itself on data it helped label—addressed by using human-validated gold data as the evaluation set rather than held-out LLM-labeled data.

The value proposition is methodology over model architecture. Eva-4B is a standard fine-tuned Qwen3-4B; there is no novel architecture, loss function, or training algorithm. The intellectual contribution is in the annotation pipeline that produced the training data, and the demonstration that data quality (via MMC) matters more than model scale for this task. The +4.3 pp improvement from three-judge consensus over single-model annotation is not a modeling contribution—it is a data quality contribution that happens to be measured through model performance.

The Theoretical Foundation: Why Evasion Detection is Harder Than It Looks

Understanding the paper's motivation requires appreciating why evasion detection is qualitatively different from superficially similar NLP tasks. The paper draws on a rich theoretical tradition that clarifies the challenge:

Gricean pragmatics (Grice, 1975) provides the normative framework: cooperative communicators follow the Maxim of Relation. Evasion is a violation. But violations can be overt ("I won't answer that"—easily detectable) or covert (appearing to answer while sidestepping—the core challenge). Covert violations exploit the fact that listeners apply a presumption of relevance: we assume speakers are being cooperative, so we strain to find relevance in their utterances even when it isn't there. This is what makes intermediate evasion so difficult for both humans and models: the response sounds relevant enough to pass casual inspection but fails to deliver the specific information requested.

Face-threatening act mitigation (Brown and Levinson, 1987) explains why speakers evade covertly rather than refusing outright. Explicit refusal is a face-threatening act—it damages the speaker's social standing by appearing uncooperative. Evasive responses that appear cooperative while avoiding the difficult substance preserve face. Bavelas et al. (1990) extended this with the concept of avoidance-avoidance conflict: when all direct answers carry costs, equivocation is the rational strategy. For executives, acknowledging competitive threats, admitting to strategic failures, or providing information that could advantage rivals can all carry real costs—hence strategic evasion.

Hedging and indirectness (Lakoff, 1973; Hyland, 1998; Searle, 1975) provide the linguistic mechanisms through which evasion is realized. Hedges ("I think," "it's possible," "we expect") signal reduced commitment to the propositional content, allowing speakers to appear responsive while avoiding accountability. Indirect speech acts allow speakers to perform one communicative function (e.g., providing general commentary) while being understood as performing another (answering a specific question). The prevalence of these devices means that evasion detection cannot rely on simple keyword or pattern matching—hedging language is also used by cooperative speakers to appropriately qualify uncertain statements.

The paper synthesizes these theoretical perspectives not merely as literature review but as design principles for the EvasionBench taxonomy. The three-level scale (direct, intermediate, fully evasive) maps to degrees of Relation maxim violation. The taxonomy definitions in Section 3.2 operationalize theoretical concepts: "hedging language" and "conditional framing" for intermediate; "explicit refusal" and "information flooding" for fully evasive. This grounding in theory distinguishes EvasionBench from a purely empirical data collection effort—the taxonomy has explanatory content, not just descriptive labels.

3. Technical Approach

3.1 Reader Orientation

EvasionBench is a pipeline that takes raw earnings call transcripts and produces two artifacts: a large, labeled dataset for training evasion detection models, and a gold-standard human-validated evaluation set for benchmarking those models. The core problem is that detecting whether a corporate executive's response actually answers an analyst's question requires discourse-level reasoning that is expensive for humans to do at scale, yet training modern classifiers requires tens of thousands of labeled examples. The solution is a Multi-Model Consensus (MMC) annotation framework that leverages agreement between multiple frontier LLMs as a proxy for annotation quality, combined with human validation on a focused evaluation set, enabling the construction of an 84K-sample training corpus and a 1K gold evaluation set at a fraction of the cost of fully human annotation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a linear pipeline:

Raw Data Ingestion (Section 4): Ingests 22.7 million Q&A pairs from S&P Capital IQ transcripts spanning 2002–2022, extracts analyst questions and management answers through structured parsing, and applies three filtering stages to retain high-quality, substantive exchanges.
Primary Dual-LLM Annotation (Section 5.2): Two frontier LLMs (Claude Opus 4.5 and Gemini 3 Flash) independently label each Q&A pair as direct, intermediate, or fully evasive using structured prompts. Samples where both models agree form the "consensus set" and proceed directly to the training corpus.
Three-Judge Arbitration (Sections 5.3–5.4): For the ~16% of samples where the primary annotators disagree, three judge models (Claude Opus 4.5, Gemini 3 Flash, GPT-5.2) independently evaluate which of the two original annotations is more accurate. Majority voting determines the final label, with randomized presentation order to mitigate position bias. These arbitrated samples join the consensus set to form the complete training data.
Two-Stage Model Training (Section 7): A 4-billion-parameter Qwen3-4B model is fine-tuned in two stages: first on 60K consensus-only samples (where both primary annotators agreed), then continued on 24K samples that include the arbitrated disagreement cases, producing the final Eva-4B classifier.
Human Validation & Evaluation (Sections 6, 8): A separate 1K gold-standard evaluation set is labeled by expert human annotators, with inter-annotator agreement measured on a 100-sample subset (Cohen's Kappa of 0.835). This gold set serves as the benchmark for evaluating all models, including Eva-4B variants and frontier LLMs, ensuring that evaluation is against human judgment rather than LLM-generated labels.

Information flows linearly: raw transcripts → filtered Q&A pairs → dual LLM annotation → consensus check → (if needed) three-judge arbitration → combined training set → two-stage fine-tuning → Eva-4B classifier → evaluation against human-labeled gold set.

3.3 Roadmap for the Deep Dive

First, the raw data extraction and three-stage filtering pipeline (Section 4), because understanding what data survives filtering and why explains the scope and quality of the foundation everything else builds on.
Second, the Multi-Model Consensus annotation framework in detail (Sections 5.1–5.4), including the dual-LLM labeling protocol, the arbitration mechanism, and the anti-bias design choices—this is the methodological core of the paper that distinguishes it from prior single-model annotation approaches.
Third, the evidence for why MMC is necessary (Figure 2 and Section 5.4), including the systematic labeling biases exhibited by different frontier LLMs and the position bias experiment—this justifies the design choices.
Fourth, the inter-annotator agreement validation (Section 6), which establishes that MMC-produced labels align with human judgment at a "near-perfect" level.
Fifth, the two-stage training pipeline and Eva-4B model architecture (Sections 7.1–7.3), including the rationale for Qwen3-4B as the base model, the specific training hyperparameters, and the three model variants created for ablation analysis.
Sixth, the dataset composition and final statistics (Section 5.5 and Table 3), which provide the concrete scale and coverage numbers that establish EvasionBench as a benchmark.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and methodology paper whose core idea is that multi-model consensus annotation can produce training data of sufficient quality to train a small classifier that matches or exceeds frontier LLMs on a discourse-level pragmatic reasoning task, and that single-model annotation introduces systematic biases that consensus mechanisms can mitigate.

Raw Data Extraction: From Transcripts to Q&A Pairs

The EvasionBench pipeline begins with the S&P Capital IQ Full Text Dataset, a corpus of earnings call transcripts containing 22.7 million potential Q&A interactions collected from 1.38 million individual transcripts featuring 420,000 unique speakers (Section 4.1). These transcripts are not raw text but structured documents with speaker identification and utterance-type annotations, which enables the extraction step.

Stage 1: Q&A Pair Extraction. The extraction process leverages the structured nature of Capital IQ transcripts, where each utterance is tagged with a type code. Analyst questions are identified as Type 3 utterances ("Question – Analyst") and management answers as Type 4 utterances ("Answer – Management"). The extraction logic pairs questions with their immediately following answers based on sequential component ordering—an analyst question followed by a management answer forms one Q&A pair. Operator instructions, pleasantries, and other non-substantive utterances are removed during this step. This structured extraction is a design choice that avoids the error-prone process of inferring turn-taking from raw text, which would be necessary with scraped or audio-transcribed transcripts.

Stage 2: Quality Filtering. The paper applies three concrete filtering criteria to remove low-quality pairs that would introduce noise into the annotation process:

Question validity: The question utterance must contain a question mark ("?"), ensuring it is an actual interrogative rather than a statement misclassified as Type 3.
Answer substance: The answer utterance must exceed 30 characters, eliminating single-word acknowledgments ("Yes.", "No.", "Okay.") that contain no substantive content to evaluate for evasion.
Transcription artifact removal: Pairs containing transcription markers such as [indiscernible], [ph] (phonetic approximation), or [inaudible] are removed, since garbled or uncertain transcription prevents reliable evasion assessment.

Stage 3: Substantial Content Selection. The third filter imposes a combined length requirement: the question and answer together must total at least 500 characters. This threshold serves two purposes. First, very short exchanges are unlikely to contain the multi-sentence context needed for the nuanced intermediate-level evasion detection that is the hardest and most valuable part of the task. Second, this filter removes trivial logistics questions ("Can you hear me?") and other non-analytical exchanges that would inflate the dataset size without contributing to evasion diversity.

Yield. After all three filtering stages, 11.27 million Q&A pairs remain from the original 22.7 million, representing a 49.6% retention rate. This is still an enormous corpus for annotation purposes—the key sampling decision for constructing the labeled dataset is made downstream, not in the filtering stage.

Design rationale for filtering choices. The thresholds (30 characters, 500 combined characters, required question mark) appear ad hoc but serve a specific purpose: they are designed to be simple, deterministic, and reproducible rather than learned, ensuring that the filtering stage does not introduce model biases before annotation begins. A learned filtering model (e.g., using a classifier to predict which Q&A pairs are "interesting") would risk encoding the same kind of systematic biases that the MMC framework is designed to mitigate. Deterministic filtering ensures that the annotation pipeline starts from a transparent, reproducible baseline.

Multi-Model Consensus (MMC) Annotation: Framework Overview

The Multi-Model Consensus framework is the central methodological contribution of the paper. It is not a single model or algorithm but a protocol for combining multiple LLM annotators in a structured pipeline that uses agreement as a signal of annotation reliability and disagreement as a trigger for deeper arbitration. The framework is illustrated in Figure 1 of the paper and operates in three stages:

Stage I: Dual LLM Annotation. Two frontier LLMs independently label each Q&A pair.

Stage II: Consensus Detection. Pairs where both models agree are assigned that label and proceed directly to the training set. Pairs where they disagree are flagged for arbitration.

Stage III: Three-Judge Arbitration. For each disagreement, three judge models independently evaluate which original annotation is more accurate. Majority voting determines the final label.

Design intuition. The key insight is that agreement between heterogeneous models with different inductive biases is a stronger signal of annotation correctness than any single model's prediction, regardless of that model's individual capabilities. If Claude and Gemini independently arrive at the same label, that label is more likely to be correct—not because either model is infallible, but because their errors are likely to be uncorrelated, making agreement an implicit error-correction mechanism. When they disagree, it signals either genuine ambiguity or model-specific bias, both of which warrant additional scrutiny.

This framework draws on multiple established paradigms:

Tri-training (Zhou and Li, 2005) leverages three classifiers' agreement to label unlabeled data, but uses homogeneous classifiers trained on different views of the data.
Self-consistency (Wang et al., 2022) samples multiple reasoning paths from a single model to improve answer quality.
RLAIF (Lee et al., 2023) uses AI feedback to scale preference learning.
LLM-as-judge approaches (Zheng et al., 2023) use strong LLMs to evaluate outputs.

The MMC framework differs from each: unlike tri-training, it uses heterogeneous frontier LLMs rather than homogeneous classifiers; unlike self-consistency, it samples from multiple models rather than multiple paths from one model; unlike RLAIF, it targets classification rather than preference learning; unlike basic LLM-as-judge setups, it incorporates a structured arbitration stage specifically for disagreement cases.

Stage I: Dual LLM Annotation Protocol

The primary annotation is performed by Claude Opus 4.5 and Gemini 3 Flash, two frontier LLMs selected for their strong general reasoning capabilities and for being developed by different organizations (Anthropic and Google, respectively), which increases the likelihood that they have different inductive biases. The choice of exactly two primary annotators (not three, not five) is a deliberate efficiency-quality tradeoff: two annotators provide an agreement signal while minimizing API costs and annotation time, since each additional primary annotator would increase Stage I cost proportionally.

Each model receives a structured prompt containing:

The task definition: classify the manager's response as direct, intermediate, or fully evasive.
The taxonomy definitions with examples, following the operational definitions developed in Section 3.2.
The specific question-answer pair to evaluate.
Instructions to output only the label.

The models are prompted independently with no communication or information sharing between them. This independence is critical: if the models could influence each other (e.g., through sequential prompting where the second model sees the first model's label), the agreement signal would be contaminated by anchoring effects rather than reflecting genuine convergence on the correct label.

Why these two models? The paper does not provide an explicit ablation comparing different primary annotator pairs, but the choice is motivated by practical considerations: Claude Opus 4.5 and Gemini 3 Flash are both state-of-the-art frontier models at the time of the study, they are accessible via commercial APIs, they represent different development philosophies (Anthropic's safety-focused approach versus Google's generalist approach), and their labeling tendencies proved to be complementary rather than redundant (as shown in Figure 2, discussed below).

Stage II: Consensus Detection

After both models label a sample, the labels are compared:

Agreement case: Both models output the same label (e.g., both label the response as "intermediate"). The sample is added to the consensus set with that label. No further processing occurs.
Disagreement case: The models output different labels (e.g., Claude labels "direct," Gemini labels "intermediate"). The sample is added to the disagreement set and proceeds to Stage III arbitration.

The paper reports that 16.1% of total samples (3,645 out of approximately 22,600 samples, based on the 84K training set being constructed from annotated data) fall into the disagreement category. This means 83.9% of samples achieve consensus at Stage I—a relatively high agreement rate that suggests the primary annotators have broadly aligned (though not identical) interpretations of the taxonomy, validating the three-level design as something frontier LLMs can reliably operationalize.

The consensus set becomes the foundation for Stage 1 model training (the 60K consensus-only training split), providing high-confidence labels that the paper's ablation analysis shows are sufficient to train a reasonably strong classifier on their own (Eva-4B Consensus achieves 81.4% Macro-F1, as shown in Table 5).

Stage III: Three-Judge Arbitration

For the 3,645 disagreement cases, the paper deploys a more intensive arbitration mechanism designed to resolve disagreements through structured multi-judge evaluation rather than through simple tie-breaking rules or discarding the samples.

Judge composition. Three models serve as judges: Claude Opus 4.5, Gemini 3 Flash, and GPT-5.2. Notably, the first two are the same models used as primary annotators—they are now being asked to evaluate their own previous annotations alongside the other model's. GPT-5.2 is introduced only at the arbitration stage, providing a third perspective from a model not involved in the original annotation, which could help break ties where Opus and Gemini have persistent biases.

Arbitration prompt. Each judge receives:

The original question-answer pair.
Both original annotations (e.g., "Annotator A: direct; Annotator B: intermediate"), with the model identities not disclosed—they are labeled only as "Annotator A" and "Annotator B."
The taxonomy definitions.
Instructions to select which annotation is more accurate, outputting only "A" or "B."

The judges evaluate independently. The final label is determined by majority voting: if at least two of the three judges select Annotator A's label, that label is assigned; otherwise, Annotator B's label is assigned.

Anti-bias mechanism: randomized presentation order. The paper identifies a specific threat to validity in the arbitration stage: position bias (also called primacy effect in the LLM-as-judge literature, citing Chen et al., 2024). If Annotator A's label is always presented first, judges might systematically favor it regardless of content—this is a known bias where LLMs disproportionately select the first option in a list. To mitigate this, the paper randomizes which model's prediction appears as "Annotator A" versus "Annotator B" for each sample, using a fixed seed of 42 for reproducibility.

The effectiveness of this anti-bias mechanism is validated through a controlled experiment (Section 9, Figure 7). The paper compared fixed-position judging (always Opus first) against randomized-position judging on 5,541 samples:

"randomization increased Opus's win rate from 63.5% to 68.6% (+5.1%), demonstrating substantial position bias in LLM-as-judge settings."

This 5.1 percentage point shift is substantively meaningful: it means that in the fixed-position condition, Opus was systematically penalized when placed second (or Gemini systematically advantaged when placed first), and randomization reveals Opus's actual judged accuracy as higher than the fixed-position measurement would suggest. The existence of this bias validates the randomization design choice—without it, the arbitration stage would systematically favor whichever model happened to be presented first, introducing a new bias into the annotation rather than resolving the existing disagreement.

Why three judges? The paper does not explicitly justify three versus five or other odd numbers, but the design choice follows standard voting logic: an odd number of judges ensures no ties (with majority voting, 2-of-3 always produces a winner). Three judges balances robustness (more than two, so no single tie-breaking judge) with cost (each additional judge linearly increases API costs for the arbitration stage). Given that only 16.1% of samples enter arbitration, the marginal cost of additional judges is limited, but three represents a pragmatic default.

Why majority voting rather than weighted voting? The paper does not weight judges by their perceived accuracy or agreement with human annotators. This is a conservative choice: without a validated weighting scheme (which would require human labels on the disagreement set, creating circularity), equal weighting avoids introducing unvalidated assumptions about which judge is "better." The judges are treated as equally reliable for arbitration purposes.

Evidence for Why MMC is Necessary: Systematic Model Biases

The justification for the MMC framework rests on empirical evidence that individual LLMs exhibit systematically different labeling tendencies that would produce biased datasets if used alone. This evidence is presented in Figure 2 (Section 5.4), which shows the label distribution across the three judge models on the 3,645 disagreement samples:

Claude Opus 4.5: 53.3% direct, 36.2% intermediate, 10.5% fully evasive
Gemini 3 Flash: 44.4% direct, 32.1% intermediate, 23.5% fully evasive
GPT-5.2: 32.2% direct, 56.7% intermediate, 11.1% fully evasive

These distributions reveal three distinct labeling "personalities":

Claude Opus 4.5 is the most conservative: It assigns "direct" to a majority (53.3%) of the samples that the other models find ambiguous enough to disagree on. It almost never labels something "fully evasive" (10.5%) compared to Gemini. This suggests Opus has a high bar for what counts as evasion—it gives executives the benefit of the doubt and requires strong evidence of non-responsiveness before labeling something evasive.

Gemini 3 Flash is the strictest: It assigns "fully evasive" to 23.5% of disagreement samples—more than double Opus's rate and more than double GPT-5.2's rate. It assigns "direct" less often than Opus (44.4% vs. 53.3%). This suggests Gemini has a lower bar for detecting evasion and is more willing to classify responses as fully non-responsive.

GPT-5.2 is the most ambiguous: It assigns "intermediate" to a majority (56.7%) of disagreement samples, far more than either Opus (36.2%) or Gemini (32.1%). GPT-5.2 rarely goes to either extreme—it makes the fewest "direct" assignments and the fewest "fully evasive" assignments, concentrating heavily in the middle category. This suggests GPT-5.2 is more reluctant to make definitive judgments and defaults to the intermediate category when the signal is ambiguous.

Implications for single-model annotation. These biases are not subtle. A dataset labeled solely by Opus would be heavily skewed toward "direct" labels—underestimating the prevalence of evasion. A dataset labeled solely by Gemini would be skewed toward "fully evasive"—overestimating it. A dataset labeled solely by GPT-5.2 would produce an ambiguous distribution dominated by "intermediate" labels that might not reflect genuine intermediate evasion prevalence but rather a model-specific tendency to hedge. None of these distributions is necessarily "correct" in isolation—the truth likely lies somewhere in their intersection.

This finding is the paper's strongest empirical argument for multi-model consensus. If all three models produced similar distributions, then single-model annotation might be sufficient (any individual model would be a reasonable proxy for the consensus). The fact that they systematically diverge demonstrates that individual LLMs have implicit thresholds for evasion detection that are not aligned, and that combining them through a consensus mechanism is necessary to cancel out these idiosyncratic biases.

Evidence for Position Bias and the Anti-Bias Mechanism

In addition to the labeling tendency differences shown in Figure 2, the paper provides a separate experiment (Section 9, Figure 7) that validates the need for the anti-bias mechanism in the arbitration stage. The experiment compares two conditions on 5,541 samples:

Fixed position: Claude Opus 4.5 is always presented as "Annotator A" (first) and Gemini 3 Flash as "Annotator B" (second).
Randomized position: The order is randomized per sample (seed 42).

The results show:

"randomization increased Opus's win rate from 63.5% to 68.6% (+5.1%), demonstrating substantial position bias in LLM-as-judge settings."

Interpretation. In the fixed-position condition, Opus (always first) wins 63.5% of the time. When position is randomized, Opus wins 68.6% of the time—meaning that in the fixed condition, when Opus happened to be placed second on some samples, it was systematically penalized relative to its true performance. The 5.1 percentage point gap is the magnitude of position bias: being presented first confers an advantage of approximately 5 percentage points in win rate.

This finding matters for two reasons. First, it validates the randomization design choice as necessary rather than optional—without it, the arbitration stage would systematically favor whichever model was presented first, introducing a new bias rather than resolving existing disagreements. Second, it provides concrete evidence for the LLM-as-judge bias literature (Chen et al., 2024), quantifying the effect in a real annotation pipeline rather than in a controlled lab setting.

Why use seed 42 for randomization? The paper specifies that randomization uses seed 42 for reproducibility. This is a deliberate transparency choice: anyone replicating the arbitration stage with the same seed will get the same assignments, making the procedure deterministic-reproducible despite involving randomization.

Inter-Annotator Agreement Validation

The paper validates the quality of MMC-produced labels against human judgment through inter-annotator agreement (IAA) measurement (Section 6). The procedure is:

A second human annotator independently labels a balanced subset of 100 samples drawn from the Gold 1K evaluation set (the set that had been labeled by the primary human annotator).
Cohen's Kappa is computed between the two human annotators' labels.

The results (Table 4) show:

"Total Samples: 100, Agreement Count: 89 (89.0%), Cohen's Kappa: 0.835, Macro-F1: 88.99%"

Cohen's Kappa of 0.835 falls in the "Almost Perfect" agreement range according to the widely-used Landis and Koch (1977) interpretation scale (0.81–1.00). This is a strong result: it means that two independent human experts agree on the evasion label at a rate far above chance, validating that the three-level taxonomy is operationalizable by humans with domain expertise.

A critical nuance: this IAA is between two humans, not between MMC and humans. The paper measures human-human agreement, not model-human agreement directly. The MMC labels are used to train Eva-4B, and Eva-4B is evaluated against the human-labeled Gold 1K set. The Cohen's Kappa of 0.835 establishes that the gold labels themselves are reliable (humans agree with each other), which indirectly validates the benchmark but does not directly measure MMC-human agreement. To directly validate MMC labels, one would need to compare the MMC-produced labels for the Gold 1K samples against the human labels—a comparison the paper does not report. Instead, the evaluation is indirect: Eva-4B, trained on MMC labels, achieves 84.9% Macro-F1 against human labels, implying that the MMC labels are close enough to human judgment to enable strong performance.

The intermediate class is the primary source of disagreement. The paper notes that:

"10 of 11 disagreements involve the intermediate class, confirming it as the most ambiguous category."

This is an important diagnostic. The direct and fully evasive categories are relatively clear-cut—both humans and models can reliably distinguish a clearly direct answer from a clearly evasive one. The intermediate category is where genuine ambiguity resides, and it is where human annotators most often disagree with each other. This finding validates the taxonomy design: if humans rarely disagreed on the boundaries between direct and intermediate or intermediate and fully evasive, it would suggest the categories are too coarse and could be split further. The fact that intermediate is the locus of disagreement suggests that the three levels capture a real gradation where the middle is genuinely harder to define precisely than the extremes.

Dataset Composition and Statistics

The final EvasionBench dataset comprises two components with distinct purposes (Section 5.5, Table 3):

Training set: 84,000 labeled Q&A pairs. This is further split into:

Train-60K: 60,000 samples drawn from the consensus set—cases where the two primary annotators agreed. These represent high-confidence labels. The set covers 6,943 unique companies with average question length of 77.5 words and average answer length of 149.8 words.
Train-24K: 24,000 samples from the disagreement set after three-judge arbitration—cases where the primary annotators disagreed and were resolved through majority voting. These represent harder boundary cases with labels assigned through the arbitration process. The set covers 4,597 unique companies with average question length of 70.0 words and average answer length of 136.3 words.

Combined, the training set covers 8,081 unique companies (the sets overlap in company coverage) with overall average question length of 75.4 words and average answer length of 145.9 words. These length statistics confirm that the filtering pipeline successfully selected for substantive exchanges—these are not short one-sentence interactions but multi-paragraph responses to detailed analyst questions.

Evaluation set: Gold 1K. 1,000 samples with expert human labels, covering 319 unique companies with average question length of 72.0 words and average answer length of 131.3 words. The slightly shorter average lengths compared to the training set likely reflect natural variation in the sampling process.

All splits are balanced: Each split is constructed to contain exactly 33.3% direct, 33.3% intermediate, and 33.3% fully evasive samples. This is an explicit design choice that prevents classifiers from achieving high accuracy through majority-class prediction and ensures that per-class metrics (F1-D, F1-I, F1-E in Table 5) reflect genuine discriminative ability rather than class imbalance effects. In the wild, evasion categories are almost certainly not balanced—direct answers likely predominate—but a balanced benchmark is standard practice for establishing a model's fundamental capability on each class before dealing with real-world distribution shift.

Temporal coverage: 2002–2022. The training data spans two decades of earnings calls, providing broad coverage of different economic conditions (expansion, recession, recovery), regulatory environments (pre- and post-Sarbanes-Oxley, Dodd-Frank), and communication norms. This temporal breadth is valuable for training models that should generalize across different eras of corporate communication rather than overfitting to a narrow time window.

Two-Stage Training Pipeline for Eva-4B

The Eva-4B classifier is trained using a two-stage fine-tuning approach designed to leverage the structure of the MMC-produced training data: high-confidence consensus samples first, then the harder boundary cases resolved through arbitration.

Base model selection: Qwen3-4B-Instruct-2507. The paper selects Qwen3-4B (Qwen Team, 2025) for three stated reasons (Section 7.1):

Strong instruction-following capability: The model achieves 83.4% on IFEval, a benchmark specifically designed to measure how well models follow instructions. For a classification task where the model must output structured labels according to a specific taxonomy, instruction-following is more relevant than raw knowledge or reasoning ability.
Efficient parameter count (4 billion): A 4B model is deployable on consumer-grade hardware (single GPU with reasonable memory), making the resulting classifier practically useful for researchers and practitioners who cannot run frontier-scale models.
Open-source availability: Using an open-source base model ensures reproducibility—anyone can replicate the fine-tuning process without API access or commercial licensing.

The choice of Qwen3-4B over other 4B-class models (such as Phi-3-mini or Llama-3-4B) is not ablated in the paper, so we cannot attribute Eva-4B's performance specifically to Qwen3-4B's architecture versus its training data quality. The paper's implicit claim is that the MMC annotation quality, not the specific base model, is the primary driver of performance—a claim partially supported by the ablation showing that different training data (consensus vs. full vs. Opus-only) produces substantially different performance from the same base model.

Stage 1: Consensus Training (60K samples). The base Qwen3-4B model undergoes full fine-tuning (all parameters updated) on the 60K consensus training set—samples where Claude Opus 4.5 and Gemini 3 Flash independently agreed on the label. The training configuration (Table 6, Appendix A):

Framework: MS-SWIFT (a fine-tuning framework for efficient LLM training; specific version not specified)
Training type: Full fine-tuning (not LoRA or other parameter-efficient methods, since the model is small enough at 4B parameters to fully fine-tune with reasonable resources)
Dataset size: 60,000 samples
Epochs: 2 (full passes through the training data)
Learning rate: $2 \times 10^{-5}$ (2e-5), a standard value for LLM fine-tuning that balances convergence speed with stability
Batch size (per GPU): 8
Gradient accumulation steps: 2 (effectively multiplying the batch size)
Effective batch size: 32 (8 per GPU × 2 accumulation steps × 2 GPUs? The paper does not specify the number of GPUs; the effective batch size of 32 is computed as per-GPU batch size × gradient accumulation steps × number of GPUs, but the number of GPUs is not explicitly stated)
Maximum sequence length: 2,500 tokens (accommodating the combined question and answer text, which averages ~220 words = ~300 tokens, plus instruction template tokens)
Precision: bfloat16 (half-precision floating point, reducing memory usage by half compared to float32 with minimal impact on training stability)
Warmup ratio: 3% (the learning rate linearly increases from 0 to 2e-5 over the first 3% of training steps, then decays according to a schedule not specified)

This stage produces Eva-4B (Consensus), which serves as both a standalone model for ablation comparison and the initialization for Stage 2.

Stage 2: Judge-Refined Training (24K samples). The consensus-trained model is further fine-tuned on the 24K training set containing both additional consensus samples and the arbitrated disagreement cases. The training configuration mirrors Stage 1 with a few adjustments:

Base model: Eva-4B (Consensus)—the model is checkpoint-loaded from Stage 1, not re-initialized
Dataset size: 24,000 samples
Epochs: 2
Learning rate: $2 \times 10^{-5}$ (same as Stage 1)
Batch size (per GPU): 8
Gradient accumulation steps: 1 (effectively halving the effective batch size relative to Stage 1, though the per-GPU count may differ)
Effective batch size: 32 (likely achieved with 4 GPUs × 8 per GPU × 1 accumulation, though GPU count is not specified)
Maximum sequence length: 2,048 tokens (slightly reduced from Stage 1's 2,500)
Precision: bfloat16
Warmup ratio: 3%

The key variation between Stage 2 model variants is the label source for the 24K samples:

Eva-4B (Full): The 24K samples use labels from the three-judge majority voting arbitration. This is the complete MMC pipeline.
Eva-4B (Opus Only): The 24K samples use labels from Claude Opus 4.5 only—introducing the systematic bias that MMC is designed to eliminate. This serves as the critical ablation for measuring the value of multi-model consensus.

Why two stages? The paper does not explicitly justify the two-stage design, but the logic is implicit in the data structure. The consensus samples are high-confidence and likely easier to classify—they represent cases where two powerful models independently arrived at the same conclusion. Training on these first establishes a strong baseline understanding of the taxonomy and the linguistic correlates of each evasion level. The disagreement samples are harder—they represent cases where even frontier models couldn't agree, involving subtle distinctions at category boundaries. Introducing these after the model has already learned the basic categories (from Stage 1) allows the model to refine its decision boundaries on the hard cases without them dominating the early training signal and potentially confusing the model about what each category means. This is analogous to curriculum learning: easy examples first, hard examples later.

Eva-4B Model Variants for Ablation Analysis

The paper creates three Eva-4B variants to isolate the contribution of different components in the training pipeline (Section 7.3):

Eva-4B (Consensus): Trained only in Stage 1 on the 60K consensus samples. Represents the performance achievable with high-confidence, dual-model-agreement labels only, without any of the arbitrated boundary cases. This variant tests the hypothesis that consensus filtering alone (discarding disagreement samples entirely) could be sufficient—that the disagreement samples are too noisy to provide useful training signal.

Eva-4B (Opus Only): Trained in Stage 1 (same as Consensus) then continued in Stage 2 on 24K samples labeled by Claude Opus 4.5 only. This variant tests the value of multi-model arbitration: if single-model labels (even from a powerful model) are as good as consensus labels, then Eva-4B (Opus Only) should perform similarly to Eva-4B (Full). If single-model labels introduce systematic bias that degrades training, Opus Only should underperform Full.

Eva-4B (Full): Trained in Stage 1 then continued in Stage 2 on 24K samples with three-judge majority voting labels. This represents the complete MMC pipeline and is the primary Eva-4B model whose performance is reported against frontier models.

What the ablation measures. The comparison between Full and Opus Only is the paper's core methodological test. It directly measures whether the consensus mechanism provides value beyond simply using the predictions of a single strong model. The +4.3 percentage point Macro-F1 advantage for Full over Opus Only (84.9% vs. 80.6%, Table 5) is the headline result that validates the MMC framework—it demonstrates that the arbitration process produces training labels that are qualitatively different from and superior to single-model labels.

Training Dynamics: Evidence of Label Quality from Loss Curves

The paper provides a striking visualization of training dynamics in Figure 5, which compares the loss curves of Eva-4B (Full) and Eva-4B (Opus Only) during Stage 2 training. The loss is presumably cross-entropy loss for the three-way classification task (though the exact loss function is not specified in the paper).

The results show a dramatic divergence:

"Eva-4B (Full) achieves a final loss of 0.007, while Eva-4B (Opus Only) converges to 0.56—an 80× difference."

Interpretation. The loss value measures how well the model's predicted probability distribution matches the target labels. A loss of 0.007 is extremely low—it means the model is nearly perfectly predicting the training labels, with very high confidence assigned to the correct class for almost every example. A loss of 0.56 is substantially higher—the model is making predictions that are significantly misaligned with the training labels.

The fact that Eva-4B (Full) can achieve near-zero loss while Eva-4B (Opus Only) plateaus at 0.56, despite both being trained on the same base model and the same dataset size, implies that the Opus-only labels contain inconsistencies that prevent the model from fitting the data perfectly. In other words, there are examples in the Opus-only training set that are labeled in ways that are inconsistent with each other—two very similar Q&A pairs might receive different labels because Opus's judgments have noise or systematic drift across the dataset. The MMC process appears to produce labels that are more internally consistent, allowing the model to fit them with greater precision.

This is strong indirect evidence for the MMC framework's value. It suggests that:

Claude Opus 4.5's single-model labels are noisy in ways that are not just random but systematically inconsistent—the model cannot simultaneously fit all training examples because the labels contain conflicting signals.
Three-judge arbitration resolves these inconsistencies, producing labels that form a more coherent pattern that the model can learn with high fidelity.
The benefit of MMC is not just about individual label accuracy (getting more individual examples right) but about dataset-level consistency—producing a training set where similar examples receive similar labels, enabling the model to learn generalizable patterns rather than memorizing contradictory examples.

Caveat about overfitting. A loss of 0.007 is suspiciously low and raises the possibility of overfitting to the training data—the model might be memorizing training examples rather than learning generalizable features. However, the strong performance on the held-out Gold 1K evaluation set (84.9% Macro-F1, Table 5) suggests that this low training loss reflects genuine learning of the classification task rather than mere memorization. If the model were overfitting, we would expect high training-set performance but low held-out performance—the opposite of what is observed (84.9% on Gold 1K is the highest performance among all evaluated models).

Dataset Construction from MMC Outputs

The final step in the technical pipeline is the assembly of the labeled datasets from the MMC annotations. The paper describes this implicitly through the training split descriptions (Section 5.5) and the training pipeline (Section 7.2):

From annotations to balanced splits. The MMC framework produces a label for each annotated Q&A pair. The paper then constructs balanced datasets by sampling evenly from the three classes (direct, intermediate, fully evasive). This means the sampling process is non-uniform with respect to the original annotation distribution—if the annotated data naturally contains more direct labels than evasive ones (as suggested by Figure 2's distributions), the balanced sampling oversamples the minority classes and undersamples the majority class.

The specific sampling procedure is not detailed, but the result is:

Train-60K: 20,000 per class from consensus samples
Train-24K: 8,000 per class from arbitrated (or Opus-only) samples
Gold 1K: 333–334 per class with human labels

Why balance? The paper's balanced design is a methodological choice that trades ecological validity for evaluation rigor. In real earnings calls, the true distribution of evasion levels is almost certainly unbalanced—direct answers likely predominate. A classifier trained and evaluated on a balanced distribution will not directly generalize to this real distribution (its precision and recall will shift depending on the mismatch). However, balanced evaluation is standard practice for benchmark construction because:

It prevents trivial baselines (always predicting the majority class) from achieving apparently strong performance.
It ensures that per-class metrics reflect the model's ability on each category equally, rather than being dominated by performance on the most common class.
It makes the benchmark harder—a model must genuinely discriminate between classes rather than exploiting class distribution cues.

The tradeoff is that Macro-F1 on a balanced set may overestimate real-world performance if the real distribution is highly skewed. This is an acknowledged limitation that the paper does not explicitly discuss but is standard for benchmark design.

Company and temporal diversity. The 8,081 unique companies in the training set and 319 in the Gold 1K set ensure that models are evaluated across a wide range of industries, company sizes, and communication styles. The temporal span (2002–2022) covers multiple economic cycles, regulatory regimes, and evolving disclosure norms. This diversity is important for establishing that Eva-4B's performance generalizes beyond a narrow slice of the earnings call domain, though the paper does not provide per-industry or per-year breakdowns that would allow direct assessment of generalization.

Summary of Design Choices and Their Justifications

Three-level taxonomy over binary (Gow et al., 2021) or seven-level (Nuaimi et al., 2025): empirically validated through IAA—five levels produced $\kappa < 0.5$ , three levels restored $\kappa = 0.83$ . The intermediate category captures the practically important distinction between partial answers and complete deflection.
Dual primary annotators (Claude Opus 4.5, Gemini 3 Flash) over single-model annotation: provides an agreement signal and mitigates the systematic biases documented in Figure 2, where individual models have fundamentally different evasion thresholds.
Three-judge arbitration over single-judge tie-breaking or discarding disagreement samples: majority voting among three heterogeneous judges reduces the impact of any individual judge's bias, while odd-number voting prevents ties. Including GPT-5.2 only at the arbitration stage provides a fresh perspective from a model not involved in the original annotation.
Randomized presentation order with seed 42 over fixed ordering: the 5.1 percentage point position bias effect documented in Figure 7 validates that randomization is necessary to prevent primacy effects from distorting the arbitration process.
Two-stage training (consensus first, then arbited samples) over single-stage combined training: curriculum learning hypothesis—consensus samples are easier, establishing basic category distinctions before introducing harder boundary cases from the disagreement set.
Full fine-tuning over parameter-efficient methods (LoRA): at 4B parameters, full fine-tuning is computationally feasible and avoids the potential performance gap between full and parameter-efficient training.
Balanced dataset construction over natural distribution: standard benchmark design choice that prevents majority-class baselines and ensures per-class metrics are informative.
Gold 1K with human labels rather than held-out LLM labels for evaluation: prevents circular evaluation where models are tested on data labeled by similar models; establishes ground truth against human expert judgment.

4. Key Insights and Innovations

Innovation 1: Multi-Model Consensus as a Bias-Cancellation Mechanism, Not Just a Quality Filter

The dominant assumption in LLM-based annotation is that using a single strong model produces labels that are "good enough" — that quality scales with model capability, and that a sufficiently capable model (like Claude Opus 4.5 or GPT-5.2) can serve as a reliable oracle. Under this view, annotation pipelines are designed around filtering: identify which samples the model is uncertain about and discard them or escalate to humans, while accepting the model's high-confidence predictions as correct.

This paper demonstrates that this assumption is fundamentally wrong for tasks involving graded, discourse-level judgment. The evidence is Figure 2: Claude Opus 4.5, Gemini 3 Flash, and GPT-5.2 produce systematically different label distributions on the same data — Opus is conservative (53% direct), Gemini is strict (24% fully evasive), GPT-5.2 is ambiguous (57% intermediate). These are not random errors that cancel with more samples; they are stable, model-specific biases that would be baked into any single-model-labeled dataset regardless of scale. A 100K-sample dataset labeled by Opus would systematically undercount fully evasive responses; one labeled by Gemini would systematically overcount them. The bias is structural, not statistical.

The intellectual move is reframing multi-model consensus from a quality-assurance mechanism (averaging out random noise) to a bias-cancellation mechanism (canceling out systematic, model-specific interpretive tendencies). This is a fundamentally different justification for ensemble annotation. Prior work on LLM-based annotation (Tan et al., 2024) treats consensus as improving reliability — a second opinion confirms the first. The MMC framework treats consensus as surfacing and canceling bias — disagreement reveals where models' implicit thresholds diverge, and the three-judge arbitration with models of different biases produces labels that are not merely "more reliable" but qualitatively different from any single model could produce.

The evidence that this matters comes from two sources. First, the training dynamics in Figure 5: Eva-4B (Full) trained on consensus labels achieves a loss of 0.007 versus Eva-4B (Opus Only) plateauing at 0.56 — an 80× difference. A loss of 0.56 on a three-way classification task means the model cannot fit its own training data because the labels contain internal contradictions. Second, the +4.3 percentage point Macro-F1 gap between Full and Opus Only (Table 5: 84.9% vs. 80.6%) means that the arbitration-produced labels generalize better to human judgment than single-model labels — the consensus mechanism is not just a training convenience, it produces labels that are closer to the ground truth.

This insight has implications beyond evasion detection. Any annotation task where individual annotators (human or LLM) have stable but divergent interpretive thresholds — detecting toxicity, assessing argument quality, identifying misinformation — would benefit from a bias-cancellation framing over a simple quality-filtering framing. The paper provides a template for how to diagnose this (compare label distributions across models on disagreement samples) and a mechanism for addressing it (heterogeneous judge panel with randomized presentation order).

The distinction from prior work is important. Gow et al. (2021) used human annotators and achieved reliability through expert training and clear guidelines — but couldn't scale. Nuaimi et al. (2025) used a single LLM annotator and achieved scale — but the Figure 2 evidence suggests their labels likely encode systematic model-specific bias. The MMC framework aims for both scale and unbiasedness by exploiting the fact that different frontier models have different biases, and their intersection is closer to the truth than any individual model's output.

Innovation 2: Evasion Detection as a Pragmatic Reasoning Task, Not a Text Classification Task

Prior work on detecting problematic language in financial communication has predominantly operated at the lexical and sentence level. Sentiment analysis extracts tone from word choices (Loughran and McDonald, 2011; Araci, 2019). Deception detection identifies linguistic markers like reduced self-reference or increased negative emotion words (Newman et al., 2003; Larcker and Zakolyukina, 2012). Tone management research (Huang et al., 2014) examines how managers strategically adjust language to influence perception.

The paper reframes evasion detection as a fundamentally different kind of task: discourse-level pragmatic reasoning. The key distinction is that evasion cannot be detected by examining the response in isolation — it requires assessing whether the response addresses the specific question that was asked. This is a relational judgment (does A satisfy B's communicative intent?) rather than an intrinsic property judgment (what tone does A have?).

The paper grounds this reframing explicitly in Gricean pragmatics (Grice, 1975), specifically the Maxim of Relation ("Be relevant"). The three-level taxonomy maps directly to degrees of maxim violation: full adherence (direct), partial violation (intermediate), and complete violation (fully evasive). This is not just theoretical window-dressing — it has concrete design consequences:

The question-answer pair is the unit of analysis, not the response alone. This makes the task fundamentally different from document-level classification — the input is inherently relational.
The taxonomy captures discourse function, not surface form. Hedging language ("I think," "we expect") can appear in both direct and intermediate responses — it's a surface feature that correlates with but does not define evasion. The actual criterion is whether the informational content satisfies the question's specific ask.
The intermediate category is theoretically necessary, not just empirically convenient. Gricean pragmatics recognizes that maxim violations exist on a spectrum — partial relevance is a genuine communicative phenomenon distinct from both full relevance and complete irrelevance.

This reframing explains several empirical findings that would be puzzling under a text-classification view:

Why the intermediate class is hardest for everyone. Both human annotators (10 of 11 disagreements involve intermediate) and all models (F1-I is the lowest per-class metric for every model in Table 5) struggle most with the intermediate category. Under a text-classification view, intermediate is just the middle of a distribution — it should be no harder than the extremes. Under a pragmatic reasoning view, intermediate is genuinely more ambiguous because it requires assessing whether a response that is partially relevant crosses the threshold into "not actually answering the question" — a judgment that depends on interpreting the question's communicative intent, not just matching keywords.

Why adjacent-class confusion dominates errors. The confusion matrix (Figure 6) shows that 95.4% of Eva-4B errors are between adjacent classes (direct↔intermediate, intermediate↔fully evasive). This is exactly what a graded pragmatic phenomenon would predict — errors cluster at category boundaries, not randomly across the taxonomy. A model that fundamentally misunderstood the task would produce more extreme errors (direct↔fully evasive confusion), which account for only 4.6% of errors.

Why hedging language causes over-prediction of evasion. The paper's qualitative error analysis (Section 9) identifies that the dominant error type (52%) involves classifying direct responses as intermediate because they contain hedging language ("we do expect," "soon"). Under a text-classification view, this is a model failure — the hedging pattern triggers the "intermediate" class incorrectly. Under a pragmatic reasoning view, this is a fundamental ambiguity: hedging language is a legitimate signal of reduced commitment, but reduced commitment does not necessarily mean the question wasn't answered. The model has learned that hedging correlates with intermediate responses (which it does, in the aggregate) but cannot distinguish between hedging-that-modifies-answer-content and hedging-that-replaces-answer-content. This is a genuinely hard pragmatic distinction that goes beyond pattern matching.

This reframing is significant because it positions evasion detection as a testbed for evaluating whether NLP systems can perform the kind of discourse-level reasoning that has been theorized in pragmatics for decades. Sentiment analysis demonstrated that models can classify sentence-level affect. Question answering demonstrated that models can extract factual information from text. Evasion detection requires something different: assessing whether one utterance fulfills the communicative obligations created by another utterance. This is a step toward evaluating models' pragmatic competence, not just their semantic or syntactic competence.

Innovation 3: Training Data Consistency as a First-Class Optimization Target, Not a Side Effect of Quality

The standard narrative in NLP is that better training data means more accurate individual labels — each sample is more likely to be correctly labeled, and the aggregate effect is better model performance. The paper's training dynamics (Figure 5) suggest a different and more specific mechanism: the value of MMC annotations is not primarily about per-sample accuracy but about dataset-level consistency.

The evidence is the loss curve divergence. Eva-4B (Full) converges to a training loss of 0.007 — near-perfect fit. Eva-4B (Opus Only) plateaus at 0.56 — substantially worse fit. Both models have the same capacity (same architecture, same parameter count), the same training procedure, and the same dataset size. The only difference is the labels. The fact that one model can fit its training data nearly perfectly while the other cannot means that the Opus-only labels contain internal contradictions — similar Q&A pairs receive different labels because Opus's judgments have systematic noise or drift.

This is a distinct claim from "MMC labels are more accurate on average." It is specifically that MMC labels form a more internally coherent pattern that a model can learn with high fidelity. A single human annotator might produce labels that are accurate on average but inconsistent — labeling the same type of hedging response as "intermediate" on Monday and "direct" on Tuesday due to fatigue, context effects, or shifting decision thresholds. The MMC framework, by requiring agreement between models with different biases, effectively filters out these inconsistencies — a label that survives the consensus and arbitration process is one that multiple heterogeneous judges independently converged on, which means it is likely to be stable across similar cases.

This insight reframes the annotation quality problem from a point-estimation problem (what is the correct label for this specific sample?) to a consistency problem (do similar samples receive similar labels?). The two are related — inconsistent labels are definitionally inaccurate for at least some samples — but the optimization target shifts. Under the point-estimation view, you improve annotation by getting better annotators or better guidelines. Under the consistency view, you improve annotation by designing processes that surface and resolve contradictions — exactly what the MMC arbitration stage does by forcing three judges to evaluate the same borderline case and produce a majority-voted label.

The practical implication is significant for anyone constructing training datasets using LLM annotators. The standard practice is to filter by confidence (discard low-confidence samples, keep high-confidence ones) or to use a single strong model. The Figure 5 evidence suggests that even a very strong model (Claude Opus 4.5) produces labels that are internally inconsistent enough to prevent convergence to low loss. Multi-model consensus is not just a way to get slightly better labels — it is a way to get labels that form a learnable pattern, which is a higher bar.

The comparison with Nuaimi et al. (2025) is instructive here. Their single-model annotation produced a dataset where the labels likely contain the same kind of internal inconsistencies that prevent Eva-4B (Opus Only) from converging below 0.56 loss. The problem is not that any individual label is "wrong" — it's that the labeling function is not smooth in the input space, so the model cannot learn a generalizable decision boundary. The MMC framework trades off some scale (requiring multiple models per sample) against this learnability, and the 80× loss difference suggests this is a favorable tradeoff for training downstream classifiers.

Innovation 4: The Evasion Taxonomy as Empirically Calibrated, Not Theoretically Imposed

Taxonomy design in NLP often follows one of two patterns: either theory-driven (impose categories from linguistic or psychological theory, then test whether annotators can use them) or data-driven (cluster examples and name the clusters post-hoc). EvasionBench's three-level taxonomy represents a third approach: empirically calibrated theory — start with a theoretically-motivated scale, measure whether humans can operationalize it reliably, and adjust the granularity based on empirical agreement rates.

The paper is unusually transparent about this calibration process. The initial design attempted a five-level scale, following the intuition that finer granularity would capture more nuanced evasion strategies (building on the seven-type psychological taxonomy of Nuaimi et al., 2025). This failed empirically: "Pilot annotation with five levels yielded low inter-annotator agreement (κ < 0.5)." The decision to collapse to three levels was not a theoretical compromise — it was a response to the empirical finding that human annotators could not reliably distinguish more than three degrees of evasion.

This calibration approach has several intellectual implications:

It provides a principled answer to "how many categories?" The number of categories is determined by where human annotators can achieve near-perfect agreement (κ > 0.8), not by theoretical desiderata. This is a defensible, replicable criterion that other taxonomy designers can adopt: start with the finest granularity you think might matter, measure agreement, and coarsen until agreement reaches an acceptable threshold.

It validates the taxonomy against human cognition, not just theory. The Gricean framing (Section 3.2) provides theoretical grounding for why three levels make sense — they map to degrees of Relation maxim violation. But the paper does not simply assert this mapping; it demonstrates that humans can actually operationalize it with high agreement (κ = 0.835). The theory and the empirics mutually reinforce each other: the theory predicts that graded relevance judgments should be possible, and the IAA results confirm that they are.

The intermediate category is justified by both theory and practice. Binary taxonomies (Gow et al., 2021) collapse the intermediate and fully evasive categories. The paper argues this "loses the distinction between 'partial answer' and 'complete deflection' — information critical for downstream applications like investor alerting." The IAA results support this: while intermediate is the locus of most disagreements, the overall κ of 0.835 means that humans can distinguish intermediate from both direct and fully evasive at well above chance levels. The category captures a real communicative phenomenon that binary classification would obscure.

This calibration methodology is a methodological contribution independent of the specific evasion domain. The pattern — start with fine-grained theory-derived categories, measure agreement, coarsen empirically — provides a template for taxonomy construction in any domain where category boundaries are inherently graded (toxicity severity, argument quality, factuality of claims). It avoids both the rigidity of purely theory-driven taxonomies and the post-hoc rationalization of purely data-driven ones.

Innovation 5: The Small-Model-on-High-Quality-Data Result as a Statement About Annotation Methodology, Not Model Architecture

Eva-4B is a 4-billion-parameter model fine-tuned from Qwen3-4B using standard full fine-tuning with no architectural innovations. The paper could have positioned its performance — 84.9% Macro-F1, surpassing Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash (Table 5) — as a demonstration that small, domain-specific models can match or exceed frontier LLMs on specialized tasks. This would be a standard (and valid) narrative in the "small models can compete" literature.

But the paper's deeper intellectual move is different. The Eva-4B result is positioned as evidence for the quality of the MMC annotation pipeline, not evidence for the quality of the model architecture or training procedure. The +4.3 percentage point gap between Eva-4B (Full) and Eva-4B (Opus Only) — trained identically except for the label source in Stage 2 — isolates the annotation methodology as the causal factor. The performance gap between Eva-4B (Full) and frontier models like GPT-5.2 (84.9% vs. 80.9%) further demonstrates that training on high-quality consensus data can compensate for a ~100× or greater parameter count difference.

This inverts the typical contribution hierarchy in ML papers. The model is not the contribution — it is the assay that measures the contribution (the annotation pipeline). The paper's release includes the dataset, the annotation framework, and the model, in that order of importance. Eva-4B is evidence that MMC annotation produces training data of sufficient quality to make a small classifier competitive with frontier models on a discourse-level reasoning task. If the annotation quality were mediocre, Eva-4B would not outperform GPT-5.2 regardless of fine-tuning.

The contrast with Nuaimi et al. (2025) sharpens this point. They also use LLM annotation and train classifiers on the resulting data, but their single-model annotation pipeline likely produces the kind of inconsistent labels that prevent convergence to low loss (à la Eva-4B Opus Only in Figure 5). The EvasionBench contribution is not "we trained a better classifier" but "we built an annotation methodology that produces consistent-enough labels to train an effective classifier at modest scale."

This framing has implications for how the field should evaluate benchmark papers. The standard evaluation is to rank competing models on the benchmark and see which performs best. But if the annotation pipeline is the primary contribution, the relevant evaluation is not just model ranking but also annotation quality diagnostics — inter-annotator agreement, training dynamics, and ablation of annotation design choices. The paper provides all three (IAA in Section 6, loss curves in Figure 5, full/consensus/opus-only ablation in Figure 4), modeling what comprehensive annotation validation looks like beyond just reporting a benchmark leaderboard.

5. Experimental Analysis

Evaluation Methodology

Dataset. The Gold 1K evaluation set, consisting of 1,000 Q&A pairs with expert human labels, balanced across the three evasion classes (33.3% each). These samples are drawn from 319 unique companies spanning 2002–2022, with average question length of 72.0 words and average answer length of 131.3 words (Table 3).
Base model(s). The primary trained model is Eva-4B, a full fine-tune of Qwen3-4B-Instruct-2507 (Qwen Team, 2025), selected for its strong instruction-following (IFEval: 83.4%) and efficient 4B parameter count enabling practical deployment (Section 7.1). The evaluation also includes the untuned Qwen3-4B base model to measure the improvement from fine-tuning.
Metrics. The primary metric is Macro-F1, computed as the unweighted average of per-class F1 scores (F1-D for Direct, F1-I for Intermediate, F1-E for Fully Evasive), providing a balanced assessment across all three classes regardless of their prevalence in the evaluation set. Accuracy is also reported. Per-class F1 scores are included separately in Table 5 to enable diagnosis of which categories are hardest for each model.
Baselines. Twelve models are evaluated in total (Table 5, expanded in Appendix B Table 7):
- Closed-source frontier models: Claude Opus 4.5, GPT-5.2, Gemini 3 Flash — representing the state of the art in general-purpose LLMs evaluated zero-shot on the evasion taxonomy.
- Open-source models: GLM-4.7, Qwen3-Coder, MiniMax-M2.1, Kimi-K2, DeepSeek-V3.2 — providing coverage across different model families, scales, and training paradigms.
- Eva-4B variants: Eva-4B (Full) — the complete MMC pipeline; Eva-4B (Consensus) — Stage 1 only; Eva-4B (Opus Only) — Stage 1 + Stage 2 with single-model labels. These three variants constitute the ablation study.
- Base model: Qwen3-4B before any fine-tuning, establishing the lower bound and measuring the contribution of training.
Generation budget / compute accounting. All models are evaluated on the same fixed 1,000-sample test set. There is no variable generation budget or test-time compute scaling in this evaluation — each model produces a single classification (direct, intermediate, or fully evasive) per sample. Compute is measured by model parameter count for comparison purposes (e.g., Eva-4B at 4B parameters vs. frontier models presumably at much larger scales, though exact parameter counts for the closed-source models are not specified).
Cross-validation / statistical protocol. The evaluation is a single-pass benchmark on the Gold 1K set. There is no cross-validation reported for the main results — the Gold 1K serves as a fixed held-out test set that was never seen during training. The two-fold cross-validation protocol mentioned in the paper (Section 3.2 of the original) applies to the compute-optimal strategy selection for search and revision experiments, not to the evasion classification task. For inter-annotator agreement validation (Section 6), a 100-sample subset of the Gold 1K was independently labeled by a second human annotator, with Cohen's Kappa computed to establish label reliability (κ = 0.835, "Almost Perfect" per Landis and Koch, 1977).

Main Quantitative Results

Benchmark Performance: Eva-4B vs. Frontier and Open-Source Models

The headline result is that Eva-4B (Full) achieves 84.9% Macro-F1 on the Gold 1K evaluation set, outperforming all other evaluated models including Claude Opus 4.5 (84.4%), Gemini 3 Flash (84.6%), GPT-5.2 (80.9%), and all open-source models tested (Table 5). The full ranking with per-class F1 scores:

Rank	Model	Macro-F1	F1-D	F1-I	F1-E	Accuracy
1	Eva-4B (Full)	84.9	82.2	80.1	92.4	84.8
2	Gemini 3 Flash	84.6	84.6	78.3	91.0	84.6
3	Claude Opus 4.5	84.4	82.4	79.3	91.5	84.1
4	GLM-4.7	82.9	84.4	74.7	89.6	83.1
5	Eva-4B (Cons.)	81.4	79.4	75.0	89.7	81.0
6	GPT-5.2	80.9	75.4	76.1	91.2	80.8
7	Eva-4B (Opus)	80.6	77.6	73.9	90.3	80.6
8	Qwen3-Coder	78.2	72.1	72.4	90.0	78.0
9	MiniMax-M2.1	71.3	72.2	59.6	82.1	71.8
10	DeepSeek-V3.2	66.9	61.4	64.3	75.0	66.7
11	Kimi-K2	66.7	66.8	53.6	79.6	67.8
12	Qwen3-4B (Base)	34.3	7.3	33.3	62.3	42.3

The top three models (Eva-4B Full, Gemini 3 Flash, Claude Opus 4.5) are clustered within 0.5 percentage points of each other (84.4–84.9%), suggesting a performance ceiling around ~85% Macro-F1 that may reflect the inherent difficulty of the intermediate class rather than model capacity limitations.

Per-class difficulty hierarchy. Across all models, the Fully Evasive class is the easiest to detect — every model achieves F1-E > 75%, and the top models cluster around 91–92%. The Direct class is intermediate in difficulty, with top models achieving 82–85% F1-D. The Intermediate class is consistently the hardest — even the best model (Eva-4B Full) achieves only 80.1% F1-I, with several strong models falling substantially lower (GLM-4.7: 74.7%, GPT-5.2: 76.1%, Qwen3-Coder: 72.4%). This pattern — Fully Evasive easiest, Direct moderate, Intermediate hardest — holds for all models, confirming that the intermediate category represents a genuinely more difficult discrimination problem rather than a model-specific weakness.

Frontier model divergence. Despite their similar Macro-F1 scores, the top models exhibit distinct per-class strengths. Gemini 3 Flash achieves the highest F1-D (84.6%) — it is best at recognizing direct answers — but falls to 78.3% F1-I. Claude Opus 4.5 shows a more balanced profile (82.4% F1-D, 79.3% F1-I, 91.5% F1-E). GPT-5.2 is notably weak on Direct (75.4% F1-D), suggesting it over-predicts evasion on straightforward answers, consistent with its tendency to favor the intermediate label (56.7% of disagreement samples in Figure 2).

The base model is essentially incapable. Qwen3-4B without fine-tuning achieves only 34.3% Macro-F1 and a catastrophic 7.3% F1-D (Table 5). This means the untuned model almost never correctly identifies direct answers — its direct-class performance is barely above random chance (33.3% in a balanced three-way classification). The base model does somewhat better on Fully Evasive (62.3% F1-E), suggesting it has some rudimentary ability to detect extreme non-responsiveness but lacks any calibrated understanding of what constitutes a direct answer. The 50.6 percentage point improvement from base to Eva-4B (Full) represents the entire contribution of fine-tuning on MMC-labeled data and underscores that evasion detection is not a capability these models possess out of the box.

Ablation Study: The Contribution of Multi-Model Consensus

The ablation comparing Eva-4B variants (Figure 4 and Table 5) isolates the value of the MMC annotation framework:

Model Variant	Macro-F1	Δ from Base	Δ from Previous
Qwen3-4B (Base)	34.3	—	—
Eva-4B (Consensus)	81.4	+47.1 pp	+47.1 pp from Base
Eva-4B (Opus Only)	80.6	+46.3 pp	−0.8 pp from Consensus
Eva-4B (Full)	84.9	+50.6 pp	+4.3 pp from Opus Only

From Base to Consensus: +47.1 pp. Training on 60K consensus samples alone (where dual LLM annotators agreed) produces a strong classifier (81.4% Macro-F1) that already outperforms GPT-5.2 (80.9%), Qwen3-Coder (78.2%), and all weaker open-source models. This demonstrates that the consensus-filtered data — representing ~84% of the original annotations that achieved agreement — contains sufficient signal to learn the evasion taxonomy effectively. The consensus samples alone produce a better classifier than GPT-5.2 evaluated zero-shot, even though GPT-5.2 is presumably a much larger model.

From Consensus to Opus Only: −0.8 pp. Continuing training on 24K samples labeled by Claude Opus 4.5 alone produces no improvement — in fact, a slight degradation from 81.4% to 80.6%. This is the critical finding. The additional 24K samples contain more information (harder boundary cases that didn't achieve consensus) but the single-model labels are inconsistent enough that training on them provides no net benefit. The model essentially learns nothing useful from Stage 2 when the labels come from a single annotator.

From Opus Only to Full: +4.3 pp. Using three-judge majority voting labels for the same 24K samples produces a substantial improvement — from 80.6% to 84.9%. This is the paper's core methodological result: the arbitration process transforms the 24K disagreement samples from noise (providing no benefit when labeled by a single model) into signal (providing a 4.3 pp gain). The only difference between Eva-4B (Opus Only) and Eva-4B (Full) is the label source for the Stage 2 samples — same data, same model architecture, same training procedure. The 4.3 pp gain is purely attributable to the MMC annotation methodology.

Per-class ablation analysis (Table 5). The gains from Full over Opus Only are concentrated in F1-D (+4.6 pp: 77.6 → 82.2%) and F1-I (+6.2 pp: 73.9 → 80.1%). F1-E is already high for Opus Only (90.3%) and improves only modestly (+2.1 pp: 90.3 → 92.4%). This pattern makes sense: the disagreement samples represent boundary cases between adjacent classes, so resolving them correctly primarily improves discrimination on those boundaries — direct vs. intermediate (boosting both F1-D and F1-I) and intermediate vs. fully evasive (boosting F1-I and F1-E). Fully evasive is the easiest class to detect overall, so additional training on boundary cases provides diminishing returns.

The gap between Consensus and Opus Only is revealing. Consensus achieves 81.4% using only 60K samples; Opus Only achieves 80.6% using 84K samples. The additional 24K samples with single-model labels effectively hurt performance (relative to just discarding them). This validates the paper's claim (Section 5.4) that single-model annotation introduces systematic bias — the Opus-only labels on disagreement cases are not just noisy, they are actively misleading, containing contradictions that prevent the model from improving beyond the consensus-only baseline.

Training Dynamics: Label Consistency as the Mechanism

The training loss curves (Figure 5) provide a mechanistic explanation for why Full outperforms Opus Only. Key observations:

Stage 1: Consensus Training (shared by all variants). The loss curve for Eva-4B (Consensus) shows steady convergence over 3,000 steps, reaching a low but not fully saturated loss (exact final value not stated, but visibly below ~0.5 based on the y-axis scale). The model learns the basic taxonomy from consensus data effectively.

Stage 2: Dramatic divergence. When continued on the 24K samples:

"Eva-4B (Full) achieves a final loss of 0.007, while Eva-4B (Opus Only) converges to 0.56 — an 80× difference."

The Full model's loss curve drops sharply and saturates near zero, indicating that the MMC labels form a highly consistent pattern that the model can fit with near-perfect accuracy. The Opus Only model's loss plateaus around 0.56 after an initial drop, never approaching zero — the model cannot reduce its training error beyond this level because the training labels contain internal contradictions.

Interpretation. A three-way classification with balanced classes has a baseline cross-entropy loss of approximately 1.1 (the entropy of a uniform distribution). A loss of 0.56 means the model is substantially better than random but cannot achieve high confidence on many training examples because similar inputs receive different labels. A loss of 0.007 means the model achieves extremely high confidence on nearly all training examples — the labels are internally consistent enough that a 4B-parameter model can memorize the pattern almost perfectly.

This 80× loss difference is not an artifact of training hyperparameters — both variants use identical settings (Table 6). It directly reflects label quality: the MMC arbitration resolves contradictions that exist in Opus's single-model judgments, producing a training set where the mapping from Q&A text to evasion label is smooth enough to be learned with high fidelity. This is the strongest evidence in the paper that the value of MMC is not just per-sample accuracy but dataset-level consistency.

Caveat. A loss of 0.007 raises the specter of overfitting — the model might be memorizing training examples rather than learning generalizable patterns. The fact that Eva-4B (Full) achieves the highest Macro-F1 on the held-out Gold 1K set (84.9%) partially allays this concern, but the near-zero training loss is unusual for a natural language task and warrants scrutiny. If the training and evaluation sets are drawn from genuinely different distributions (different companies, different time periods), overfitting would manifest as a gap between training and evaluation performance. The Gold 1K accuracy of 84.8% (Table 5) is high but not near-perfect, suggesting some generalization gap exists but is within normal bounds for fine-tuned classifiers.

Error Analysis: Where Eva-4B Fails

The confusion matrix (Figure 6) and qualitative analysis (Section 9) characterize Eva-4B (Full)'s 152 errors on the Gold 1K set (15.2% error rate, since accuracy is 84.8%):

Confusion matrix structure (Figure 6). The matrix shows three key patterns:

Direct → Intermediate: 79 errors (23.7% of gold-direct samples), the single largest error category.
Intermediate → Direct: 23 errors (6.9% of gold-intermediate samples).
Intermediate → Fully Evasive: 18 errors (5.4% of gold-intermediate samples).
Fully Evasive → Intermediate: 25 errors (7.5% of gold-fully-evasive samples).
Cross-extreme errors (Direct ↔ Fully Evasive): only 7 errors total (1.2% + 0.9% = 2.1%).

Adjacent-class dominance. The paper reports that "95.4% of errors" involve adjacent classes. This is computed as 1 − (7 cross-extreme errors / 152 total errors) = 95.4%. This pattern is what a well-calibrated ordinal classifier should produce — errors cluster at category boundaries rather than representing fundamental confusion about the task structure. If the model were randomly guessing or fundamentally misunderstanding evasion, extreme errors (labeling a fully evasive response as direct, or vice versa) would be more common.

Primary error pattern: hedging-induced over-prediction of evasion (52% of errors). The dominant error type involves classifying direct responses as intermediate because they contain hedging language:

"These cases typically feature executives providing clear answers but using hedging language ('we do expect,' 'soon') that triggers false evasion signals. For example, when asked 'Is that still expected to occur?' the response 'We do expect it to occur soon' is gold-labeled direct but predicted intermediate — the model mistakes temporal vagueness for evasion."

This misclassification rate means that in 52% of Eva-4B's errors, the model sees hedging vocabulary and infers evasion, even though the response substantively answers the question. This is a pragmatic reasoning failure: hedging is correlated with intermediate evasion in the training data (executives hedge when they are being evasive), but hedging is also used by cooperative speakers to appropriately qualify uncertain statements. The model has learned the correlation but not the distinction.

Secondary error pattern: qualitative vs. quantitative directness. When analysts ask quantitative questions ("how much," "what percentage"), executives sometimes provide detailed qualitative explanations listing specific actions without providing the requested numbers. Eva-4B classifies these as direct due to their specificity and actionability, while human annotators label them intermediate because they fail to address the numeric core of the question. This reveals a limitation in the model's understanding of question-answer alignment at the level of specific informational content — it recognizes that relevant information is provided but does not verify whether the specific type of information requested (a number, a percentage, a timeline) is present.

Shared difficulty across models. The paper reports:

"Only 10.5% of Eva-4B errors are unique to our model; 33.6% of error samples are misclassified by 5–6 of the top models."

This is a robustness check demonstrating that Eva-4B's errors are not idiosyncratic — they are concentrated on samples that multiple strong models also get wrong. This suggests that some fraction of the Gold 1K samples may be genuinely ambiguous even for expert human annotators (recall that IAA κ = 0.835 means humans disagree on ~11% of samples, mostly involving intermediate). The 33.6% of samples misclassified by most top models likely represent cases near the boundary where even human judgment is uncertain.

Position Bias Validation

The position bias experiment (Section 9, Figure 7) is an evaluation of the arbitration mechanism rather than the classifier, but it provides critical validation of a design choice. On 5,541 samples:

"Fixed position (always Opus first) vs. randomized position. Randomization reveals a +5.1% win rate shift, confirming position bias in LLM judges."

In the fixed-position condition, Opus's win rate is 63.5%. When position is randomized (Opus sometimes first, sometimes second), Opus's win rate rises to 68.6%. The 5.1 percentage point gap is the primacy effect — the judge models disproportionately favor whichever annotation is presented first, regardless of content. This finding validates the paper's decision to randomize presentation order (seed 42) in the arbitration stage and provides quantitative evidence for the position bias phenomenon documented in the LLM-as-judge literature (Chen et al., 2024). Without randomization, the arbitration labels would be systematically biased toward the model placed in the first position.

Ablation Studies and Robustness Checks

Consensus-only vs. full training (Eva-4B Consensus vs. Eva-4B Full, Table 5, Figure 4): Training on 60K consensus samples alone achieves 81.4% Macro-F1. Adding 24K arbitrated samples (Full) improves this to 84.9% (+3.5 pp), demonstrating that the disagreement cases contain valuable signal when their labels are resolved through MMC arbitration rather than discarded. The 3.5 pp gain is substantial — it represents the marginal value of the entire MMC Stage II/III pipeline over simple consensus filtering.
Three-judge majority vs. single-model labels (Eva-4B Full vs. Eva-4B Opus Only, Table 5, Figure 4): The +4.3 pp Macro-F1 advantage for Full over Opus Only (84.9% vs. 80.6%) on the same 24K samples demonstrates that the arbitration mechanism produces qualitatively better labels than relying on a single strong model. This is the paper's core methodological claim and the primary evidence supporting MMC over single-model annotation.
Training data consistency (Figure 5): The loss curve divergence (0.007 vs. 0.56 final loss, an 80× difference) shows that MMC labels produce a more learnable pattern than single-model labels. This is not a standard ablation (there is no held-out metric being compared) but rather a diagnostic that explains why the Full model outperforms the Opus Only model — the labels are internally consistent enough to achieve near-zero training loss, which implies smoother decision boundaries and better generalization.
Per-class performance stability (Table 5): The hierarchy F1-E > F1-D > F1-I holds across all 12 models. Eva-4B (Full) achieves F1-E of 92.4%, F1-D of 82.2%, and F1-I of 80.1%. The intermediate class is the hardest for every model, confirming this is a property of the task rather than a weakness of any specific model or training procedure.
Model scale comparison (Table 5): The 4B-parameter Eva-4B outperforms frontier models presumed to be orders of magnitude larger (exact parameter counts for Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash are not disclosed, but these are almost certainly 100B+ parameter models). This demonstrates that training on high-quality domain-specific data can compensate for model scale on this task — but the gradient is shallow at the top (Eva-4B Full at 84.9%, Gemini 3 Flash at 84.6%, Claude Opus 4.5 at 84.4%), suggesting a performance ceiling around 85% that may reflect inherent ambiguity in the intermediate class rather than model capacity.
Base model capability (Table 5): The untuned Qwen3-4B achieves 34.3% Macro-F1 with F1-D of 7.3%, confirming that evasion detection is not an emergent capability of the base model. The near-zero F1-D means the untuned model almost never correctly identifies direct answers — it likely defaults to predicting intermediate or fully evasive for most inputs. This validates that the fine-tuning on MMC-labeled data is essential and that the performance of Eva-4B reflects learned evasion detection rather than surfacing latent base model knowledge.
Cross-model error overlap (Section 9 qualitative analysis): "33.6% of error samples are misclassified by 5–6 of the top models." This means that one-third of Eva-4B's errors are on samples that most strong models also get wrong, suggesting these are genuinely difficult cases possibly at the boundary of human annotation reliability. Only "10.5% of Eva-4B errors are unique to our model," indicating that Eva-4B's error profile largely overlaps with frontier models rather than exhibiting model-specific failure modes.

Missing ablations. Several ablations that would strengthen the paper are absent:

Number of primary annotators: The paper uses exactly two primary annotators (Claude Opus 4.5 and Gemini 3 Flash). Would three or five primary annotators improve consensus quality? Would a different pair of models (e.g., Claude + GPT-5.2) produce different consensus rates and downstream performance?
Number of arbitration judges: Three judges are used for arbitration. Would five judges improve label quality? Is the majority-voting decision stable or would different judge compositions change the labels?
Judge identity in arbitration. GPT-5.2 appears only in the arbitration stage. What if it were a primary annotator instead? The paper provides no systematic comparison of different model role assignments.
Dataset balancing effect: All splits are balanced (33.3% per class). The paper does not report performance on naturally distributed data, which would almost certainly have different class proportions. A model achieving 84.9% Macro-F1 on balanced data might perform differently when direct answers dominate.
Random seed sensitivity: The position randomization uses seed 42. Are the arbitration labels stable across different random seeds? The 5.1% position bias effect suggests that individual label assignments could shift under different randomizations.
Prompt sensitivity: The annotation prompts are not reproduced in the paper. How sensitive are the primary annotator labels and judge decisions to prompt wording, examples, or instruction format?
Company and temporal generalization: Table 5 reports aggregate performance. Is Eva-4B's performance consistent across different industries, company sizes, or time periods? A model that performs well on average but poorly on specific subpopulations (e.g., technology companies with highly technical Q&A, or earnings calls from the 2008 financial crisis period) would have limited practical utility.

Critical Assessment

Claim: Eva-4B achieves 84.9% Macro-F1, outperforming Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash.

The experiments support this claim with qualifications. Eva-4B (Full) does achieve the highest Macro-F1 in Table 5 (84.9%), but the margin over Gemini 3 Flash (84.6%) and Claude Opus 4.5 (84.4%) is extremely narrow — 0.3 and 0.5 percentage points, respectively. These differences are within the range where a slightly different test set, prompt wording, or evaluation protocol could reorder the top models. The claim of "outperforming" is technically true but overstates the practical significance — the top three models are effectively tied at ~84.5–85.0% Macro-F1. The more meaningful comparison is Eva-4B vs. GPT-5.2 (84.9% vs. 80.9%), a 4.0 pp gap that is clearly significant, and vs. the Qwen3-4B base model (84.9% vs. 34.3%), which demonstrates the full contribution of MMC-labeled training data.

Claim: Multi-model consensus labeling significantly outperforms single-model annotation (+4.3 pp Macro-F1).

Strongly supported. The comparison between Eva-4B (Full) at 84.9% and Eva-4B (Opus Only) at 80.6% (Table 5, Figure 4) isolates the annotation methodology as the only variable — same base model, same training procedure, same Stage 1 consensus data, same 24K Stage 2 samples, differing only in whether Stage 2 labels come from three-judge majority voting or Claude Opus 4.5 alone. The loss curve divergence in Figure 5 (0.007 vs. 0.56 final loss) provides a mechanistic explanation: single-model labels contain internal inconsistencies that prevent the model from fitting the training data, while MMC labels form a learnable pattern. This is a clean and convincing demonstration.

However, the comparison is specific to Claude Opus 4.5 as the single model. The paper does not test what would happen if GPT-5.2 or Gemini 3 Flash were the single annotator. Figure 2 shows that these models have different labeling biases — GPT-5.2 heavily favors "intermediate" (56.7%), while Gemini favors "fully evasive" (23.5%). A model trained on GPT-5.2-only labels might perform differently than one trained on Opus-only labels. The claim that "multi-model consensus outperforms single-model annotation" is supported for Opus specifically but would be stronger with evidence that it holds regardless of which single model is used as the baseline.

Claim: MMC achieves a Cohen's Kappa of 0.835 on human inter-annotator agreement.

Supported but requires careful interpretation. The reported κ = 0.835 (Table 4) is the agreement between two human annotators on 100 samples from the Gold 1K set. This validates that the Gold 1K labels are reliable — humans can operationalize the three-level taxonomy with "Almost Perfect" agreement. However, this is human-human agreement, not MMC-human agreement. The paper does not directly measure how well the MMC-produced labels agree with human labels. Instead, it measures Eva-4B's performance when trained on MMC labels and evaluated against human labels — an indirect validation. If MMC labels agreed perfectly with human labels, Eva-4B trained on MMC data should approach the human IAA ceiling (~89% Macro-F1 per Table 4). Eva-4B achieves 84.9%, which is close but not at ceiling, suggesting the MMC labels are good but not perfect proxies for human judgment.

The 100-sample IAA subset is also relatively small. While κ = 0.835 is a strong result, the confidence interval around this estimate is not reported, and the stability of the agreement estimate with only 11 disagreements (10 of which involve the intermediate class) depends on whether those 11 disagreements represent typical ambiguity or unusual edge cases.

Claim: The three-level taxonomy captures a genuine gradation that a binary classification would obscure.

Supported by the IAA and error analysis. The IAA result (κ = 0.835 for three levels, vs. κ < 0.5 for five levels in pilot annotation) validates that three levels are distinguishable by humans with high reliability while five are not. The error analysis shows that 95.4% of Eva-4B's errors involve adjacent classes (Figure 6) — an ordinal pattern consistent with a genuinely graded phenomenon where boundaries are fuzzy but the ordering is clear. The per-class F1 hierarchy (F1-E > F1-D > F1-I across all 12 models) shows that the intermediate class is consistently the hardest, which would not be the case if intermediate were an arbitrary middle category rather than a genuinely more ambiguous region of the evasion spectrum. The paper does not, however, compare directly against a binary-classification baseline — training Eva-4B on a two-way (direct vs. evasive) task and evaluating whether the lost distinction between intermediate and fully evasive matters for downstream applications.

Claim: Small models trained on high-quality consensus data can match or exceed frontier LLMs on this task.

Supported with an important caveat about the performance ceiling. Eva-4B at 4B parameters achieves 84.9% Macro-F1, while the best frontier model (Gemini 3 Flash) achieves 84.6%. This supports the claim that data quality can compensate for model scale. However, all top models cluster within 0.5 pp of each other, suggesting a performance ceiling around 85% that may reflect inherent task ambiguity rather than model capacity. If the ceiling is 85% (driven by the inherent difficulty of distinguishing intermediate from adjacent categories), then demonstrating that a small model reaches the ceiling is less surprising than demonstrating that a small model exceeds what larger models can achieve — the result shows equivalence at ceiling, not superiority through scale.

The base model's abysmal performance (34.3% Macro-F1, 7.3% F1-D) also contextualizes this result. The fine-tuning adds 50.6 pp of improvement. The base model essentially has no evasion detection capability — the entire performance comes from the training data. This means the Eva-4B result is fundamentally a statement about the MMC annotation pipeline's ability to produce training data that teaches evasion detection from scratch, not about Qwen3-4B's inherent suitability for the task.

Genuine weaknesses in the experimental design:

Single benchmark, single domain. All evaluation is on earnings call Q&A from U.S.-listed companies (English only, 2002–2022). There is zero evidence about whether Eva-4B generalizes to other adversarial Q&A settings (political interviews, legal depositions) or even to earnings calls from different markets or languages. The paper acknowledges this limitation in its Limitations section but the experiments provide no cross-domain signal.
No confidence intervals or statistical testing. The paper reports point estimates for Macro-F1, per-class F1, and accuracy without confidence intervals, standard deviations, or statistical significance tests. With a test set of 1,000 samples, the standard error for Macro-F1 around 85% is approximately 1.1 percentage points (assuming binomial approximation). The 0.3 pp gap between Eva-4B Full (84.9%) and Gemini 3 Flash (84.6%) is well within this margin, meaning the ranking of top models is not statistically reliable. The paper should not claim clear superiority without demonstrating statistical significance.
No comparison against fine-tuned versions of the frontier models. The closed-source models are evaluated zero-shot. What would happen if Claude Opus 4.5 or GPT-5.2 were fine-tuned on the same 84K MMC-labeled training data? The paper's claim that "a 4B model can outperform frontier LLMs" conflates model architecture with training regime — it compares a fine-tuned small model against zero-shot large models. A fairer comparison would be fine-tuned small model vs. fine-tuned large model, or zero-shot small model vs. zero-shot large model. The current comparison shows that fine-tuning on MMC data helps, not that 4B parameters is inherently better than ~100B+ parameters for this task.
The Gold 1K set is single-annotator with 100-sample IAA validation. Only 100 of the 1,000 gold labels are verified by a second annotator. The remaining 900 labels reflect a single human expert's judgment, which could introduce individual annotator bias. The IAA κ of 0.835 indicates 89% agreement, meaning approximately 11% of labels would differ if a different annotator had been used. On a 1,000-sample set, this means ~110 samples may have unstable labels, which is more than enough to shift model rankings by several percentage points.
No analysis of model calibration or confidence. The paper reports accuracy and F1 but not calibration error, confidence distributions, or entropy of predictions. For a task intended to support investor alerting and regulatory monitoring (Section 9: Practical Applications), knowing whether the model is appropriately confident in its predictions — and whether it can identify when it is uncertain — is at least as important as aggregate accuracy. A model that is 85% accurate but overconfident on its errors is less useful for deployment than one that is 85% accurate and well-calibrated.
The Opus Only ablation uses Claude Opus 4.5, but the paper does not test whether the result generalizes to other single-model annotators. Figure 2 shows that GPT-5.2 and Gemini 3 Flash have systematically different labeling tendencies. Would training on GPT-5.2-only labels produce similarly poor consistency (high loss, plateaued convergence)? Or would a different single model produce cleaner labels? Without this comparison, the claim that single-model annotation is insufficient rests on evidence from one specific model.
The practical value of +4.3 pp Macro-F1 from MMC is unclear. The MMC framework requires approximately 3× the API calls of single-model annotation (two primary annotators + three judges for 16.1% of samples ≈ 2.48 model calls per sample vs. 1.0 for single-model annotation). The paper does not analyze whether the +4.3 pp improvement justifies this cost, either in terms of downstream task performance or in comparison to alternative ways of spending the annotation budget (e.g., using a single more expensive model, or doing two rounds of single-model annotation with human adjudication of disagreements).

Experiments that would have strengthened the paper:

MMC-human agreement: Directly measure how well the MMC-produced labels agree with human labels on the Gold 1K set. This would provide a direct validation of MMC quality rather than the indirect validation through Eva-4B performance.
Cross-domain transfer: Evaluate Eva-4B on political interviews, legal depositions, or press conferences to test whether evasion detection learned from earnings calls transfers to other adversarial Q&A settings.
Binary vs. ternary comparison: Train an Eva-4B binary variant (direct vs. evasive) and compare its performance and utility to the three-class version.
Natural distribution evaluation: Report performance on an imbalanced test set reflecting natural evasion prevalence to assess real-world applicability.
Confidence calibration analysis: Measure expected calibration error (ECE) and reliability diagrams for Eva-4B and frontier models to assess deployment readiness.
Per-industry, per-year, and per-company-size breakdowns: Test whether performance is uniform across different subpopulations of the evaluation set.
Single-model annotator comparison: Extend the Opus Only ablation to GPT-5.2 Only and Gemini Only variants to determine whether the MMC advantage is specific to Opus's labeling tendencies or generalizes across annotator models.

6. Limitations and Trade-offs

6.1 The Cost of Multi-Model Consensus Annotation Is Not Accounted For

The assumption or constraint. The MMC framework requires substantially more inference calls per labeled sample than single-model annotation. The pipeline uses two primary annotators for every sample (Stage I), plus three arbitration judges for the 16.1% of samples where the primary annotators disagree (Stage III). This means the expected number of LLM calls per training sample is:

$2 + 3 \times 0.161 = 2.48$

in other words, roughly 2.5× the inference cost of single-model annotation (which requires 1 call per sample). For the 84K training set, this translates to approximately 208,000 LLM calls versus 84,000 for single-model annotation—a substantial API cost difference, especially when using frontier models like Claude Opus 4.5 and GPT-5.2.

The consequence. The headline result—that MMC labeling yields a +4.3 percentage point Macro-F1 improvement over single-model annotation (Eva-4B Full vs. Opus Only, Table 5: 84.9% vs. 80.6%)—is presented without any cost-benefit analysis. A practitioner deciding whether to adopt the MMC framework needs to know: is the +4.3 pp improvement worth 2.5× the annotation cost? The answer depends on the downstream value of those 4.3 percentage points. For a production investor alerting system where each additional correct evasion detection has measurable financial value, the tradeoff might be favorable. For a research project with a fixed annotation budget, it might be more efficient to use single-model annotation on a larger corpus (2.5× more samples for the same cost) rather than MMC on a smaller one. The paper provides no evidence about which of these strategies would produce better downstream performance—would 210K single-model-labeled samples outperform 84K MMC-labeled samples?

Furthermore, the cost analysis is complicated by the specific models used. Claude Opus 4.5 and GPT-5.2 are among the most expensive frontier models available. The paper does not test whether a cheaper model could serve as the second annotator or as a judge—for instance, using GPT-5.2 only for arbitration and a more cost-effective model like Gemini 3 Flash for primary annotation. The MMC framework's cost is tied to the specific models chosen, and alternative configurations could dramatically change the cost-performance tradeoff.

What evidence exists in the paper. The paper does not report API costs, token counts, or any economic analysis of the MMC framework. The 16.1% disagreement rate is reported in Section 5.4, which is the only number needed to compute the expected annotation cost multiplier (2.48×). The +4.3 pp improvement is documented in Table 5 and Figure 4. But the paper never connects these two numbers—cost and benefit—in a single analysis.

Mitigation status. Not addressed. The paper does not acknowledge the annotation cost as a limitation, does not compare cost-efficiency against single-model annotation at larger scale, and does not propose cost-reduction strategies (e.g., using cheaper models for the second annotator, adaptive arbitration where only a subset of disagreement cases go to three judges, or distilling a single-model annotator from an MMC ensemble). This is a significant gap because the MMC framework is presented as a general annotation methodology, but its practical adoption depends on whether the performance gains justify the additional cost.

6.2 The Annotation Quality Validation Is Indirect and Incomplete

The assumption or constraint. The paper validates the MMC annotation framework through two measurements: (1) human-human inter-annotator agreement on 100 samples from the Gold 1K set (Table 4: Cohen's κ = 0.835), and (2) Eva-4B's performance when trained on MMC labels and evaluated against the Gold 1K human labels (Table 5: 84.9% Macro-F1). Neither measurement directly assesses the quality of the MMC-produced training labels themselves.

The human-human agreement (κ = 0.835) validates that the gold evaluation labels are reliable—two human experts agree at a "near-perfect" level. This is important for establishing that the benchmark is well-defined, but it says nothing about whether the MMC pipeline produces labels that agree with human judgment. Eva-4B's 84.9% Macro-F1 against human labels is an indirect validation: if the MMC labels were very different from human labels, a model trained on them would not achieve high performance on a human-labeled test set. But this is a mediated validation—the training labels could differ from human labels in systematic ways that the model learns to compensate for, or the model could be learning from label patterns that happen to align with human judgment on most but not all test cases.

The consequence. Without direct MMC-human agreement measurement, we cannot answer critical questions about label quality:

What is the upper bound on model performance given MMC labels? If MMC labels agree with human labels 90% of the time, then a perfect classifier trained on MMC data would achieve at most ~90% accuracy against human labels—the MMC labeling process itself imposes a ceiling. Eva-4B's 84.9% Macro-F1 might be close to this ceiling, meaning further improvements require better annotation, not better modeling. If MMC labels agree with humans 95% of the time, there is still room for modeling improvements. The paper provides no way to distinguish these scenarios.
Are the remaining Eva-4B errors (15.2% of Gold 1K samples) caused by model limitations or by MMC label errors? The error analysis in Section 9 identifies hedging-induced over-prediction as the dominant error pattern, but it does not trace these errors back to whether the training labels were correct. If the MMC framework systematically labeled certain hedging responses as "intermediate" when humans would label them "direct," then Eva-4B's errors reflect annotation bias rather than model failure. The 33.6% of error samples misclassified by multiple top models (Section 9) is consistent with both interpretations: either these are genuinely ambiguous cases, or they are cases where both LLM-based annotators and LLM-based classifiers share the same systematic bias absent from human judgment.
How reliable are the labels on the 24K disagreement samples specifically? These are the hardest cases—where two frontier models disagree—and they contribute a crucial 4.3 pp improvement when resolved through three-judge arbitration (Table 5: Full vs. Opus Only). If the arbitration labels are unreliable, the claimed improvement might come from fitting the model to patterns that happen to align with the Gold 1K labels but would not generalize to a different test set.

What evidence exists in the paper. The paper provides no direct MMC-human agreement measurement. The 100-sample IAA validation (Section 6) compares two humans, not MMC vs. human. The Gold 1K labels are single-annotator for 900 of the 1,000 samples, with only the 100-sample subset verified by a second annotator (Section 6). The single human annotator could have individual biases that make the Gold 1K labels an imperfect measure of "true" evasion.

Mitigation status. Partially addressed through IAA validation of the gold labels, but the central gap—direct validation of MMC-produced labels against human judgment—remains unaddressed. The paper does not acknowledge this as a limitation. A direct validation would require having human annotators label a subset of the MMC-annotated training data (or at least a held-out set of MMC-labeled samples) and computing agreement statistics. This would be expensive but is feasible for even a modest sample (e.g., 500–1,000 samples) and would provide the missing direct evidence about MMC label quality.

6.3 The Benchmark Is Domain-Specific with No Evidence of Transfer

The assumption or constraint. All data—both the 84K training set and the Gold 1K evaluation set—comes from a single domain: earnings call Q&A transcripts from U.S.-listed companies, 2002–2022, in English. The paper acknowledges this limitation explicitly in Section 9 (Limitations):

"Our dataset and models are trained exclusively on earnings call transcripts. Generalization to other domains (political interviews, legal depositions) requires further validation."

and:

"The current benchmark covers only English-language transcripts from U.S.-listed companies."

The paper hypothesizes that "both our annotation framework and taxonomy could transfer to other adversarial Q&A settings" (Section 1), but provides zero empirical evidence for this claim.

The consequence. A practitioner interested in evasion detection for a different domain—political interviews, legal depositions, press conferences, customer service interactions, academic Q&A sessions—cannot assume that Eva-4B or the EvasionBench taxonomy will work without modification. Several specific failures are plausible:

Linguistic register shift. Earnings calls have a specific linguistic register—formal business English with technical financial terminology, structured turn-taking, and predictable question types (quantitative, temporal, binary). Evasion in political interviews might use different linguistic strategies (e.g., attacking the questioner, invoking broader values, pivoting to campaign talking points) that are underrepresented or absent in the earnings call training data.
Taxonomy boundary drift. The three-level taxonomy was empirically calibrated on earnings call data (Section 3.2: five levels collapsed to three because κ < 0.5 for five levels). In a different domain, the optimal number of categories might differ—political evasion might require distinguishing between "deflection to talking points" and "explicit refusal," which are both "fully evasive" in the current taxonomy but have different practical and theoretical implications. The taxonomy's grounding in Gricean pragmatics (Section 3.2) suggests theoretical domain-generality, but the empirical calibration was domain-specific.
Eva-4B overfitting to domain-specific cues. Eva-4B was fine-tuned on earnings call data and likely learned domain-specific correlates of evasion—particular financial jargon patterns, common question templates, industry-specific hedging phrases. When applied to political interviews, these cues would be absent or misleading, and performance would degrade. The paper provides no out-of-domain evaluation to quantify this degradation.
Company and temporal coverage gaps. Even within earnings calls, the training data covers 8,081 unique companies (Table 3), but this is a subset of all publicly traded companies. Small-cap companies, non-U.S. companies, or companies in specialized industries might have different communication patterns that are underrepresented. The temporal span (2002–2022) includes substantial regulatory changes (Sarbanes-Oxley implementation, Dodd-Frank, Regulation FD enforcement evolution) that could have shifted evasion strategies over time. The paper does not report per-industry or per-year breakdowns, so we cannot assess whether Eva-4B's aggregate 84.9% Macro-F1 masks substantial variance across subpopulations.

What evidence exists in the paper. No cross-domain, cross-lingual, or subpopulation analysis is reported. All experiments in Section 8 are on the Gold 1K evaluation set drawn from the same domain and distribution as the training data. The paper does not evaluate Eva-4B on any out-of-domain dataset, nor does it test whether the MMC annotation framework produces reliable labels when applied to non-earnings-call text. The mention of potential transfer to "political interviews, legal depositions" in Section 1 is purely speculative.

Mitigation status. Acknowledged as a limitation in Section 9 but not addressed experimentally. The paper frames cross-domain validation as "future work." This is standard for a benchmark paper—establishing the in-domain benchmark first, then encouraging the community to test transfer—but it means that EvasionBench's practical utility is currently limited to financial NLP applications. A practitioner in a different domain would need to replicate the entire pipeline (data collection, MMC annotation, model training) without knowing whether the taxonomy or annotation methodology will transfer.

6.4 The Gold 1K Evaluation Set Is Small and Single-Annotator for 90% of Samples

The assumption or constraint. The primary evaluation benchmark for all 12 models is the Gold 1K set: 1,000 samples with expert human labels (Table 3). Of these 1,000 samples, only 100 (10%) were independently labeled by a second human annotator to establish inter-annotator agreement (Section 6). The remaining 900 samples (90%) reflect a single annotator's judgment with no verification.

The inter-annotator agreement on the 100-sample subset is strong (κ = 0.835, 89% agreement, Table 4), which the paper uses to validate the gold labels as reliable. However, this validation covers only 10% of the evaluation set. The 11 disagreements on the 100-sample subset reveal that approximately 11% of labels would differ with a different annotator. Extrapolated to the full 1,000 samples, this means approximately 110 gold labels are potentially unstable—a different human annotator would assign a different label.

The consequence. The ranking of top models in Table 5 depends on these potentially unstable labels. The top three models are clustered within 0.5 percentage points of each other:

Eva-4B (Full): 84.9% Macro-F1
Gemini 3 Flash: 84.6% Macro-F1
Claude Opus 4.5: 84.4% Macro-F1

If 110 labels are unstable, the model ranking could shift substantially depending on which annotator's labels are used. A model that correctly predicts the "true" label (as determined by a consensus of annotators) might be penalized because the single annotator assigned a different label, while another model that happens to agree with that annotator's idiosyncratic judgments gets credited. The 0.3 pp gap between Eva-4B and Gemini, or the 0.5 pp gap between Eva-4B and Claude, is well within the range that could be explained by annotation noise rather than genuine model superiority.

This problem is compounded by the concentration of disagreements in the intermediate class. The paper reports that "10 of 11 disagreements involve the intermediate class" (Section 6). This means the intermediate labels are disproportionately unstable—the category where model performance varies most widely (F1-I ranges from 53.6% for Kimi-K2 to 80.1% for Eva-4B in Table 5) is also the category where the gold labels themselves are least reliable. A model that disagrees with the single annotator on intermediate cases might be penalized more heavily than one that disagrees on direct or fully evasive cases, even if the model's judgment aligns with the consensus of multiple human annotators.

Additionally, 1,000 samples is a relatively small evaluation set for distinguishing models with sub-percentage-point performance differences. Assuming a standard error of approximately sqrt(p(1-p)/n) for accuracy around 85%, the 95% confidence interval is roughly ±2.2 percentage points. This means that for any pairwise model comparison, a gap of less than ~4.4 percentage points is not statistically significant at the 95% level. The top four models (84.9%, 84.6%, 84.4%, 82.9%) are all within this margin—the paper cannot reliably claim that any of these models outperforms the others based on this test set alone.

What evidence exists in the paper. The IAA validation on 100 samples (Table 4) provides the only direct evidence about label stability. The 89% agreement rate and κ = 0.835 are encouraging but do not eliminate the concern—11% label instability on a 1,000-sample set means approximately 110 unstable labels, which is comparable to the number of errors made by the best models (Eva-4B makes 152 errors on the Gold 1K, Figure 6). The paper does not report confidence intervals for any metric, does not perform bootstrap resampling to estimate ranking stability, and does not discuss the implications of the single-annotator design for model comparison.

Mitigation status. Partially addressed. The 100-sample IAA validation is better than no validation, and the κ of 0.835 is genuinely strong—it establishes that the three-level taxonomy is operationalizable with high reliability, which is an important contribution. But the paper does not address the gap between validating 10% of the test set and relying on the other 90% for model ranking without verification. A stronger design would be: (1) double-annotate the entire Gold 1K set, using adjudication for disagreements to produce consensus labels; (2) report confidence intervals for all metrics; or (3) use a larger evaluation set (e.g., 5K–10K samples) to reduce the standard error to a level where sub-percentage-point gaps are statistically meaningful. The paper acknowledges the scale limitation in its Limitations section ("While our Gold 1K set is validated, single-annotator labeling with 100-sample IAA verification represents a limitation. Larger-scale human annotation would strengthen validity") but does not adjust its claims about model superiority accordingly.

6.5 The Comparison Against Frontier Models Is Zero-Shot vs. Fine-Tuned, Not Fairly Matched

The assumption or constraint. The paper's headline claim—that Eva-4B (4B parameters) "outperforms Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash" (Section 1)—compares a fine-tuned Eva-4B against zero-shot prompted frontier models. Table 5 shows Eva-4B (Full) at 84.9% Macro-F1 versus Gemini 3 Flash at 84.6% and Claude Opus 4.5 at 84.4%. These frontier models receive only a task description and taxonomy definitions in a prompt—they are not fine-tuned on the 84K MMC-labeled training data.

This is not a fair comparison between model architectures or scales. It compares:

Eva-4B: Qwen3-4B fine-tuned on 84K in-domain labeled examples
Frontier models: Much larger models (likely 100B+ parameters, though exact counts are not disclosed) prompted zero-shot with taxonomy definitions

The relevant question is: what would happen if the frontier models were also fine-tuned on the same training data? Given that fine-tuning adds 50.6 percentage points to Qwen3-4B's Macro-F1 (from 34.3% to 84.9%, Table 5), it is plausible that fine-tuning a frontier model would produce even larger gains, potentially exceeding Eva-4B's performance.

The consequence. The paper's claim of "outperforming" conflates two effects: (1) the quality of the MMC training data, and (2) the difference between fine-tuned and zero-shot models. The experiments demonstrate that fine-tuning on MMC data helps and that a fine-tuned 4B model can approach the performance of zero-shot frontier models, but they do not demonstrate that a 4B model is inherently better than larger models for this task. The fair comparison would be:

Fine-tuned Eva-4B vs. fine-tuned frontier models: This would test whether the 4B parameter count is sufficient to capture all relevant signal in the training data, or whether larger models would benefit more from the same data.
Zero-shot Eva-4B vs. zero-shot frontier models: This would test whether Qwen3-4B has any inherent evasion detection capability that competes with larger models when no training data is provided.

The paper provides neither comparison. The zero-shot Eva-4B baseline is partially available—Qwen3-4B (Base) achieves 34.3% Macro-F1 (Table 5)—but this likely reflects the base model prompted for classification, not given the full taxonomy definitions and examples that the frontier models receive. The relative improvement from fine-tuning (50.6 pp for Qwen3-4B) is much larger than the gap between Eva-4B and frontier models (0.3–4.0 pp), suggesting that access to training data, not model architecture, drives the result.

Practitioners deciding which model to deploy face a different comparison than the one presented. If a practitioner has access to the MMC-labeled training data and can fine-tune models, they should compare fine-tuned versions of all candidate models, not fine-tuned Eva-4B against zero-shot frontier models. If the frontier models are only available via API (no fine-tuning access), then the zero-shot comparison is practically relevant, but the paper should be explicit that the advantage comes from fine-tuning access rather than inherent model superiority.

What evidence exists in the paper. Table 5 shows the comparison as described. The paper does not report fine-tuned versions of Claude, GPT-5.2, or Gemini (likely because these models do not support public fine-tuning at the time of writing). The base model performance (Qwen3-4B: 34.3% Macro-F1) demonstrates the magnitude of the fine-tuning effect, but no analogous measurement exists for the frontier models. The paper does not discuss this comparison asymmetry as a limitation.

Mitigation status. Not addressed. The paper presents the zero-shot vs. fine-tuned comparison without caveat. A fairer presentation would: (1) explicitly note that frontier models are evaluated zero-shot while Eva-4B is fine-tuned, (2) report both zero-shot and fine-tuned performance where possible, (3) frame the result as "a fine-tuned 4B model can match or exceed zero-shot frontier models" rather than "a 4B model outperforms frontier models," and (4) discuss the practical implications of the comparison asymmetry for deployment decisions.

6.6 Hard Cases Near Classification Boundaries May Be Fundamentally Unresolvable

The assumption or constraint. The paper's three-level taxonomy assumes that every Q&A pair can be assigned a single-evasion label (direct, intermediate, or fully evasive) that corresponds to a ground-truth communicative fact about whether the question was answered. The empirical evidence in the paper itself suggests that this assumption breaks down for a non-trivial fraction of cases, particularly those near the direct-intermediate and intermediate-fully evasive boundaries.

Multiple independent lines of evidence converge on this point:

Human-human disagreements cluster at the intermediate boundary. The IAA validation on 100 samples finds 11 disagreements, and "10 of 11 disagreements involve the intermediate class" (Section 6). This means that even trained human annotators disagree on intermediate boundary cases approximately 11% of the time. These are not random errors—they reflect genuine ambiguity about whether a response that is partially relevant crosses the threshold into "not actually answering the question."
All models struggle most with intermediate classification. The per-class F1 hierarchy (Fully Evasive > Direct > Intermediate) holds for all 12 evaluated models (Table 5). Eva-4B (Full) achieves F1-E of 92.4% and F1-D of 82.2%, but only F1-I of 80.1%. The worst-performing models show even steeper drops: Kimi-K2 achieves F1-E of 79.6% but F1-I of only 53.6%. This is not a model-specific weakness—it is a task property.
Multiple top models converge on the same errors. The qualitative error analysis (Section 9) finds that "33.6% of error samples are misclassified by 5–6 of the top models." This means one-third of Eva-4B's errors are on samples that most strong models also get wrong, suggesting these are cases where the "correct" label is genuinely ambiguous, not cases where Eva-4B makes a clear mistake that a better model would avoid.
The performance ceiling appears to be around 85% Macro-F1. The top four models (Eva-4B Full, Gemini 3 Flash, Claude Opus 4.5, GLM-4.7) cluster within 2.0 percentage points of each other (82.9–84.9%), despite spanning model scales from 4B to presumably 100B+ parameters and including both fine-tuned and zero-shot approaches. This clustering suggests a ceiling effect—the remaining errors may reflect inherent task ambiguity rather than insufficient model capacity.

The consequence. The 85% Macro-F1 ceiling has practical implications for deployment. No model, regardless of scale or training methodology, can achieve near-perfect accuracy on this task because some fraction of Q&A pairs are genuinely ambiguous—reasonable human annotators would disagree on the label. A system deployed for investor alerting or regulatory monitoring would produce errors on approximately 15% of cases, and some of those errors would be on cases where the "correct" answer is debatable.

This also complicates model ranking. If 15% of test samples are inherently ambiguous (labels are unstable across annotators), then the maximum achievable Macro-F1 on this test set is approximately 85%—and the difference between an 84.9% model and an 84.4% model might reflect random variation in which ambiguous cases each model happens to get "right" (according to the specific annotator's labels) rather than any genuine superiority. The paper's ranking of top models cannot distinguish between models that are near the ceiling.

Furthermore, the taxonomy itself may need refinement for edge cases. The paper's qualitative error analysis identifies specific ambiguity patterns: hedging language that reduces commitment without evading the question core (Section 9), and qualitative responses to quantitative questions that provide relevant but not numerically precise information. These patterns represent genuine communicative phenomena where the binary question "did they answer the question?" has no clear answer—the executive provided some information but not the specific information requested. The three-level taxonomy forces a binary decision (intermediate vs. direct) on a genuinely continuous phenomenon.

What evidence exists in the paper. The IAA results (Table 4: 89% agreement, κ = 0.835) provide the most direct evidence for inherent ambiguity. The per-class F1 hierarchy (Table 5) and the cross-model error overlap analysis (Section 9) provide converging evidence that the intermediate class is fundamentally harder. The overall performance ceiling is visible from the clustering of top models in Table 5. The qualitative error patterns are documented in Section 9.

Mitigation status. Partially addressed. The paper acknowledges the difficulty of intermediate classification (Section 9: "Why is Intermediate difficult?") and documents the ambiguity patterns in the qualitative analysis. The three-level taxonomy itself is presented as an empirically calibrated solution—five levels proved unreliable (κ < 0.5), three levels restored reliability (κ = 0.83)—suggesting the authors made design choices to minimize but not eliminate ambiguity. However, the paper does not fully confront the implications: that 15% of test cases may be unresolvable, that the 85% ceiling limits the practical value of further model improvements, and that model rankings near the ceiling are unreliable. A more complete treatment would: (1) report the human performance ceiling (the IAA Macro-F1 from Table 4 is 88.99%, suggesting room for improvement above 85% but still below 90%), (2) analyze whether certain types of ambiguity are systematic and could be addressed through refined taxonomy guidelines, and (3) compute confidence intervals for model rankings to acknowledge that gaps of less than ~2 pp are not statistically distinguishable given the test set size and label ambiguity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a methodological shift in how the NLP community can approach annotation for tasks involving graded, discourse-level judgment. The central conceptual move is reframing multi-model consensus from a quality-assurance mechanism (reducing random noise through averaging) to a bias-cancellation mechanism (canceling out systematic, model-specific interpretive tendencies that would otherwise be baked into the training data). This is not a paradigm shift in the sense of introducing a fundamentally new model architecture or learning algorithm, but it is a meaningful reframing of the annotation problem that changes what researchers should optimize for when constructing datasets for subjective classification tasks.

The evidence that makes this reframing stick comes from two sources. First, Figure 2 demonstrates that three frontier LLMs—Claude Opus 4.5, Gemini 3 Flash, and GPT-5.2—exhibit systematically different labeling tendencies on the same data, with Opus favoring "direct" (53.3% of disagreement samples), Gemini favoring "fully evasive" (23.5%), and GPT-5.2 favoring "intermediate" (56.7%). These are not random variations around a consensus; they represent stable, model-level interpretive biases. A dataset labeled by any single model would encode that model's bias regardless of scale—a 100K-sample Opus-labeled dataset would systematically undercount fully evasive responses. Second, the training dynamics in Figure 5 show the practical consequence: Eva-4B trained on Opus-only labels plateaus at a loss of 0.56, while Eva-4B trained on MMC-arbitrated labels reaches 0.007—an 80× difference that reflects the internal consistency of the labels, not just their individual accuracy.

This finding has implications beyond evasion detection. It suggests that for any annotation task where individual annotators (human or LLM) have stable but divergent interpretive thresholds—detecting toxicity severity, assessing argument quality, identifying misinformation, rating response helpfulness—single-model annotation can produce datasets that are internally inconsistent in ways that prevent models from learning smooth decision boundaries. The MMC framework provides a template for diagnosing this problem (compare label distributions across annotators on disagreement samples) and a mechanism for addressing it (heterogeneous judge panel with randomized presentation order and majority voting). This is a concrete methodological contribution that other annotation projects can adopt directly.

The paper also establishes evasion detection as a testbed for discourse-level pragmatic reasoning, distinguishing it from the sentence-level or document-level classification tasks that dominate NLP benchmarks. Evasion detection requires assessing whether a response satisfies the communicative obligations created by a question—a relational judgment about discourse coherence rather than an intrinsic property judgment about the response text. The three-level taxonomy, grounded in Gricean pragmatics (degrees of Relation maxim violation) and empirically calibrated to match human agreement thresholds (five levels collapsed to three because κ < 0.5), provides a principled operationalization of this pragmatic reasoning challenge. This opens a line of evaluation that goes beyond what sentiment analysis, stance detection, or factual QA can assess—whether models can perform the kind of discourse-level reasoning that pragmatics has theorized for decades.

A key reconciliation the paper enables is between the theoretical richness of evasion research across disciplines (political communication, legal testimony, financial disclosure) and the practical paucity of NLP resources for studying it. Prior work demonstrated that evasion is pervasive (Gow et al., 2021: 11% of analyst questions receive non-answers), consequential (Barcellos, 2025; Larcker and Zakolyukina, 2012: predicts earnings misses and stock underperformance), and theoretically well-motivated (Grice, 1975; Brown and Levinson, 1987; Bavelas et al., 1990). But the field had no benchmark to make cumulative progress—each study constructed its own small dataset, used different taxonomies, and could not compare methods. EvasionBench fills this gap with the first large-scale resource that provides both training data (84K samples) and a rigorously validated evaluation set (1K human-labeled samples with κ = 0.835 IAA). This changes evasion detection from a niche topic where each paper starts from scratch to a shared task where modeling innovations can be compared on a common benchmark.

The small-model-on-high-quality-data result—Eva-4B at 4B parameters matching or exceeding frontier models at presumably 100B+ parameters on this task—inverts the typical contribution hierarchy in ML benchmark papers. The primary contribution is the annotation pipeline and dataset, not the model architecture. Eva-4B serves as an assay that measures the quality of the MMC annotation methodology: the +4.3 pp improvement from Eva-4B (Opus Only) to Eva-4B (Full) isolates the causal effect of multi-model consensus, and the fact that Eva-4B (Full) outperforms GPT-5.2 by 4.0 pp demonstrates that MMC data quality can compensate for a ~100× or greater parameter count difference. This shifts attention from "how do we build better evasion classifiers?" to "how do we produce training data that is internally consistent enough to teach evasion detection effectively?"—a reframing with implications for any domain where annotation quality, not model capacity, is the binding constraint.

Follow-Up Research This Work Enables

Direct MMC-human agreement measurement to establish the annotation quality ceiling. The paper validates MMC indirectly—through Eva-4B's performance against human labels (84.9% Macro-F1, Table 5) and through human-human IAA on the gold labels (κ = 0.835, Table 4)—but never directly compares MMC-produced labels to human judgments on the same samples. A direct measurement would randomly sample 500–1,000 MMC-labeled training examples, have 2–3 human experts label them independently, and compute agreement statistics (Cohen's κ, Macro-F1 treating one set as ground truth). This would answer two critical questions the current paper leaves open: (1) What is the upper bound on classifier performance given MMC labels? If MMC agrees with humans 92% of the time, then the ~85% Macro-F1 achieved by Eva-4B and frontier models might be near the ceiling, meaning further progress requires better annotation, not better models. If MMC agrees with humans 97% of the time, substantial modeling headroom remains. (2) Are the remaining errors in Eva-4B (15.2% of Gold 1K samples, Figure 6) caused by MMC label errors or by model limitations? Tracing each Gold 1K error back to whether the corresponding training labels were correct would distinguish annotation-driven vs. model-driven error sources.

Cross-domain transfer of the EvasionBench taxonomy and models to political interviews. The paper hypothesizes that the annotation framework and taxonomy "could transfer to other adversarial Q&A settings (e.g., political interviews, legal depositions)" (Section 1) but provides zero evidence. A direct test would involve: (1) constructing a 500–1,000 sample political interview dataset (e.g., from press conferences, Sunday morning talk shows, parliamentary question time) following the same Q&A pair extraction and filtering approach; (2) having 2–3 human annotators label it using the EvasionBench three-level taxonomy without modification; (3) measuring IAA to determine whether the taxonomy is operationalizable in the new domain or needs recalibration; (4) evaluating Eva-4B zero-shot on this political dataset to measure domain transfer performance; (5) fine-tuning Eva-4B on in-domain political data (potentially using MMC annotation again) and comparing against the earnings-call-trained model. The key measurements would be: (a) Does the three-level taxonomy achieve κ > 0.8 in political interviews without modification, or does the optimal number of categories differ across domains? (b) Does Eva-4B transfer at all (Macro-F1 significantly above 33.3% random baseline), or does the linguistic register shift cause catastrophic forgetting? (c) Are evasion strategies domain-specific (e.g., politicians use "pivoting to talking points" while executives use "citing competitive sensitivity"), or do they share underlying pragmatic patterns that transfer?

Combining Eva-4B with the PRM search and revision mechanisms to study evasion detection as a multi-step reasoning task. The current paper treats evasion detection as a single-pass classification: input a question-answer pair, output a label. But the paper's own error analysis reveals that the dominant failure mode involves pragmatic reasoning failures—the model confuses hedging language with evasion (52% of errors, Section 9) and cannot distinguish qualitative answers to quantitative questions from genuine evasion. These are cases where a human annotator performs multi-step reasoning: first parse the question's core informational demand (a number, a timeline, a yes/no), then extract the response's actual informational content, then assess whether the latter satisfies the former. A natural extension would be to frame evasion detection as a chain-of-thought reasoning task rather than a direct classification, fine-tuning Eva-4B to generate structured reasoning traces (question type → information requested → information provided → match assessment → label) before producing the final classification. The MMC-labeled training data could be augmented with reasoning traces generated by frontier models (using the three-judge framework to validate trace quality). The testable prediction is that reasoning-augmented models would show disproportionate improvement on the intermediate class (F1-I, currently 80.1% for Eva-4B Full) by reducing hedging-induced false positives and qualitative-vs-quantitative mismatches. A negative result—reasoning traces providing no improvement—would suggest that the ambiguity in intermediate cases is inherent to the pragmatic phenomenon and cannot be resolved through additional reasoning steps, which would be an important finding about the limits of discourse-level reasoning in current architectures.

Verifier over-optimization analysis adapted from the test-time compute literature for evasion detection calibration. The paper that this summary accompanies (on compute-optimal test-time scaling) documented how verifier-guided search can over-optimize to exploit reward model weaknesses, leading to degrading performance on easy problems at high budgets. An analogous phenomenon may exist in evasion detection: training on MMC labels that achieve high agreement with humans (κ = 0.835) could still produce a classifier that is overconfident on boundary cases. A specific experiment would: (1) measure Eva-4B's calibration error (expected calibration error, ECE) and reliability diagrams on the Gold 1K set, stratified by difficulty (using the IAA disagreement rate as a proxy for sample difficulty—the 11% of samples where human annotators disagree most are the hardest); (2) test whether Eva-4B's confidence correlates with human annotator agreement (the model should be less confident on samples where humans disagree); (3) compare calibration against zero-shot frontier models to test whether fine-tuning on MMC data improves or degrades calibration; (4) apply test-time compute strategies (e.g., generating multiple classifications with different prompting, using majority voting) specifically to the subset of samples where the model's confidence is below a threshold, measuring whether selective compute allocation improves accuracy without over-optimization. The key finding would be whether there is a "calibration ceiling" mirroring the ~85% performance ceiling, and whether test-time compute can push through it on the hardest cases or whether it simply over-optimizes against an imperfect training signal (analogous to beam search degrading easy-problem performance through PRM exploitation).

Binary vs. ternary classification comparison with downstream utility measurement. The paper's taxonomy uses three levels (direct, intermediate, fully evasive) and argues that binary classification "loses the distinction between 'partial answer' and 'complete deflection'—information critical for downstream applications like investor alerting" (Section 3.2). This claim is asserted but never tested. A direct experiment would: (1) train an Eva-4B binary variant (direct vs. evasive, collapsing intermediate and fully evasive) on the same MMC-labeled data with binary labels; (2) compare both models on a downstream task—e.g., predicting whether an earnings call containing evasive responses precedes an earnings miss or negative stock return (using the established findings of Barcellos, 2025; Larcker and Zakolyukina, 2012); (3) measure whether the three-class model's ability to distinguish intermediate from fully evasive provides incremental predictive power over the binary model. If the intermediate category captures a genuinely distinct communicative phenomenon with distinct downstream consequences (e.g., intermediate evasion predicts smaller earnings misses while fully evasive predicts larger ones), the three-class model should show superior predictive performance. If intermediate and fully evasive responses have similar downstream consequences, the extra granularity provides no practical benefit for this application, and the cost of three-way annotation (harder to achieve high IAA) may not be justified.

Cost-efficiency analysis comparing MMC against single-model annotation at larger scale. The MMC framework requires approximately 2.48× the API calls of single-model annotation (two primary annotators + three judges for 16.1% of samples). The paper demonstrates that MMC labels produce a +4.3 pp Macro-F1 improvement over Opus-only labels on the same 84K samples (Table 5). But a practitioner with a fixed annotation budget faces a different choice: spend the budget on MMC annotation of a smaller corpus, or on single-model annotation of a larger corpus (2.48× more samples for the same cost). A direct comparison would: (1) train Eva-4B on 84K MMC-labeled samples (current Eva-4B Full, 84.9% Macro-F1); (2) train Eva-4B on 208K Opus-only-labeled samples (2.48× the data, same total cost); (3) train Eva-4B on 208K GPT-5.2-only-labeled samples (to test whether single-model label quality varies by annotator, given Figure 2's distributional differences); (4) measure whether data quantity can compensate for label quality at realistic annotation budgets. The prediction from the training dynamics (Figure 5: Opus-only labels plateau at 0.56 loss) is that additional Opus-only data would not improve performance beyond the 80.6% achieved by Eva-4B (Opus Only), because the label inconsistency prevents the model from learning a smooth decision boundary regardless of sample count. If this prediction holds, it provides a strong economic argument for MMC: you cannot buy your way out of label inconsistency with more samples. A negative result—208K Opus-only samples matching or exceeding 84K MMC samples—would suggest that practitioners should prefer single-model annotation at larger scale.

Practical Applications and Downstream Use Cases

Automated screening for investor relations and corporate disclosure monitoring. The paper's findings enable a deployment where Eva-4B (or a similarly trained classifier) processes earnings call transcripts in near real-time, flagging Q&A exchanges classified as intermediate or fully evasive for review by investor relations teams, analysts, or compliance officers. The 84.9% Macro-F1 and 92.4% F1-E (Table 5) mean the system is highly reliable at detecting clear evasion and reasonably reliable at detecting partial evasion, while the 82.2% F1-D means it rarely misclassifies clearly direct answers—which is important for a screening tool that should minimize false alarms. The 4B parameter count and open-source availability enable deployment on institutional hardware without API costs, processing thousands of transcripts per quarter at negligible compute expense. The specific use case builds on the established finding that evasive communication predicts negative financial outcomes (Barcellos, 2025; Larcker and Zakolyukina, 2012): a screening system that quantifies evasion patterns across companies and over time could serve as an early-warning indicator, with systematic increases in evasion frequency at a particular company preceding negative disclosures.

Training data generation for self-improving financial NLP models. The MMC annotation framework is not specific to evasion detection—it provides a general methodology for generating high-quality labeled data for any financial NLP classification task where individual LLM annotators exhibit systematic biases. A financial institution with access to proprietary earnings call data (e.g., from non-U.S. markets or private company calls not in S&P Capital IQ) could replicate the EvasionBench pipeline: (1) extract Q&A pairs using the same structured parsing; (2) run dual-LLM annotation with Claude Opus 4.5 and Gemini 3 Flash; (3) apply three-judge arbitration to the ~16% of disagreement samples; (4) train a domain-specific classifier on the resulting data. The key finding that enables this is the loss curve divergence in Figure 5: MMC labels produce a training set that a 4B-parameter model can fit to near-zero loss, suggesting the annotation methodology is the binding constraint, not model capacity. The 84K-sample training set provides a template for the scale needed—practitioners can estimate that ~80K MMC-annotated samples suffice to train a strong classifier. The cost of MMC annotation (~2.48× single-model cost, with the 16.1% disagreement rate from Section 5.4 enabling budget estimation) can be weighed against the cost of expert human annotation at comparable scale, which would be prohibitive for most organizations.

Academic research on strategic communication across disciplines. EvasionBench provides a shared resource that can align evasion research across accounting, finance, political science, and communication studies—fields that have historically studied evasion using different taxonomies, small datasets, and incompatible methods. A political communication researcher studying whether evasion rates in press conferences predict electoral outcomes could: (1) construct a political Q&A dataset using the same extraction and filtering methodology; (2) use Eva-4B as an off-the-shelf evasion detector (acknowledging domain transfer limitations) or fine-tune it on a small amount of in-domain MMC-labeled data; (3) conduct the kind of large-scale correlational analysis that is routine in financial NLP (e.g., Koval et al., 2023; Huang et al., 2014) but has been infeasible in political communication due to the lack of automated evasion detection. The three-level taxonomy provides the granularity needed to distinguish between partial evasion (intermediate) and complete deflection (fully evasive), which may have different effects on audience perception—a distinction that binary taxonomies cannot capture and that prior small-scale human-coded studies could not test systematically. The grounding in Gricean pragmatics (Section 3.2) provides theoretical coherence across disciplines, enabling findings about evasion in one domain to inform hypotheses in another.