M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset

ArXiv: 2506.02510

🎯 Pitch

Even the best long-context LLMs miss more than half the QA pairs hidden in financial meeting transcripts. M3FinMeeting reveals this gap by testing models on real meetings in three languages across all 11 GICS sectors, demanding they generate answers, summaries, and entire question–answer pairs from hours-long discussions. The result is a stark benchmark showing that top models like Qwen2.5-72B-Instruct only reach a GPT-4-judged score in the low 70s, leaving core financial comprehension far from solved.

---

1. Executive Summary

This paper introduces M³FinMeeting, a novel benchmark dataset for evaluating large language model understanding of financial meetings — moving beyond conventional static sources like news articles and earnings reports to capture the real-world dialogue dynamics of transcribed meeting speech. The framework mandates evaluation across three tightly scoped tasks — summarization, QA pair extraction, and question answering — on 600 meetings spanning English, Chinese, and Japanese, covering all 11 GICS-defined industry sectors. Seven long-context LLMs (including GPT-4o and Qwen2.5-72B-Instruct) are scored via a GPT-4–based judge across five quality criteria, revealing that even the top-performing model, Qwen2.5-72B-Instruct, achieves an overall GPT-4-Judge score only above 70 out of 100, with QA pair extraction recall failing to exceed 45.65% — establishing that today’s most advanced long-context models leave substantial room for improvement on meeting-length financial reasoning, particularly when both questions AND answers must be discovered rather than simply answered.

2. Context and Motivation

The Core Problem: Financial Meeting Understanding Remains Unevaluated

Financial meetings — whether earnings calls, roadshows, brokerage strategy sessions, or industry exchanges — are where critical decisions get made, strategies get debated, and information gets transmitted between professionals in real time. Unlike static financial documents (annual reports, SEC filings, press releases), meetings feature multi-party dialogue, turn-taking dynamics, implicit references, and a mixture of prepared remarks and spontaneous Q&A. Understanding these meetings requires models to track who said what, resolve anaphoric references across long temporal spans, distinguish substantive financial content from procedural chitchat, and synthesize coherent summaries across multiple discussion topics.

The paper argues that current financial NLP benchmarks systematically fail to capture this reality because they draw from fundamentally different data sources. Table 1 catalogs 15+ existing financial benchmarks, and a clear pattern emerges: none use meeting transcripts as their primary source. FinQA (Chen et al., 2021) and ConvFinQA (Chen et al., 2022) use earnings reports. ECTSum (Mukherjee et al., 2022) uses earnings call transcripts but only for telegram-style bullet point summaries — it doesn't evaluate question answering or QA pair extraction from the transcript itself. FLUE (Shah et al., 2022) and FLARE (Xie et al., 2024) aggregate existing datasets covering sentiment detection, NER, and headline classification — none of which come from meetings. The CCKS series (Tianchi, 2019–2022), DCFEE (Yang et al., 2018), Doc2EDAG (Zheng et al., 2019), and CFinDEE (Zhang et al., 2024a) focus on event extraction from Chinese financial announcements and news. CFLUE (Zhu et al., 2024) and BBT-CFLEB (Lu et al., 2023) offer heterogeneous Chinese financial benchmarks, but again: no meetings.

This matters because the gap between existing benchmarks and real-world financial workflows is large and consequential. Financial analysts don't spend their days classifying sentiment in pre-chunked sentences — they listen to (or read transcripts of) hour-long calls and need to extract what matters: what questions were raised, what were the answers, what are the key takeaways per topic, and what action items emerged. A model that excels at FinQA-style numerical reasoning over a structured earnings table may be completely lost when the same information is buried in a meandering 15,000-token dialogue where the CEO hedges a revenue forecast across three separate exchanges. The paper's opening framing captures this directly:

"it remains uncertain how [LLMs] can effectively understand and process lengthy speech texts to help financial professionals expedite their work. Key capabilities, such as summarizing crucial points, responding to inquiries, and extracting question-answer pairs, are particularly beneficial for enhancing productivity and facilitating informed discussions in this context."

Why This Gap Is Practically Urgent

The paper identifies several converging trends that make financial meeting understanding a timely benchmark problem:

1. LLMs are increasingly deployed in real-time meeting contexts. With models like GPT-4o supporting audio inputs and long-context windows exceeding 100K tokens, the technical barriers to processing full meeting transcripts have largely disappeared. What's missing is an evaluation framework that tells us whether these models actually understand what happened in the meeting — or merely produce plausible-looking outputs that miss critical financial nuance. Without a purpose-built benchmark, both model developers and financial practitioners are flying blind.

2. Financial meetings have structural properties that stress-test core LLM capabilities. Meeting transcripts are long (averaging ~10,000–13,000 tokens in this dataset, with the longest exceeding 25,000), multi-speaker, and loosely structured. They require:

• Long-context reasoning: a question asked at minute 5 might get answered at minute 45, with the answer only making sense if you remember the question.
• Information density: most of the transcript is filler, but scattered throughout are precise numerical claims, forward-looking statements, and negotiation signals that carry enormous financial weight.
• Multilingual real-world content: global financial markets operate in multiple languages, and any practical meeting understanding system must handle the linguistic diversity of international business.
• Implicit topic segmentation: meetings naturally shift between topics (Q1 revenue → product pipeline → regulatory update → competitive landscape), and understanding requires tracking these transitions without explicit section markers.

3. There's a mismatch between how current benchmarks evaluate and what users need. Most financial QA benchmarks provide questions to the model. In real meetings, the model must discover both what questions were asked and what the answers were. The paper's QA pair extraction task explicitly targets this gap: it tests whether a model can read a transcript and identify "here is a question that was posed, and here is where it was answered" — a fundamentally different capability from answering a pre-extracted question.

Where Existing Benchmarks Fall Short

The paper's critique of prior work is specific and structural, not merely that past benchmarks are "too easy." The shortcomings are organized along several dimensions:

Source Material: Static Documents vs. Dynamic Dialogue

Existing financial benchmarks overwhelmingly source from edited, structured, monologic text:

• Earnings reports and SEC filings (FinQA, ConvFinQA, FinanceBench, FinTextQA): These are carefully written, legally reviewed documents with explicit numerical tables and standardized section headers. Understanding them is largely an exercise in information retrieval and table-parsing — there's no turn-taking, no hedging, no spontaneous clarification.
• Financial news and announcements (FLUE, FLARE, CCKS series, CFLUE): These are professionally written for public consumption, with clear narrative arcs and minimal ambiguity. They don't capture the messy, incremental nature of spoken financial communication.
• Existing meeting corpora (AMI, ICSI): These do contain meeting speech but are domain-general (academic project meetings, research discussions) and lack financial content entirely. They're also relatively small and English-only.

ECTSum (Mukherjee et al., 2022) comes closest to this paper's domain — it uses earnings call transcripts — but is limited to English, provides only bullet-point summaries, and doesn't include QA extraction or question answering tasks. It also doesn't span the full GICS sector taxonomy, which matters because financial discussions in healthcare versus energy versus real estate involve different terminology, different regulatory frameworks, and different types of quantitative reasoning. A model that can summarize a tech earnings call may struggle with the specialized vocabulary and metrics of a utilities sector discussion.

Language Coverage: Monolingual by Design

With the exception of heterogeneous benchmarks that aggregate across datasets (FLUE, FLARE, CFLUE), nearly every financial benchmark is monolingual — English-only or Chinese-only. None prior to M³FinMeeting support Japanese alongside English and Chinese. For global financial applications, this is a severe limitation. An analyst covering Asian markets needs to process earnings calls in Japanese and Chinese alongside English-language reports. A benchmark that only evaluates English performance provides no signal about cross-lingual financial understanding, which is the actual deployment scenario for most international financial firms.

The paper's choice of English, Chinese, and Japanese is deliberate and motivated by real market coverage: these three languages together represent the communication languages of the three largest economies in the world (U.S., China, Japan), and financial firms operating across these markets routinely handle meetings in all three.

Task Design: What Gets Evaluated vs. What Matters

Existing benchmarks prioritize tasks that are relatively easy to evaluate with automated metrics:

• Numerical reasoning over extracted tables (FinQA, ConvFinQA, BizBench): The model is given a structured table and a question; the answer is a number. Evaluation is exact match or execution accuracy. This is clean but bears little resemblance to finding the right number in an unstructured 15,000-token dialogue.
• Multiple-choice QA (CFLUE, some FLUE tasks): Even further from real use — financial analysts don't get four answer options for their actual questions.
• Classification tasks (sentiment, NER, news headline): Low-dimensional categorical outputs that miss the richness of what meeting participants actually need (synthesis, extraction, summarization).

The paper argues that for meeting understanding to be practically useful, models must handle open-ended generation tasks with long outputs: producing a multi-section summary that covers all discussion topics, extracting a structured list of all QA pairs that occurred, or answering a batch of questions with complete, grounded responses. These tasks are inherently harder to evaluate (requiring LLM-as-judge approaches rather than exact match), which is precisely why they haven't been benchmarked before — but they're also precisely what financial professionals need.

Length: Most Benchmarks Are Short-Document

The paper's data statistics (Table 2) reveal that M³FinMeeting meetings average ~10,000–13,000 tokens, with the longest fifth of the dataset exceeding 20,000 tokens (and Set5 averaging 25,281 tokens). Even the "short" meetings (Set1, 0–5K tokens) average 3,546 tokens — longer than most paragraphs in standard QA datasets. This is a direct stress test for long-context LLMs: can they maintain coherent understanding across 60+ minutes of transcribed speech, where relevant information for any given question may be scattered across non-contiguous sections?

The paper explicitly connects this to recent work on long-context evaluation (Li et al., 2024's LooGLE; Wang et al., 2024's Loong; Bai et al., 2024b's LongBench), but notes that none of these long-context benchmarks include financial meeting data. The combination of domain-specific financial complexity + extreme length is unique to M³FinMeeting and represents a qualitatively different challenge from either generic long-context QA or short-document financial QA.

How This Paper Positions Itself

The paper does not claim to solve financial meeting understanding. It positions M³FinMeeting as a diagnostic benchmark — a tool for measuring where current models succeed and where they fail, with sufficient granularity (per-language, per-sector, per-length, per-task) to guide future research.

The framing is explicitly gap-focused rather than method-focused. Section 1 states the motivation plainly: existing benchmarks are sourced from news/reports, are monolingual, and lack meeting-specific tasks. M³FinMeeting fills all three gaps simultaneously: it replaces static sources with meeting transcripts, supports three languages, and tasks that mirror real analyst workflows (summarize the meeting, extract all QA pairs, answer a batch of questions about the meeting).

The paper draws a deliberate contrast with the broader trend in financial NLP toward training-domain datasets (FLANG, Pixiu, InvestLM, FinGPT, DianJin-R1), which provide large corpora for fine-tuning but are not designed as standardized evaluation benchmarks. M³FinMeeting is positioned purely for evaluation — its 600 meetings with manual annotations serve as a held-out test of understanding, not a training resource.

The choice to evaluate seven models (two API-based, five open-source) spanning a wide range of parameter counts (7B–72B+) and context windows (16K–1000K) serves a dual purpose: it establishes baseline performance levels and simultaneously reveals which model properties correlate with meeting understanding capability. The finding that Qwen2.5-72B-Instruct — an open-source model — outperforms GPT-4o on this benchmark is itself a signal that meeting understanding may require different architectural or training choices than those prioritized in general-purpose LLM development.

Finally, the paper's multi-metric evaluation design (precision/recall/F1 for section and question alignment, plus GPT-4-Judge for overall quality across five criteria, plus BLEU/ROUGE as traditional baselines, plus human evaluation with Fleiss' Kappa) reflects a recognition that meeting understanding is too complex for any single automated metric. By reporting performance across this full suite, the paper provides a rich diagnostic picture: for example, the observation that LLMs achieve near-perfect F1 on question repetition in the QA task (>90% for most models in Table 5) but abysmal recall on discovering questions in the QA pair extraction task (<46% for all models) reveals that the bottleneck is not in following instructions or generating coherent text — it's in detecting which utterances in a long dialogue constitute financially meaningful questions in the first place. This kind of fine-grained diagnosis is exactly what the field needs to make progress, and it's only possible because the benchmark pushes beyond simple answer-matching into tasks that require genuine comprehension of dialogue structure.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a dataset and benchmark paper — it does not propose a new model or algorithm, but rather constructs a carefully designed evaluation framework (M³FinMeeting) and uses it to measure the current capabilities of long-context LLMs on financial meeting understanding. The system being built is a multi-dimensional evaluation pipeline: given a financial meeting transcript, it tests a model's ability to perform three distinct comprehension tasks, scores the outputs using both alignment-based metrics (precision/recall/F1) and an LLM-as-judge quality assessment across five criteria, and reports performance broken down by language, industry sector, and input length. The core idea is that financial meeting understanding — unlike understanding news articles or earnings reports — requires models to process long, multi-speaker, loosely structured dialogue to discover both the questions being asked AND their answers, to synthesize coherent section-by-section summaries, and to do all of this across multiple languages. The "solution" is not an architectural innovation but an evaluation design decision: by constructing tasks that mirror what financial analysts actually need from meeting transcripts, and by annotating ground-truth references at the section level rather than just document level, the benchmark reveals where models genuinely comprehend dialogue structure versus where they produce plausible-sounding but semantically misaligned outputs.

3.2 Big-Picture Architecture (Diagram in Words)

The M³FinMeeting evaluation pipeline consists of five major components, processed in sequence:

1. Meeting Transcript Corpus (600 annotated meetings): Raw financial meeting audio is collected across three languages (100 English, 400 Chinese, 100 Japanese) spanning all 11 GICS sectors, transcribed via OpenAI's Whisper ASR toolkit, and manually corrected by trained financial analysts. Each corrected transcript serves as the input document for all downstream tasks.

2. Task Definition Layer (three evaluation tasks): For each meeting transcript, the benchmark defines three separate tasks — summarization (generate section-level summaries that collectively cover the full meeting), QA pair extraction (discover all financially meaningful question-answer pairs embedded in the dialogue), and question answering (given the gold-standard questions, produce accurate answers within the context of the source sections). Each task has its own prompt template and output format specification.

3. Model Inference (seven LLMs evaluated): Each task-specific prompt is fed to each of seven LLMs (GPT-4o-128K, GPT-3.5-turbo-16K, GLM4-9B-Chat-1000K, Llama3.1-8B-Instruct-128K, Qwen2-7B-Instruct-128K, Qwen2-72B-Instruct-128K, Qwen2.5-72B-Instruct-128K) in a zero-shot setting. The models generate structured outputs: section summaries for summarization, extracted QA pairs for QA pair extraction, and question-repetition-plus-answer sequences for question answering.

4. Multi-Metric Scoring Layer: Generated outputs are evaluated along two parallel tracks. Track A computes alignment metrics (precision, recall, F1) by embedding both generated and reference structural units (section summaries or individual questions) using OpenAI's text-embedding-3-small model and counting how many align with cosine similarity ≥ 0.75. Track B uses GPT-4-Judge (gpt-4-turbo-2024-04-09) to assign holistic quality scores (0–100) across five criteria: Coverage, Redundancy, Readability, Accuracy, and Consistency. A secondary judge (Qwen-plus) and human evaluation with Fleiss' Kappa confirm reliability.

5. Stratified Analysis Layer: Results are disaggregated along three orthogonal dimensions — language (EN/ZH/JA), GICS sector (all 11 categories), and input length (five token-count bins from 0–5K to >20K) — to produce diagnostic insights about where and why models succeed or fail.

Information flows linearly: a corrected meeting transcript enters the task definition layer → the task-specific prompt is constructed → the LLM generates an output → the output is scored by both the alignment metrics module and the GPT-4-Judge module → scores are aggregated and stratified by language, sector, and length for analysis.

3.3 Roadmap for the Deep Dive

• First, the annotation process and quality control mechanisms, because the validity of the entire benchmark rests on the quality and consistency of the human-created reference summaries, section segmentations, and extracted QA pairs against which models are evaluated.
• Second, the summarization task design, including how sections are defined, how summaries are structured, and how alignment metrics (precision/recall/F1) specifically measure whether models produce semantically equivalent coverage of the right topics — this establishes the core evaluation pattern reused across tasks.
• Third, the QA pair extraction task and its unique challenge: unlike standard QA where questions are provided, models must autonomously identify which utterances constitute meaningful financial questions and locate their answers in the surrounding dialogue. This task reveals the question-discovery bottleneck.
• Fourth, the question answering task design, including why questions are batched into a single prompt rather than answered individually, and how this connects to real-world analyst workflows.
• Fifth, the evaluation protocol in full detail: the dual-track scoring system (alignment metrics + GPT-4-Judge), the five quality criteria, the judge prompt templates, the bias-mitigation strategy using Qwen-plus as an alternative judge, and the human validation via Fleiss' Kappa.
• Sixth, the model configurations, prompt design, and experimental settings — the specific LLMs chosen, their context windows, the zero-shot prompting strategy, and any preprocessing or postprocessing steps.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset construction and evaluation methodology paper. Its technical contribution lies in (a) the annotation protocol and quality control that produce reliable ground-truth references for open-ended meeting understanding tasks, and (b) the multi-metric evaluation design that captures both structural alignment and holistic quality without depending on any single brittle automated metric.

---

Data Collection: Audio Sourcing, Transcription, and Manual Correction

The benchmark's raw material consists of audio recordings from real financial meetings collected under four explicit criteria stated in Section 3.3.1:

1. Timeliness: "Most meetings should be from recent years" — ensuring the benchmark reflects contemporary financial language and market conditions rather than historical artifacts.
2. Length: "Preference is given to longer audio files" — because the benchmark is explicitly designed to stress-test long-context understanding, shorter meetings would not exercise the target capability.
3. Categorizability: "The audio files must align with categories defined in the GICS" — each meeting must be assignable to one of the 11 Global Industry Classification Standard sectors (Communication Services, Consumer Discretionary, Consumer Staples, Energy, Financials, Healthcare, Industrials, Information Technology, Materials, Real Estate, Utilities), enabling stratified analysis by industry.
4. Authoritativeness: "All audio files are sourced from our financial firm partners and are protected by our copyright" — this addresses both legal compliance and data provenance; the meetings are genuine financial events (public roadshows, brokerage strategy meetings, industry exchanges, earnings presentations) rather than synthetic or web-scraped content.

The collection spans 600 meetings total: 100 in English (EN), 400 in Chinese (ZH), and 100 in Japanese (JA). The strong Chinese skew (two-thirds of the corpus) reflects the authors' institutional context at Alibaba Cloud and the availability of partner-provided Chinese financial meeting audio, but is explicitly noted rather than disguised.

Audio transcription pipeline. All audio files are processed through OpenAI's Whisper automatic speech recognition (ASR) toolkit. Whisper is a transformer-based encoder-decoder model trained on 680,000 hours of multilingual and multitask supervised data. The authors choose it because it provides state-of-the-art transcription quality across multiple languages without requiring language-specific fine-tuning — critical for a trilingual benchmark.

However, ASR output is inherently noisy for financial meetings: speaker disfluencies, domain-specific terminology (ticker symbols, financial metrics, regulatory acronyms), overlapping speech, and accents all degrade accuracy. The paper addresses this through "a thorough manual correction process," where trained annotators — "experienced analysts fluent in different languages" (Section 3.3.2) — listen to the original audio while editing the ASR transcript to fix errors.

De-identification. A crucial ethical and legal step: "we remove company identifiers and sensitive details using GPT-4, followed by manual verification, to maintain privacy without compromising research value." This means the released transcripts will have company names and personally identifiable information (PII) replaced or redacted, but in a way that preserves the financial content and dialogue structure. The two-stage process (automated GPT-4 de-identification + human verification) balances scalability with reliability — GPT-4 handles the bulk identification of sensitive spans, while human reviewers catch edge cases that a model might miss (e.g., indirect references to a specific company through context rather than explicit naming).

Final transcript statistics (Table 2). After processing, the corpus has these properties:

Token counts use the tiktoken tokenizer with the cl100k_base encoding (OpenAI's standard tokenizer for GPT-4 and GPT-3.5-turbo), which the authors explicitly note: "We employ the tiktoken tokenizer (cl100k_base) to process all transcriptions." This is a deliberate choice for comparability — since the evaluated models include OpenAI's API-based LLMs, using their tokenizer ensures that the reported token counts match what the models actually consume in their context windows.

Japanese meetings average the highest token count (13,284) despite having similar average duration to English (1.01 hours vs. 0.96 hours), reflecting the higher token-to-character ratio of Japanese text in the cl100k_base encoding.

Length distribution (Table 2, "Length Set" section). The corpus is further binned into five length groups for stratified analysis:

• Set1 (0–5K tokens): 59 meetings, average 3,546 tokens
• Set2 (5–10K tokens): 164 meetings, average 7,509 tokens
• Set3 (10–15K tokens): 195 meetings, average 12,476 tokens
• Set4 (15–20K tokens): 124 meetings, average 17,419 tokens
• Set5 (>20K tokens): 58 meetings, average 25,281 tokens

The distribution is roughly normal with a peak at Set3 (10–15K), which is the most common meeting length. This binning matters for the experimental analysis in Section 4.2, where the paper examines whether model performance degrades with increasing input length — a key diagnostic for long-context LLM evaluation.

GICS sector distribution. The meetings span all 11 sectors, with the most data in Consumer Discretionary (122 meetings, 11,104 avg tokens), Industrials (111 meetings), and Information Technology (98 meetings), and the least in Utilities (8 meetings, 17,290 avg tokens — the highest average token count per meeting, suggesting utility-sector meetings tend to be longer and more technically detailed). This imbalance is acknowledged implicitly in the benchmark: some sectors have more evaluation data than others, which means per-sector performance estimates have different confidence intervals. The paper reports per-sector performance for the top three models (GPT-4o, Qwen2-72B-Instruct, Qwen2.5-72B-Instruct) in Table 12 (Appendix E).

---

Annotation Process and Quality Control

The ground-truth references — section summaries and extracted QA pairs — are created by human annotators following a structured process designed to maximize consistency across annotators and languages (Section 3.3.2, Appendix F, Figure 10).

Annotator qualifications. The annotators are "experienced analysts fluent in different languages" who are "recruited from various projects." The key requirement is domain expertise: annotators must understand financial terminology well enough to identify substantive discussion topics, distinguish meaningful questions from procedural chitchat, and produce summaries that capture financially relevant information rather than surface-level descriptions. This is non-trivial — a generic annotator without finance background might summarize "the speaker talked about revenue" while missing that the specific revenue figure was adjusted EBITDA versus GAAP revenue, a distinction that matters enormously in financial analysis.

Training and calibration. Before starting annotation work, annotators undergo "comprehensive training" and receive "onboarding and guidance materials" including the annotation guidelines shown in Figure 10 (Appendix F). The paper states that annotators "undergo extensive training and discussions to achieve a high level of consensus" — this calibration phase is critical because summarization and QA pair extraction are inherently subjective tasks where different annotators might reasonably produce different outputs (e.g., what counts as a "financially meaningful" question versus a procedural one like "Can you go back to slide 12?").

Ongoing quality monitoring. A separate group of analysts "review the annotators' work and provide ongoing feedback," creating a two-tier quality process: annotators produce initial work, reviewers check for errors or inconsistencies, and feedback is incorporated into subsequent annotations. This iterative review loop is standard practice in high-quality dataset construction but is explicitly documented here, which adds credibility to the annotation quality claims.

The three annotation sub-tasks:

1. Transcribed speech correction. Annotators have "original audio files available for reference during the correction process." This means they can listen to the meeting while reading the ASR output, comparing the two to identify mistranscribed words, particularly domain-specific terms that Whisper may not have encountered frequently in training (e.g., "EBITDA" mis-transcribed as "even to that," or Chinese financial terms rendered phonetically rather than semantically). The availability of audio also helps resolve ambiguities in speaker attribution and overlapping speech.

2. Summarization. The annotation process for summarization proceeds in two stages:

• Stage A — Document segmentation: "The annotators segment each document into sections based on distinct topics, ensuring that only sections with clear boundaries are selected for summarization." This is the crucial design decision: rather than asking annotators to produce one monolithic summary of the entire hour-long meeting, the benchmark requires them to first identify where topic boundaries occur, then summarize each section independently. The instruction to only use sections "with clear boundaries" means that if a meeting's topic transitions are gradual or ambiguous, annotators should not force artificial segmentation — they should mark only the sections where a distinct topic clearly starts and ends.

• Stage B — Section summarization: "For each valid section, annotators are encouraged to use simple sentences in the summary." The emphasis on simple sentences reflects a deliberate choice: complex, multi-clause summaries are harder to evaluate (more surface forms can express the same meaning) and harder for models to generate consistently. Simple sentences make the alignment-based evaluation (cosine similarity matching) more reliable because the embedding of a simple declarative sentence is less ambiguous than that of a convoluted paragraph.

Compression ratios (Table 3). The paper reports token-level and sentence-level compression ratios between the source document and the reference summary, following Koh et al. (2022). For English meetings, the average meeting contains 9.20 section summaries totaling 927 tokens (from a source averaging 10,086 tokens, implying roughly a 10.9× token-level compression). The compression ratio at the token level and sentence level quantifies how much information the summary retains relative to the source — a lower ratio means more aggressive condensation. The paper reports these statistics per language in Table 3, though the specific values for Chinese and Japanese are not reproduced in the main text excerpt.

Why section-level rather than document-level summarization? This design choice is central to the benchmark's diagnostic value. Document-level summarization conflates two separate capabilities: (a) identifying topic boundaries and (b) condensing information within each topic. By providing section-level gold summaries, the benchmark can separately evaluate how well models perform both sub-tasks. The alignment metrics (precision/recall/F1 from Section 3.2.1) explicitly measure this: if a model generates a section summary that semantically matches a gold section, it gets credit; if it generates content that doesn't align to any gold section, it loses precision; if it misses a gold section entirely, it loses recall. A document-level summary evaluation (like standard ROUGE against a full-meeting reference summary) would not reveal whether poor performance stems from bad topic segmentation or bad condensation — M³FinMeeting's section-level design disambiguates these.

3. QA pair extraction. This annotation task is more constrained than summarization but still requires significant judgment:

• Question identification: Annotators "manually extract financially relevant questions from the text." The key qualifier is "financially relevant" — questions like "What were we just talking about?" or "Can you repeat that?" should be disregarded because they are procedural rather than substantive. The annotation guidelines (Figure 10) presumably provide examples of what constitutes a financially relevant question, though the specific criteria are not enumerated in the main paper text.

• Answer localization: "For each identified question, they search for corresponding answers in the subsequent content." The "subsequent content" constraint is important: in meetings, answers sometimes precede questions (e.g., a clarification request about something said earlier), but the annotation protocol restricts answers to text that appears after the question, creating a directional dependency that models can exploit.

• Validity filtering: "Only questions with valid answers are retained for further analysis." This means the final QA pair dataset contains only questions where the annotator could find a clear, substantive answer in the transcript. Questions that were asked but not adequately answered (e.g., "We'll have to get back to you on that") are excluded, which makes the QA task cleaner but also less realistic — real meeting analysis often involves identifying which questions went unanswered.

QA pair statistics (Table 4). The paper reports per-language statistics including: average token length of all QA pairs per meeting (QA token), average number of QA pairs per meeting (#QA), average token length of individual questions (Q token), and average token length of individual answers (A token). The specific numeric values are in Table 4, which is referenced but not fully reproduced in the main text excerpt.

Annotation quality assurance. Beyond the reviewer feedback loop, the paper does not report inter-annotator agreement metrics (e.g., Cohen's Kappa or Krippendorff's alpha) for the summarization segmentation or QA extraction tasks. This is a notable omission — for a benchmark that emphasizes annotation quality, quantitative evidence of annotator consistency would strengthen the validity claims. The paper does report Fleiss' Kappa for the evaluation phase (agreement between human evaluators and GPT-4-Judge on output quality), but not for the annotation phase that creates the ground truth. This gap is acknowledged indirectly through the "Limitations" section, which notes "High Annotation Costs and Challenges" and that summarization "often involves open-ended responses."

---

Task 1: Summarization — Design and Evaluation Mechanics

The summarization task (Section 3.2.1) evaluates whether an LLM can produce a structured, section-by-section summary of a full financial meeting transcript. The design encodes two assumptions about real-world financial analyst workflows:

Design principle 1 — Implicit segmentation. The prompt (Figure 5, Appendix C) does not provide section boundaries to the model. Instead, it instructs the LLM to "implicitly identify and segment the document into various sections and then extract key point from each" (Section 3.2.1). This means the model must perform topic segmentation from scratch — it receives the raw transcript with no section markers, headers, or hints about where topic transitions occur. The output format requires the model to produce multiple section summaries that collectively cover the meeting's content, but the model itself must decide how many sections to create and where the boundaries fall.

Design principle 2 — Hierarchical output structure. The generated output consists of multiple section summaries, each summarizing one discussion topic. The gold reference also consists of multiple section summaries (average 9.20 for English meetings). The evaluation then checks how well the generated sections align with the gold sections. This hierarchical structure is what enables the precision/recall/F1 alignment metrics: it transforms the abstract problem of "is this a good summary?" into the more tractable problem of "do the generated topic-level summaries cover the same topics as the human-annotated reference summaries?"

Prompt template (Figure 5). The prompt is written in English for all three languages, with an explicit instruction that "the output language must align with the meeting content." The template instructs the model to:

1. Read the transcript and identify distinct discussion topics.
2. For each topic, produce a concise summary.
3. Return the summaries in a structured format.

Evaluation via alignment metrics (Section 3.2.1, Appendix B). The core evaluation mechanism is section-level semantic alignment. The process works as follows:

• Step 1: Embed all sections. Each gold section summary and each generated section summary is passed through OpenAI's text-embedding-3-small model, producing a dense vector representation (embedding) for each section.

• Step 2: Compute pairwise cosine similarities. For every generated section summary $i$ and every gold section summary $j$, compute:

$\text{cosine\_similarity}(i, j) = \frac{\mathbf{e}_i \cdot \mathbf{e}_j}{\|\mathbf{e}_i\| \|\mathbf{e}_j\|}$

where $\mathbf{e}_i$ is the embedding vector for generated section $i$ and $\mathbf{e}_j$ is the embedding vector for gold section $j$.

What it computes: the cosine of the angle between the two embedding vectors, ranging from -1 (opposite direction, semantically dissimilar) to 1 (identical direction, semantically identical). Values close to 1 indicate that the text-embedding-3-small model represents the two sentences as having very similar meanings.

Why this form: cosine similarity is the standard metric for comparing semantic similarity in embedding spaces because it is insensitive to vector magnitude (which typically correlates with sentence length rather than meaning). Using Euclidean distance would penalize summaries of different lengths even if they express the same content; using dot product would be length-sensitive. Cosine similarity isolates the angular difference, which better captures semantic alignment.

• Step 3: Threshold alignment. A generated section is considered "aligned" to a gold section if their cosine similarity exceeds 0.75. The threshold 0.75 is a hyperparameter chosen by the authors (not justified with an ablation study, which is a minor limitation). Higher thresholds would be stricter (requiring near-identical meaning), while lower thresholds would be more permissive (allowing looser semantic correspondence). The choice of 0.75 reflects a balance: it's high enough that random pairings of unrelated financial sentences would rarely exceed it, but low enough that paraphrases and minor wording variations still count as matches.

• Step 4: Count alignments. Let $m$ be the total number of generated section summaries, and $n$ be the total number of gold section summaries. After thresholding, count:

  • $m_a$ = number of generated sections that align to at least one gold section
  • $n_a$ = number of gold sections that align to at least one generated section

The paper explicitly notes: "$m_a$ and $n_a$ do not need to be equal, as one section summary from one side can be aligned to multiple section summaries on the other side." This means the alignment is many-to-many: a single generated section that covers multiple topics could align to multiple gold sections (reducing recall), and multiple generated sections that redundantly cover the same topic would all align to the same gold section (reducing precision).

• Step 5: Compute precision, recall, F1. The metrics follow standard definitions but applied to section alignment counts:

$\text{Precision} = \frac{m_a}{m}$

where $m_a$ is the number of generated sections that found a semantically matching gold section, and $m$ is the total number of generated sections.

What it computes: the fraction of generated section summaries that are semantically grounded in the reference. Low precision means the model is producing section summaries that don't correspond to any topic the human annotators identified — either because the model hallucinated topics, created overly fine-grained splits that don't match the gold segmentation, or generated content that is factually off-topic.

Why this form: precision penalizes over-generation. If a model generates 30 section summaries but only 10 of them match gold sections, precision is 10/30 = 33.3%, even if those 10 matches are excellent. This incentivizes models to produce a number of sections proportional to the actual meeting structure rather than inflating the count with spurious summaries.

$\text{Recall} = \frac{n_a}{n}$

where $n_a$ is the number of gold sections that have at least one semantically matching generated section, and $n$ is the total number of gold sections.

What it computes: the fraction of reference topics that the model's summary covers. Low recall means the model missed important discussion topics entirely — the gold reference has sections that the generated summary never addresses.

Why this form: recall penalizes under-generation or topic omission. If a meeting covers 10 distinct financial topics but the model only produces summaries that align to 4 of them, recall is 4/10 = 40%, even if those 4 are perfectly summarized. This captures a critical failure mode for meeting summarization: models that latch onto a few salient topics and ignore others.

$\text{F1} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

What it computes: the harmonic mean of precision and recall, providing a single balanced score. F1 is low if either precision OR recall is low — it rewards models that achieve both good coverage (high recall) and good alignment (high precision) simultaneously.

Why this form: the harmonic mean penalizes extreme imbalance more severely than the arithmetic mean would. A model with precision = 100% and recall = 0% (generated one perfect section but missed all others) gets F1 = 0%, not 50%. This is desirable because both aspects matter for practical meeting summarization: missing important topics is as harmful as generating spurious ones.

The paper reports that "precision, recall, and F1 scores for section-level summaries are all below 30%" for all models (Section 4.2), indicating severe difficulty with both topic segmentation and coverage — models neither produce the right number of sections nor cover the right topics within them.

Document-level GPT-4-Judge evaluation. In addition to section-level alignment, the paper uses GPT-4 (gpt-4-turbo-2024-04-09) as a judge to evaluate the complete document-level summary across five criteria (Figure 8, Appendix C):

• Coverage: Does the summary include all important points from the meeting?
• Redundancy: Does the summary avoid repeating the same information?
• Readability: Is the summary well-written and easy to follow?
• Accuracy: Are the facts in the summary correct according to the transcript?
• Consistency: Is the summary internally consistent without contradictions?

Each criterion receives a score from 0 to 100. The reported GPT-4-Judge score for summarization is the average across these five criteria. The prompt template (Figure 8) provides explicit scoring rubrics for each criterion to standardize GPT-4's judgments.

Why both alignment metrics AND LLM judging? The two evaluation tracks capture different aspects of quality. Alignment metrics (precision/recall/F1) measure structural correspondence — does the summary cover the right topics? — but don't assess factual accuracy, readability, or conciseness. A summary could achieve high alignment (all gold topics are mentioned) while being verbose, redundant, or containing hallucinated details within each section. GPT-4-Judge captures these holistic qualities but is itself imperfect (potential biases toward certain model outputs, sensitivity to prompt wording, inconsistent scoring across runs). Using both provides a more complete picture than either alone.

---

---

Task 2: QA Pair Extraction — Design and Evaluation Mechanics

The QA pair extraction task (Section 3.2.2) is perhaps the most distinctive and challenging component of M³FinMeeting. Unlike standard QA benchmarks where the questions are provided to the model, QA pair extraction requires the model to autonomously discover which utterances in the transcript constitute meaningful financial questions and then locate their answers. This directly tests a capability that financial analysts need: given a long meeting transcript, what questions were raised during the discussion, and what were the responses?

The core challenge: question identification in dialogue. Financial meeting transcripts contain many types of utterances that could be syntactically questions but are not financially substantive:

• Procedural questions ("Can we go to the next slide?")
• Clarification requests ("What did you just say?")
• Rhetorical questions ("Isn't that obvious?")
• Backchanneling ("Really?")

The benchmark requires models to distinguish these from substantive financial questions like "What is your revenue guidance for Q3?" or "How do you plan to address the competitive pressure from your main rival in the Asia-Pacific market?" This is fundamentally an information extraction task operating over long-form dialogue: the model must scan thousands of tokens of multi-speaker conversation and identify the small subset of utterances that carry financial inquiry intent.

Prompt design (Figure 6, Appendix C). The prompt instructs the LLM to extract financially relevant questions from the transcript and provide their corresponding answers. The two-stage nature of the prompt ("we first prompt the LLMs to extract all questions, then provide answers for each sequentially" — Section 4.1) means the model is guided to first identify questions, then search for answers, rather than being asked to produce question-answer pairs in a single unstructured pass. This separation is intended to improve completeness: by making question identification an explicit intermediate step, the model is less likely to skip over questions that are hard to answer.

Output format. The model must produce a structured list of QA pairs, with each pair containing the extracted question and its answer from the transcript. The expected output is longer and more structured than typical QA outputs — it requires the model to both extract content (finding the question and answer spans) and format it (presenting them as paired units).

Evaluation via alignment metrics (same mechanism as summarization). The QA pair extraction task is evaluated using the same precision/recall/F1 framework as summarization, but with generated questions replacing generated section summaries, and gold questions replacing gold section summaries:

• Each generated question and each gold question is embedded using text-embedding-3-small.
• A generated question aligns to a gold question if cosine similarity exceeds 0.75.
• Let $m$ be the number of generated questions, $n$ be the number of gold questions, $m_a$ be the number of generated questions that align to at least one gold question, and $n_a$ be the number of gold questions aligned to at least one generated question.
• Precision = $m_a / m$, Recall = $n_a / n$, F1 = harmonic mean.

What these metrics capture in the QA context:

• Precision: The fraction of extracted questions that correspond to actual substantive financial questions from the meeting. Low precision means the model is extracting procedural, rhetorical, or irrelevant utterances as "questions" — a common failure mode where the model over-extracts, treating every question mark in the transcript as a financial query.
• Recall: The fraction of gold-standard questions that the model successfully discovers. Low recall means the model is missing financially important questions — perhaps because they are embedded in long multi-turn discussions where the question intent is spread across several utterances, or because the financial terminology is subtle enough that the model doesn't recognize the inquiry as a distinct question.
• F1: The balanced measure of extraction quality, penalizing both over-extraction and under-extraction.

GPT-4-Judge evaluation. As with summarization, GPT-4-Judge evaluates the extracted QA pairs using the same five criteria (Coverage, Redundancy, Readability, Accuracy, Consistency), but applied to the question-answer pairs rather than section summaries (Figure 9, Appendix C). The judge considers:

• Coverage: Are all important Q&A exchanges from the meeting captured?
• Accuracy: Are the extracted answers factually correct based on the transcript?
• Readability/Consistency/Redundancy: Are the QA pairs well-formed, non-redundant, and internally coherent?

The question-discovery bottleneck. The experimental results (Table 5) reveal a stark performance asymmetry: in the question answering task (where questions are provided to the model), most LLMs achieve F1 scores above 90% on question repetition — they can faithfully reproduce the provided questions and generate answers. But in the QA pair extraction task (where questions must be discovered), the best model (Qwen2.5-72B-Instruct) achieves only 45.65% recall, "missing more than half of the gold questions" (Section 4.2). This gap is the key diagnostic finding of the benchmark: the bottleneck in meeting understanding is not answer generation, but question detection — the ability to recognize that a financial question has been asked in the first place, amidst the noise of multi-party dialogue. This is a fundamentally harder problem that existing financial QA benchmarks (which supply questions) cannot measure.

---

Task 3: Question Answering — Design and Evaluation Mechanics

The question answering task (Section 3.2.3) evaluates the model's ability to answer questions about the meeting transcript when the questions are provided. This is the most conventional of the three tasks — it resembles standard reading comprehension QA — but with three distinctive design choices.

Design choice 1: Questions are drawn from the QA pair extraction task. The paper states: "For simplicity, we use the QA pairs described above for this task." This means the QA task reuses the gold-standard questions that were manually extracted during annotation (Section 3.3.2), removing the question-discovery burden and isolating the answer-localization capability. This creates a clean experimental contrast: QA pair extraction measures question discovery + answer localization; question answering measures answer localization only. The performance difference between these tasks quantifies the question-discovery difficulty.

Design choice 2: Section-level answer localization with distractors. The paper adds an important constraint: "the QA task tests the LLMs' capability to find evidence within that designated section, while other sections with similar but unrelated content act as noise." Meeting transcripts are divided into topic-based sections (from the summarization annotation). For each question, the model must locate the answer within the correct section of the transcript, while other sections that discuss similar topics (and might contain plausible but incorrect answers) serve as distractors. This is a more realistic test than typical extractive QA where the answer is guaranteed to be in a provided passage — in real meetings, the model doesn't know a priori which part of the hour-long discussion contains the relevant information.

Why this matters for long-context evaluation: Current long-context LLMs often struggle with "needle-in-a-haystack" retrieval — finding a specific piece of information buried in a very long context (Li et al., 2024; Wang et al., 2024). The M³FinMeeting QA task is a domain-specific instantiation of this challenge: the "needle" is the answer to a financial question, and the "haystack" is a 10,000–25,000-token transcript with multiple sections containing related financial content. The presence of similar-but-irrelevant sections creates a harder distractor condition than random-text distractor baselines.

Design choice 3: Batched question answering rather than one-at-a-time. The paper explicitly departs from conventional QA evaluation: "instead of addressing one question at a time, we combine related questions into a single prompt, allowing the LLM to produce a comprehensive response that includes all the answers" (Section 4.1). This design decision is motivated by two considerations:

• Realism: Financial analysts don't query a model with individual questions sequentially — they provide a list of questions they want answered about a meeting and expect a compiled response. Batching questions in a single prompt better approximates this workflow.
• Efficiency: It "reduces API calls," which matters practically for evaluation cost (the paper reports ~$2,500 in OpenAI API costs for the full evaluation).

The prompt template (Figure 7, Appendix C) lists all questions from a meeting and instructs the LLM to answer them sequentially in a single response. Each question is repeated before its answer, creating a structured output where question spans and answer spans are interleaved. This format also simplifies evaluation: the model's repeated questions can be aligned against the gold questions using the same embedding-based precision/recall/F1 framework, and the answers are evaluated via GPT-4-Judge.

Evaluation via question-alignment metrics. The QA task uses the same embedding-and-threshold alignment approach as QA pair extraction, but applied to question repetition rather than question discovery:

• Each model-repeated question is embedded and compared against the gold questions.
• A repeated question aligns to a gold question if cosine similarity exceeds 0.75.
• Precision = fraction of repeated questions that align to gold questions.
• Recall = fraction of gold questions that are aligned to by at least one repeated question.
• F1 = harmonic mean.

High F1 scores (>90% for most models in Table 5) indicate that LLMs are reliable at faithfully repeating provided questions — an instruction-following capability. The QA task F1 measures output formatting quality rather than comprehension, which is why it's used alongside GPT-4-Judge (which measures answer quality).

GPT-4-Judge evaluation for QA. The same five-criteria scoring (Coverage, Redundancy, Readability, Accuracy, Consistency) is applied to the question-answer pairs, with the prompt template shown in Figure 9 (Appendix C). The judge evaluates whether the answers are factually correct, comprehensive, and well-written.

---

The Evaluation Protocol: Dual-Track Scoring with Bias Mitigation

The M³FinMeeting evaluation framework employs a sophisticated multi-metric design that reflects the paper's recognition that no single automated metric is adequate for open-ended meeting understanding tasks.

Track A: Alignment Metrics (Precision, Recall, F1)

As described in the per-task sections above, alignment metrics measure structural correspondence between generated outputs and gold references using semantic embedding similarity. The key parameters:

• Embedding model: OpenAI text-embedding-3-small
• Similarity threshold: cosine similarity ≥ 0.75
• Alignment granularity: section-level (summarization) or question-level (QA pair extraction, question answering)
• Metric computation: precision, recall, and F1 using the formulas in Section 3 of the main paper and Appendix B

What these metrics are good for: Measuring whether the model's output has the right structure and covers the right content at a semantic level. They are fully automated, reproducible, and don't require API calls beyond the embedding computation.

What these metrics miss: Factual accuracy of the content within each aligned section or answer. A model could produce a section summary that semantically aligns to a gold section (high cosine similarity) but contains hallucinated or incorrect details — alignment metrics would not catch this. Similarly, a model could repeat a question perfectly (high F1) while providing a completely wrong answer — again, alignment metrics are blind to factual correctness.

Track B: LLM-as-Judge (GPT-4-Judge Scores)

To address the limitations of alignment metrics, the paper uses GPT-4 (gpt-4-turbo-2024-04-09) as a holistic quality judge. For each task, GPT-4 receives the meeting transcript, the model's output (summary, QA pairs, or question-answer pairs), and the gold reference, and produces scores (0–100) for each of five criteria:

1. Coverage: Does the output include all important information?
2. Redundancy: Does the output avoid unnecessary repetition?
3. Readability: Is the output well-structured and easy to understand?
4. Accuracy: Is the information factually correct based on the transcript?
5. Consistency: Is the output internally consistent without contradictions?

The prompt templates for GPT-4-Judge are shown in Figure 8 (summarization) and Figure 9 (QA pair extraction and question answering) of Appendix C. The prompts include explicit scoring descriptions for each criterion (e.g., what constitutes a score of 80 vs. 60 vs. 40 on Coverage) to standardize GPT-4's judgments.

The reported GPT-4-Judge score for each task is the average across the five criteria. The "Overall" GPT-4-Judge score in Table 5 is the micro-average across all three tasks (summarization, QA pair extraction, question answering).

Why GPT-4 as judge? The paper cites prior work (Wang et al., 2023; Zhang et al., 2023; Liu et al., 2024b) showing that GPT-4 evaluations correlate well with human judgments for open-ended NLG tasks. The alternative — manual human evaluation of all model outputs across 600 meetings × 7 models × 3 tasks — would be prohibitively expensive and slow. GPT-4-Judge provides a scalable proxy, with human evaluation used for validation rather than as the primary metric.

Bias Mitigation: Qwen-plus as Alternative Judge

A well-known concern with LLM-as-judge evaluation is self-bias: GPT-4 may systematically prefer outputs generated by GPT-family models over outputs from other model families, inflating the apparent performance of OpenAI models. The paper addresses this by running a parallel evaluation using Qwen-plus as an alternative judge, with identical prompt templates (Section 4.3, Table 6). The finding:

"The performance trends in Table 6 align closely with those in Table 5, where Qwen2.5-72B-Instruct remains the top-performing model, followed by GPT-4o and Qwen2-72B-Instruct, which show similar performance."

The consistency between GPT-4-Judge and Qwen-plus-Judge rankings suggests that self-bias is not substantially distorting the results. If GPT-4 were strongly biased toward GPT-4o, we would expect Qwen-plus-Judge to rank Qwen models higher and GPT-4o lower, but the rankings remain stable — Qwen2.5-72B-Instruct leads under both judges.

Human Validation: Fleiss' Kappa Agreement

To further validate the LLM-judge approach, the paper conducts a human evaluation on a random subset of 100 meetings. Five expert human annotators rate GPT-4o's outputs (summarization, QA pair extraction, question answering) on a 1–5 scale using the same five criteria. The process:

1. GPT-4o generates outputs for all 100 meetings across all three tasks.
2. Five annotators independently rate each output on a 1–5 scale per criterion.
3. The final human score for each output is the average across the five annotators.
4. GPT-4-Judge evaluates the same outputs on the standard 0–100 scale.
5. For comparison, GPT-4-Judge's scores are divided by 20 to map to a 1–5 scale.

Table 7 compares the average performance ratings from GPT-4-Judge and human annotators across the three tasks. The results show that GPT-4-Judge ratings broadly track human ratings, though specific numerical comparisons depend on the mapping function (dividing by 20 assumes linear correspondence between the 1–100 and 1–5 scales, which may not hold exactly).

Fleiss' Kappa computation (Table 8). The paper calculates two Kappa scores:

1. All six evaluators (five humans + GPT-4-Judge, with GPT-4 scores mapped to 1–5 scale).
2. Five human evaluators only (excluding GPT-4-Judge).

The finding: "the agreement among GPT-4-Judge and human annotators is still higher than that among humans." This is a striking result — it suggests that GPT-4-Judge is not only a reasonable proxy for human evaluation, but actually more consistent than individual human annotators when measured against the group. One interpretation: GPT-4-Judge applies a stable internal rubric, while human annotators exhibit more variance due to fatigue, differing interpretations of the criteria, or inconsistent application of standards across meetings. The higher Kappa with GPT-4 included means GPT-4 is not an outlier — its ratings fall within the human agreement range and actually increase overall consistency.

This is a strong validity argument for the benchmark's evaluation methodology. If GPT-4-Judge produced ratings that were systematically different from human annotators, the Kappa score including GPT-4 would drop below the human-only Kappa. The fact that it increases suggests GPT-4-Judge captures the shared understanding that humans have about output quality without introducing idiosyncratic biases that pull it away from the human consensus.

---

Model Configurations and Experimental Settings

The paper evaluates seven LLMs chosen to span a range of architectures, scales, and accessibility levels (Section 4.1).

API-based models (two):

• GPT-4o-2024-08-06 with 128K context window (OpenAI, 2023). Sampled at zero temperature (deterministic) for reproducibility.
• GPT-3.5-turbo-0125 with only a 16K context window. This is an intentionally weaker baseline with limited long-context capability — included to demonstrate how models fail when context windows are insufficient for meeting-length documents. Its 16K limit means it cannot process meetings in Sets 4 and 5, and the paper reports that "GPT-3.5-turbo fails for the Utilities sector due to input length exceeding its 16K token limit" (Section 4.2).

Open-source models (five):

• GLM4-9B-Chat with a massive 1000K (1M) context window (Zeng et al., 2022). Included to test whether extreme context length without extreme parameter count can handle these tasks.
• Llama3.1-8B-Instruct with 128K context (Dubey et al., 2024). The smallest model by parameters (8B), providing a lower-bound baseline for open-source models.
• Qwen2-7B-Instruct with 128K context (Yang et al., 2024a). Another small-model baseline from the Qwen family.
• Qwen2-72B-Instruct with 128K context (Yang et al., 2024a). A larger Qwen model, enabling within-family scaling comparison (7B vs. 72B).
• Qwen2.5-72B-Instruct with 128K context (Yang et al., 2024b). The latest Qwen generation, included to test whether newer training recipes improve financial meeting understanding.

All models support all three benchmark languages (English, Chinese, Japanese).

Zero-shot prompting. The paper evaluates all models in a zero-shot setting — no few-shot examples, no task-specific fine-tuning, no retrieval augmentation in the main evaluation. The prompt templates (Figures 5–7, Appendix C) provide task instructions and output format specifications, but no examples of correct outputs. This is a deliberate choice: it evaluates the models' inherent capability to understand the task from instructions alone, which is the most realistic deployment scenario (users typically don't provide examples when asking a model to summarize a meeting).

Generation configuration. The paper does not explicitly report generation parameters (temperature, top-p, max tokens) for most models. For GPT-4o, deterministic sampling is implied by the use case (evaluation should be reproducible). For open-source models, the paper uses eight NVIDIA A100/80G GPUs for inference, suggesting efficient but not exotic hardware requirements.

Cost. The paper reports: "The total cost for OpenAI API calls was about $2,500, while experiments with other LLMs use eight NVIDIA A100/80G GPUs." This transparency about cost is valuable for researchers planning to replicate or extend the evaluation.

---

RAG-Based Question Answering Variant (Section 4.2)

In addition to the main batched QA evaluation, the paper explores a retrieval-augmented generation (RAG) variant for question answering, comparing it against two non-RAG baselines (Section 4.2, Figure 3).

Experimental setup. The comparison uses Qwen2.5-72B-Instruct on a random subset of 50 meetings (10 from each of the five length sets). Five configurations are tested:

• Baseline 1: Answer all questions in a single batched prompt (the standard approach in the main evaluation).
• Baseline 2: Answer one question per API call (decomposing the batch into individual queries, which gives each question a dedicated context window but increases API calls).
• RAG (top 1): For each question, retrieve the single most relevant 1,024-token chunk from the document using OpenAI Embeddings, and answer based on only that chunk.
• RAG (top 3): Retrieve the top 3 most relevant chunks and provide them as context.
• RAG (top 5): Retrieve the top 5 most relevant chunks and provide them as context.

Chunking strategy. Following Wang et al. (2024), the document is divided into 1,024-token chunks. For each question, the question text is embedded using the OpenAI Embedding model, and cosine similarity is computed against all chunk embeddings. The top-k chunks with the highest similarity scores are concatenated and provided as context to the LLM, which then answers the question.

Key findings (Figure 3):

• For documents exceeding 15K tokens, Baseline 1 (batched QA with full context) outperforms all other variants, including RAG-based approaches. This suggests that for very long meetings, providing the full transcript to the model (and relying on its long-context attention to locate relevant information) is more effective than retrieval-based chunking, which inevitably loses cross-chunk context and discourse structure.
• For documents between 10K and 15K tokens, the observed performance ordering is Baseline 2 (one question at a time with full context) > RAG (top 5) > RAG (top 3) > RAG (top 1). Larger retrieved contexts help, but full-document context helps more — even when only answering one question at a time.
• For documents shorter than 10K tokens, RAG with top-5 retrieval surpasses the non-RAG variants. For short documents, the RAG approach benefits from focused context without losing much cross-document context, and the batched approach loses the advantage of having all information available simultaneously.
• RAG (top 1) performs worst across all document lengths, indicating that single-chunk retrieval is insufficient — relevant information for a financial question is typically spread across multiple non-contiguous segments of the meeting (the question itself, the immediate answer, follow-up clarifications, and contextual setup).

Why this result matters: It provides empirical evidence that current RAG chunking strategies are not yet a substitute for genuine long-context understanding when processing meeting-length financial documents. The fact that a model with 128K context window performs better when given the full transcript than when given retrieved chunks suggests that (a) the model's attention mechanism can effectively locate relevant information in very long contexts, and (b) the information needed to answer financial questions is distributed across the meeting in ways that simple similarity-based retrieval fails to capture. This aligns with the paper's broader claim that long-context understanding — not just retrieval — is essential for meeting comprehension.

---

Design Choices and Their Justifications: A Synthesis

The paper's technical design reflects a coherent set of priorities that distinguish M³FinMeeting from prior financial benchmarks:

1. Section-level annotation rather than document-level. By requiring annotators to segment meetings into topic-based sections and produce section-level summaries, the benchmark enables fine-grained evaluation of both topic segmentation AND content condensation. A document-level summary evaluation (the standard approach in summarization benchmarks) would conflate these capabilities and obscure the specific failure modes the paper identifies (low precision and recall, indicating models struggle with both boundary detection and coverage).

2. Embedding-based alignment over lexical overlap metrics. The choice of cosine similarity on OpenAI embeddings (threshold 0.75) rather than BLEU/ROUGE for the primary structural evaluation reflects the open-ended nature of the generation tasks. Meeting summaries can express the same content in many surface forms, and n-gram overlap metrics severely penalize valid paraphrases. Embedding similarity captures semantic equivalence more robustly, though at the cost of introducing dependence on the embedding model (text-embedding-3-small) and the similarity threshold (0.75). The paper includes BLEU/ROUGE in Appendix D as a supplementary metric, showing that lexical metrics produce different model rankings (Qwen2-72B-Instruct leads on summarization BLEU/ROUGE while Qwen2.5-72B-Instruct leads on GPT-4-Judge and alignment F1), confirming that semantic and lexical evaluation capture different aspects of quality.

3. Dual-track evaluation (alignment + LLM judge + human). No single metric adequately captures meeting understanding quality. Alignment metrics measure structural correspondence but miss factual accuracy. GPT-4-Judge measures holistic quality but introduces potential model bias. Human evaluation is the gold standard but is expensive and doesn't scale. By triangulating across all three — and showing that GPT-4-Judge agrees with human annotators (Fleiss' Kappa) and that an alternative judge (Qwen-plus) produces consistent rankings — the paper builds a credible case that the reported performance numbers reflect genuine model capability rather than metric artifacts.

4. QA pair extraction as the diagnostic differentiator. The task that most separates M³FinMeeting from prior work is QA pair extraction, and the paper's design choices maximize its diagnostic value: by reusing the same gold questions in the question answering task (where they're provided) and QA pair extraction task (where they must be discovered), the benchmark cleanly isolates question-discovery difficulty from answer-generation difficulty. The large performance gap between these tasks (F1 > 90% for QA vs. recall < 46% for QA pair extraction on the best model) demonstrates that the benchmark is measuring what it claims to measure, and that question detection — not answer synthesis — is the current bottleneck.

5. Stratified evaluation across three orthogonal dimensions. The decision to report per-language, per-sector, and per-length performance (Figures 2a–c, Appendix E Tables 10–12) rather than only aggregate scores transforms M³FinMeeting from a leaderboard benchmark into a diagnostic instrument. A model developer can see not just that their model scores 70 overall, but that it performs well on Japanese healthcare meetings but poorly on Chinese energy-sector meetings longer than 15K tokens — actionable information for targeted improvement.

6. Realistic task batching for question answering. The choice to batch all questions into a single prompt rather than evaluating one-at-a-time aligns the benchmark with real analyst workflows and reveals that for very long documents, providing full context in one pass outperforms retrieval-based or sequential approaches (Figure 3). This finding has practical implications for deployment: when a user has a list of questions about a long meeting, the best strategy is to provide the full transcript and all questions in a single prompt, not to chunk and retrieve.

7. Manual correction of ASR transcripts. By investing in human correction of Whisper outputs rather than using raw ASR text, the benchmark ensures that evaluation measures comprehension of meeting content rather than robustness to transcription errors. This is a deliberate scope limitation: the benchmark tests financial understanding, not ASR robustness. A system that performed well on raw ASR text but poorly on corrected text would be good at error correction but bad at financial reasoning — a capability profile that M³FinMeeting is explicitly designed not to reward.

4. Key Insights and Innovations

Innovation 1: The Benchmark Itself IS the Contribution — A Domain-Centric Evaluation that Reveals a Previously Invisible Bottleneck

Most NLP benchmark papers position themselves as measurement tools for tracking progress. M³FinMeeting does something more intellectually distinctive: by carefully designing tasks that mirror real financial analyst workflows, it discovers a capability bottleneck that no prior benchmark could have observed — namely, that question detection in long-form financial dialogue is the dominant failure mode for current LLMs, not answer generation.

What the field assumed before this work. The financial NLP evaluation landscape (catalogued in Table 1) operated under an implicit assumption: if you want to measure whether models understand financial text, you test them on answering financial questions. FinQA provides the questions and the structured tables. ConvFinQA provides the questions in a conversational format. FinanceBench provides the question and expects evidence retrieval. CFLUE provides multiple-choice options. In every case, the question is given. This design choice makes evaluation clean — you can measure answer accuracy with exact match or execution-based metrics — but it assumes away precisely the capability that turns out to be hardest: recognizing that a question has been asked in the first place.

What M³FinMeeting reveals. The QA pair extraction task (Section 3.2.2) flips the script: the model receives a 10,000–25,000-token meeting transcript and must autonomously extract all substantive financial question-answer pairs. This is not a small variant on existing QA — it's a fundamentally different cognitive demand. The model must scan thousands of tokens of multi-speaker dialogue, distinguish financially substantive questions from procedural ones ("Can you go to the next slide?"), rhetorical ones ("Isn't that obvious?"), and backchanneling ("Really?"), and then locate the corresponding answers which may appear minutes later in the conversation.

The performance gap between this task and the standard question answering task (where questions are provided) is the paper's most important empirical finding, even though it's not framed as a "contribution" in the traditional sense. Table 5 shows: for the best model (Qwen2.5-72B-Instruct), question answering F1 exceeds 90% across all languages — the model can faithfully reproduce provided questions and generate plausible answers. But QA pair extraction recall is only 45.65% — the same model misses more than half of the financially meaningful questions that human analysts identified in the same transcripts. This asymmetry — near-perfect answer generation vs. severely deficient question discovery — was invisible to the field because no prior benchmark tested for it.

Why this is a fundamental insight, not just a benchmark feature. The finding reframes what "financial language understanding" means. Prior benchmarks implicitly defined it as "can the model compute the right answer given a clear question and clean context." M³FinMeeting demonstrates that this definition misses the hardest part: in real financial workflows, the questions aren't pre-extracted and handed to you. They're buried in dialogue, often implied rather than explicitly marked, and spread across speakers and time. A model that scores 95% on FinQA could still be useless to an analyst who needs to know what questions were raised during an hour-long earnings call — because the model can answer questions but can't find them.

This insight has implications beyond the financial domain. Any real-world deployment of LLMs for meeting understanding — whether in healthcare, legal, corporate, or academic settings — faces the same question-discovery problem. The M³FinMeeting framework provides a template for diagnosing this capability in other domains: pair an open-ended extraction task with a matched closed-book task, and the performance delta isolates the discovery bottleneck.

Evidence anchor. Table 5, comparing QA pair extraction F1/Recall (all models < 46% recall) versus Question Answering F1 (most models > 90%) on the same underlying questions. The 45+ percentage point gap is the signal.

---

Innovation 2: Section-Level Structural Evaluation as a Diagnostic Instrument

Summarization benchmarks typically evaluate outputs holistically — compare the full generated summary against a reference summary using ROUGE, BLEU, BERTScore, or LLM-as-judge holistic ratings. These metrics answer "is this a good summary?" but not "where did the model succeed or fail?" M³FinMeeting introduces a section-level alignment evaluation (Section 3.2.1, Appendix B) that decomposes summarization quality into two separately measurable sub-capabilities: topic segmentation (did the model identify the right discussion boundaries?) and topic condensation (did it accurately summarize each identified topic?). This decomposition transforms summarization evaluation from a single scalar judgment into a diagnostic instrument.

How it works, conceptually (mechanism details in Section 3). Human annotators segment each meeting transcript into topic-based sections and produce a summary per section. The benchmark then evaluates model-generated summaries by checking how many generated sections semantically align to gold sections (precision) and how many gold sections are covered by at least one generated section (recall), using embedding-based cosine similarity with a 0.75 threshold. A model that generates one perfect summary covering all gold topics would score high recall but potentially low precision (too few sections, compressing multiple topics into one). A model that generates one summary per paragraph would score high precision (each section aligns to something) but low recall (the sections are too fine-grained to match the gold topic boundaries).

What the alignment metrics reveal that holistic scoring hides. Table 5 shows that precision, recall, and F1 "are all below 30%" for all models on the summarization task, even as GPT-4-Judge scores (the holistic quality metric) reach above 70 for the best models. This is a revealing discrepancy. It suggests that models are producing summaries that a judge deems well-written, non-redundant, and approximately accurate (high GPT-4-Judge scores) but that fail to align at the section level in terms of both topic boundaries and coverage (low alignment F1). In practical terms: a model might produce a fluent, readable summary that touches on most of the meeting's content, but it's organizing that content differently than a human analyst would — merging topics that should be separate, splitting topics that should be unified, or both.

Why this matters beyond the financial domain. The section-level alignment framework is domain-agnostic. Any summarization task where documents have natural topic boundaries — scientific papers (sections), legal documents (clauses), medical records (findings), customer service transcripts (issue resolution stages) — could adopt this evaluation approach to get a richer diagnostic signal than holistic scoring provides. The specific similarity threshold (0.75) and embedding model (text-embedding-3-small) are implementation choices, but the conceptual move — decompose summarization into segmentation + condensation and evaluate each separately — is a reusable contribution.

The paper also reports BLEU/ROUGE in Appendix D (Table 9), and the rankings differ from the alignment-based and GPT-4-Judge rankings: Qwen2-72B-Instruct leads on summarization BLEU/ROUGE while Qwen2.5-72B-Instruct leads on GPT-4-Judge and alignment metrics. This confirms what the summarization community has long suspected — lexical overlap metrics and semantic quality metrics measure different things — but M³FinMeeting provides a clean, task-specific demonstration that the gap is large and consequential when evaluating meeting-length financial summaries.

Evidence anchor. Table 5 summarization columns (P/R/F1 all < 30% vs. GPT-4-Judge scores > 70), Table 9 (different rankings under BLEU/ROUGE vs. alignment+judge metrics). The multi-metric divergence is the evidence that section-level evaluation captures something neither holistic judging nor lexical overlap captures alone.

---

Innovation 3: Long-Context Robustness Profiling Across Three Simultaneously-Varying Dimensions

Existing long-context benchmarks (LooGLE, LongBench, Loong) typically evaluate models by varying input length and measuring aggregate performance degradation — the standard finding is "performance drops as documents get longer." M³FinMeeting does something more nuanced: it profiles model robustness simultaneously along three orthogonal axes — input length (five bins from 0–5K to >20K tokens), language (English, Chinese, Japanese), and industry sector (all 11 GICS categories) — and reports results with sufficient granularity to reveal interaction effects between these dimensions. This moves the evaluation from a unidimensional "long-context capability score" to a multi-dimensional capability profile.

Specific interaction patterns the data reveals. Figure 2 and the detailed breakdowns in Appendix E (Tables 10–12) show that the relationship between length and performance is not uniform across tasks, models, or sectors:

• GPT-3.5-turbo (16K context window) provides a clean ablation: its performance "sharply drops" when input exceeds 15K tokens (Figure 2c), and it "fails for the Utilities sector due to input length exceeding its 16K token limit" — a direct demonstration that insufficient context window causes catastrophic rather than gradual degradation.
• Qwen2-72B-Instruct shows "a declining trend in the QA pair extraction task as input length increases" (Figure 2c), but maintains stable performance on summarization and question answering across the same length range. This is a task-specific length sensitivity — it's not that the model can't handle long contexts generally, but that longer contexts specifically impair its question-detection capability. This would be invisible in an aggregate score.
• Qwen2.5-72B-Instruct and GPT-4o demonstrate "stable and competent performance" across all length bins for summarization and QA, but still show QA pair extraction sensitivity. The newest-generation models handle length better, but question discovery still degrades.
• Cross-sector performance (Table 12, Figure 2b) shows that Communication Services, Consumer Discretionary, and IT sectors generally yield higher scores, while Energy, Utilities, and Real Estate are harder. The top models (GPT-4o, Qwen2-72B, Qwen2.5-72B) show much smaller sector-to-sector variance than weaker models — suggesting that stronger models have more domain-general financial understanding that transfers across industry terminology differences.

Why multi-dimensional profiling is conceptually important. Most long-context research treats "long-context capability" as a single latent variable that can be measured by one scalar metric. M³FinMeeting's data argues against this: the same model can be simultaneously good at long-context summarization and weak at long-context question detection; good at long-context processing in Japanese but weak in Chinese; robust to length in IT-sector meetings but brittle in Energy-sector meetings. These dissociations imply that "long-context capability" is not a unitary construct but a collection of sub-capabilities (cross-attention over distant mentions, discourse segmentation, domain-specific terminology tracking) that can develop independently. A benchmark that reports only an aggregate score obscures all of this.

The practical implication for model selection. A financial firm evaluating which LLM to deploy for meeting analysis should not ask "which model has the best long-context score?" but rather "which model performs well on our specific combination of languages, meeting lengths, and industry sectors?" M³FinMeeting's stratified reporting enables this kind of targeted model selection — a contribution to evaluation methodology that goes beyond ranking models.

Evidence anchor. Figure 2 (a, b, c) for the three-way stratification visualization; Table 10 for per-language breakdowns; Table 11 for per-length; Table 12 for per-sector. The task × length interaction for Qwen2-72B-Instruct (QA pair extraction declining with length while summarization holds stable) is the key dissociative evidence.

---

Innovation 4: Empirical Refutation of RAG as a Universal Long-Context Solution

The retrieval-augmented generation (RAG) paradigm has become the dominant practical approach for handling documents that exceed model context windows: chunk the document, embed the chunks, retrieve the most relevant ones for each query, and generate answers from the retrieved subset. The implicit assumption is that RAG can substitute for genuine long-context understanding — you don't need a model that can attend over 20,000 tokens if you can retrieve the right 2,000-token chunk. M³FinMeeting's RAG experiment (Section 4.2, Figure 3) provides direct counter-evidence to this assumption, at least for the specific domain of financial meeting QA.

What the experiment tests. On a random subset of 50 meetings (10 per length bin), Qwen2.5-72B-Instruct answers questions under five conditions: (1) batched full-context, (2) one-at-a-time full-context, (3) RAG with top-1 retrieved 1,024-token chunk, (4) RAG with top-3 chunks, (5) RAG with top-5 chunks. The key comparison is between condition 1 (the model sees the whole transcript and all questions at once) and condition 5 (the model sees only the 5 most embedding-similar chunks for each question).

The finding that challenges conventional wisdom. For documents exceeding 15K tokens, the batched full-context approach (Baseline 1) outperforms the best RAG variant (top-5 chunks). In other words, giving the model the entire 15,000+ token transcript is better than giving it 5 × 1,024 = 5,120 tokens of retrieval-selected content, despite the full context being 3×+ longer and containing substantially more distractor content. The model's attention mechanism, when given the full transcript, is better at locating relevant information than the embedding-based retriever is at selecting it.

Why this matters conceptually. RAG's core premise is that retrieval can approximate the information access that long-context attention provides, but with lower computational cost. M³FinMeeting's result suggests that this premise breaks down for meeting-length financial documents — at least with current retrieval quality. The likely explanation is that financial meeting questions require information distributed across non-contiguous sections: the question context, the immediate answer, a follow-up clarification three minutes later, and a numerical detail mentioned in passing during a different topic all contribute to a complete answer. Embedding similarity to the question text can retrieve the immediate answer passage but misses the distributed context that the model's attention can integrate when given the full transcript.

The finding is length-dependent, which strengthens it. For documents shorter than 10K tokens, RAG (top-5) does outperform the non-RAG variants. This makes sense: for short documents, the full-context advantage is smaller (less distributed context to integrate), and the retrieval advantage (focused, noise-free context) dominates. The crossover point where full-context becomes better — around 10K–15K tokens — provides a concrete threshold for practitioners deciding between RAG and full-context approaches for meeting-length documents.

Connection to the broader RAG debate. The NLP community is actively debating whether RAG can substitute for long-context models (exemplified by the "Lost in the Middle" phenomenon and recent work showing long-context models can match or exceed RAG performance on certain tasks). M³FinMeeting provides domain-specific, length-granular evidence on one side of this debate: for financial meeting QA on documents longer than ~15K tokens, full-context processing outperforms chunked retrieval. This is not a universal claim — the paper doesn't test other domains or retrieval methods — but it's a concrete, empirical counterexample to the assumption that RAG is always the right scaling strategy for long documents.

Evidence anchor. Figure 3, specifically the crossover where Baseline 1 (batched full-context) surpasses RAG (top-5) for documents >15K tokens. The progressive improvement from RAG (top-1) → RAG (top-3) → RAG (top-5) across most length bins also validates that retrieval quality matters, but the ceiling (full-context) is higher than the best retrieval can reach for long meetings.

---

Innovation 5: Multi-Judge Validation with Human Agreement Exceeding Inter-Human Agreement

LLM-as-judge evaluation (using GPT-4 to score model outputs) has become standard practice in the field, but it carries a well-known risk: the judge model may exhibit systematic biases — favoring its own model family's outputs, penalizing non-native-like phrasing from smaller models, or amplifying subtle stylistic preferences that don't reflect actual quality differences. Most benchmark papers that use LLM judges simply report the scores and note the limitation. M³FinMeeting goes significantly further in establishing the credibility of its LLM-judge scores through a validation methodology that produces a striking and counterintuitive result: GPT-4-Judge's agreement with human annotators is higher than the agreement among the human annotators themselves.

The validation architecture. The paper runs three parallel evaluations (Section 4.3, Section 4.4):

1. GPT-4-Judge (gpt-4-turbo-2024-04-09) scores all model outputs on a 0–100 scale across five criteria, producing the primary results in Table 5.
2. Qwen-plus-Judge scores the same outputs using identical prompt templates, producing the parallel results in Table 6. This tests for judge-model family bias: if GPT-4 systematically favors GPT-4o outputs, Qwen-plus should produce different rankings.
3. Five human annotators on the same team rate a random subset of 100 meetings' worth of GPT-4o outputs across all three tasks on a 1–5 scale, using the same five criteria. Fleiss' Kappa is computed both including and excluding GPT-4-Judge (with GPT-4 scores mapped to 1–5 by dividing by 20).

What the results show. Table 6 confirms that Qwen-plus-Judge produces rankings consistent with GPT-4-Judge: "Qwen2.5-72B-Instruct remains the top-performing model, followed by GPT-4o and Qwen2-72B-Instruct, which show similar performance." The correlation in rankings between the two judge models substantially weakens the self-bias concern — if GPT-4 were systematically overrating GPT-4o, Qwen-plus would rank Qwen models higher and GPT-4o lower. The fact that both judges agree on the ordering suggests the scores reflect genuine output quality differences rather than judge-specific preferences.

Table 8 reports the Fleiss' Kappa results: the agreement among all six evaluators (five humans + GPT-4-Judge) is higher than the agreement among the five human annotators alone. This is the paper's most methodologically important — and most counterintuitive — finding. It means that GPT-4-Judge is not merely an acceptable proxy for human evaluation; it is actually more aligned with the human consensus than individual human annotators are with each other.

Why this is a conceptual contribution, not just a methodological detail. This finding challenges a deeply held assumption in the NLP evaluation community: that human judgment is the gold standard that automated metrics should aspire to approximate, and that any gap between automated and human evaluation represents metric error. The M³FinMeeting Fleiss' Kappa result suggests a more nuanced picture: human annotators exhibit variance due to fatigue, inconsistent application of rubrics, differing interpretations of vague criteria, and attention lapses — all well-documented in the annotation quality literature. GPT-4-Judge, by contrast, applies a consistent internal rubric every time. When GPT-4's rubric aligns with the central tendency of human judgment (which the Kappa score confirms it does), it can actually produce more reliable aggregate quality estimates than a small number of individual human annotators.

The practical implication for future benchmark design. If this finding replicates across other domains and tasks, it would justify a shift in how the community approaches evaluation for open-ended generation: rather than treating LLM-as-judge as a cheap-but-imperfect substitute for human evaluation, treat it as a component of a multi-evaluator ensemble where the LLM judge contributes a stable, rubric-consistent perspective that may actually improve overall reliability when combined with human judgments. The M³FinMeeting validation framework — dual LLM judges + human subset + Fleiss' Kappa — provides a template for establishing this credibility on a per-benchmark basis rather than assuming it.

Caveat. The finding is based on 100 meetings and 5 annotators evaluating only GPT-4o outputs (not all seven models). The claim that GPT-4-Judge agrees with humans more than humans agree with each other is specific to this annotator pool, this task, and this evaluation rubric. Replication on broader model outputs and different annotator pools would be needed to establish generality. But within the scope of M³FinMeeting, it provides strong validation that the benchmark's primary reported metric (GPT-4-Judge scores in Table 5) is measuring something real and human-recognizable.

Evidence anchor. Tables 6–8, with particular weight on Table 8 (Fleiss' Kappa comparison) and the statement that "the agreement among GPT-4-Judge and human annotators is still higher than that among humans."

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the M³FinMeeting benchmark: 600 financial meeting transcripts (100 English, 400 Chinese, 100 Japanese) spanning all 11 GICS sectors. Each meeting has human-annotated reference data for three tasks — section-level summaries, extracted QA pairs, and gold-standard questions with answers. The corpus is split into five length bins for stratified analysis: Set1 (0–5K tokens, 59 meetings), Set2 (5–10K, 164), Set3 (10–15K, 195), Set4 (15–20K, 124), Set5 (>20K, 58). All transcripts are ASR-generated via Whisper and manually corrected; company identifiers and PII are removed via GPT-4 with human verification (Section 3.3.1).

• Base models. Seven long-context LLMs are evaluated, chosen to span architectures, scales, and accessibility tiers: GPT-4o-2024-08-06 (128K context, API-based, zero temperature), GPT-3.5-turbo-0125 (16K context, API-based, included as a limited-context baseline), GLM4-9B-Chat (1000K context, open-source), Llama3.1-8B-Instruct (128K, open-source), Qwen2-7B-Instruct (128K, open-source), Qwen2-72B-Instruct (128K, open-source), and Qwen2.5-72B-Instruct (128K, open-source). The paper states all models "support the languages in M³FinMeeting" (Section 4.1). The Qwen family provides a within-family scaling comparison (7B vs. 72B vs. 72B-v2.5); GPT-3.5-turbo's restricted 16K window serves as an ablation demonstrating what happens when context capacity is insufficient for meeting-length documents. The selection intentionally mixes API-based and open-source models to test whether evaluation results are robust across deployment paradigms.

• Metrics. The paper employs a dual-track scoring system. Track A computes alignment-based precision, recall, and F1 at the structural unit level (section summaries for summarization, individual questions for QA pair extraction and question answering). Generated and gold structural units are embedded using OpenAI's text-embedding-3-small model; a pair is considered aligned if cosine similarity exceeds 0.75. Precision = (aligned generated units) / (total generated units); Recall = (aligned gold units) / (total gold units); F1 is the harmonic mean. The paper notes that alignment is many-to-many — "one section summary from one side can be aligned to multiple section summaries on the other side" (Appendix B). Track B uses GPT-4-Judge (gpt-4-turbo-2024-04-09) to assign holistic quality scores (0–100) across five criteria: Coverage, Redundancy, Readability, Accuracy, and Consistency. The reported GPT-4-Judge score per task is the average across the five criteria; the overall score is the micro-average across all three tasks. The paper also reports BLEU and ROUGE (Appendix D, Table 9) as supplementary lexical metrics, and validates the GPT-4-Judge against both Qwen-plus-Judge (Section 4.3, Table 6) and human evaluation with Fleiss' Kappa (Section 4.4, Tables 7–8).

• Baselines. This paper is a benchmark evaluation — there is no "proposed method" competing against baselines. Rather, the seven LLMs are compared against each other and against the gold-standard human annotations (which represent the ceiling). Within the question answering task, the paper compares batched QA (all questions in one prompt, the primary approach) against sequential QA (one question per API call) and RAG-based QA (retrieve top-1, top-3, or top-5 1,024-token chunks via OpenAI Embeddings and answer from retrieved context only). The RAG comparison uses Qwen2.5-72B-Instruct on a random 50-meeting subset (10 per length bin). These comparisons establish whether full-context processing outperforms retrieval-based approaches for meeting-length financial documents.

• Generation budget / compute accounting. The paper does not frame experiments in terms of a "generation budget" (there are no multiple samples, beam search, or revisions being traded off). All evaluations use zero-shot, single-pass generation — each model receives a prompt containing the full meeting transcript and task instructions, and produces one output. The primary constraint is the model's context window (16K for GPT-3.5-turbo, 128K for most models, 1000K for GLM4-9B-Chat), which determines whether the full transcript fits without truncation. Compute cost is reported transparently: ~$2,500 in OpenAI API calls total, with open-source models run on eight NVIDIA A100/80G GPUs (Section 4.2, footnote 6). The RAG ablation adds an additional cost dimension: retrieval from pre-computed chunk embeddings, but this is not quantified in dollar or FLOP terms.

• Cross-validation / statistical protocol. The paper does not apply cross-validation for model selection (there are no hyperparameters being tuned or strategies being selected). Statistical reliability is addressed through: (1) the size of the test corpus — 600 meetings with manual annotations, providing reasonable per-language and per-sector sample sizes for the top models; (2) stratified reporting that disaggregates results by language, length, sector, and task, so that conclusions about model behavior are not based on aggregate scores alone; (3) human validation on a 100-meeting random subset (Section 4.4), establishing that the primary metric (GPT-4-Judge scores) correlates with human judgment; (4) dual judge models (GPT-4 and Qwen-plus) to check for systematic judge bias. However, no confidence intervals, standard errors, or significance tests are reported for any metric — all numbers are presented as point estimates. For the RAG ablation (Figure 3), the 50-meeting random subset is relatively small, and performance differences between RAG (top-5) and Baseline 1 at >15K tokens are not accompanied by any measure of whether the difference is statistically reliable. The Fleiss' Kappa analysis (Table 8) does provide a formal agreement metric, but only for the evaluation methodology's validity, not for model-vs-model comparisons.

---

Main Quantitative Results

Overall Task Performance (Table 5)

The headline finding: even the best-performing model on M³FinMeeting achieves GPT-4-Judge scores only in the low-to-mid 70s out of 100, with QA pair extraction recall failing to exceed 45.65%. The seven evaluated LLMs fall into three performance tiers:

• Tier 1 (overall GPT-4-Judge near or above 70): Qwen2.5-72B-Instruct (highest overall), GPT-4o, and Qwen2-72B-Instruct. Qwen2.5-72B-Instruct leads across all three tasks in GPT-4-Judge scores and achieves the highest QA pair extraction F1.
• Tier 2 (overall around 60): Qwen2-7B-Instruct and GLM4-9B-Chat.
• Tier 3 (overall lowest): LLaMA3.1-8B-Instruct outperforms GPT-3.5-turbo, but both trail substantially.

Summarization results (Table 5). The alignment metrics reveal a striking pattern: precision, recall, and F1 are all below 30% for every model — even the top-tier ones that score above 70 on GPT-4-Judge. Qwen2.5-72B-Instruct achieves only 5.51% F1 (English), 20.16% (Chinese), and 35.29% (Japanese) on section-level alignment, while its GPT-4-Judge score for the same task ranges from 72.74 (Chinese) to 82.40 (Japanese). This multi-metric divergence is the paper's clearest signal that models produce fluent, well-structured summaries (high judge scores) that nonetheless fail to align with human-annotated topic boundaries and coverage (low alignment F1). GPT-3.5-turbo's summarization GPT-4-Judge plummets to 0 for Set5 (>20K) meetings (Table 11), consistent with its 16K context window being exceeded.

QA pair extraction results (Table 5). This is the most challenging task. Best recall across all models: Qwen2.5-72B-Instruct achieves 45.65% (aggregated across languages in Table 5; per-language breakdowns in Table 10 show 28.81% English, 39.34% Chinese, 55.14% Japanese). The best F1: Qwen2.5-72B-Instruct at 39.34% (Chinese), 55.14% (Japanese). GPT-4-Judge scores for Qwen2.5-72B-Instruct range from 63.59 (Chinese) to 83.45 (Japanese). The gap between QA pair extraction performance and question answering performance (next paragraph) is the paper's core diagnostic: provided questions are answered reliably, but discovering questions autonomously is profoundly difficult.

Question answering results (Table 5). In stark contrast to QA pair extraction, most models achieve F1 scores above 90% on question repetition — they can faithfully reproduce provided questions in the structured output format. Qwen2.5-72B-Instruct achieves F1 of 97.33% (English), 92.35% (Chinese), 89.70% (Japanese). GPT-4-Judge scores are correspondingly high: 76.76 (English) to 86.95 (Japanese) for the best model. LLaMA3.1-8B-Instruct is the exception: it "has difficulty following instructions and often fails to repeat the questions, resulting in lower GPT-4-Judge scores" (Section 4.2, footnote 7), with F1 dropping to 37.21% in Chinese.

The paper also reports BLEU and ROUGE in Appendix D (Table 9), where the rankings shift: Qwen2-72B-Instruct achieves the best BLEU/ROUGE on summarization, while Qwen2.5-72B-Instruct leads on both QA pair extraction and question answering. This confirms that lexical overlap metrics and semantic/alignment metrics capture different aspects of quality — a finding consistent with the summarization community's known concerns about BLEU/ROUGE for abstractive tasks, but concretely demonstrated here for the financial meeting domain.

---

Effect of Language (Figure 2a, Table 10)

Performance varies systematically by language, but the direction is not uniform across models or tasks:

• Japanese consistently yields the highest scores. For Qwen2.5-72B-Instruct, overall GPT-4-Judge reaches 84.10 (Japanese) vs. 74.48 (English) vs. 69.49 (Chinese). The paper hypothesizes that "LLMs are more consistent in Japanese, likely because they adhere to the instructions more effectively in this language" — a speculative interpretation based on the observation that models follow the structured output format (question repetition, section separation) more reliably in Japanese. An alternative hypothesis (not discussed) is that the Japanese meeting transcripts may be inherently simpler or shorter in semantic content despite having higher token counts (Japanese averages 13,284 tokens/meeting vs. English's 10,086, reflecting the cl100k_base tokenizer's higher token-to-character ratio for Japanese text).

• English and Chinese performance are comparable but task-dependent. For summarization alignment F1, Chinese models outperform English (Qwen2.5-72B-Instruct: 20.16% Chinese vs. 5.51% English), possibly because the Chinese meetings (400 total, more training-like exposure) have more consistent meeting structures. For QA pair extraction recall, Japanese > Chinese > English across all models, suggesting question detection is linguistically easier in Japanese or that the Japanese meetings contain more explicitly marked question utterances.

• The three Qwen models show similar performance in summarization across languages, but diverge most in QA pair extraction (Section 4.2). Qwen2.5-72B-Instruct achieves GPT-4-Judge scores of 72.76 (EN), 63.59 (ZH), and 83.45 (JA) on QA pair extraction, while Qwen2-7B-Instruct reaches only 29.85, 40.47, and 31.82 — a gap of 40+ points in Japanese that indicates model scale and generation matter enormously for question discovery.

---

Effect of GICS Sector (Figure 2b, Table 12)

The paper reports per-sector performance for the three top-tier models (GPT-4o, Qwen2-72B-Instruct, Qwen2.5-72B-Instruct) in Table 12. Key patterns:

• Communication Services, Consumer Discretionary, and IT generally yield higher GPT-4-Judge scores across summarization and question answering. Qwen2.5-72B-Instruct achieves overall GPT-4-Judge of 73.10 (Com. Services), 73.10 (Con. Discretionary), 74.80 (IT).

• Energy, Real Estate, and Utilities produce the highest scores for the top models, counter to what one might expect from sector complexity. Qwen2.5-72B-Instruct achieves overall GPT-4-Judge of 80.30 (Energy), 79.10 (Real Estate), 78.78 (Utilities) — higher than the supposedly "easier" Consumer sectors. The paper does not discuss possible explanations, but two hypotheses: (1) these sectors have fewer meetings (Energy: 32, Real Estate: 13, Utilities: 8), so scores may be less reliable due to small sample sizes; (2) meetings in these sectors may be more structured (e.g., regulatory hearings, project updates) with clearer topic boundaries, making them easier for models to parse.

• The performance gaps among sectors are much smaller for the Tier 1 models than for GPT-3.5-turbo and LLaMA3.1-8B-Instruct (Section 4.2). Stronger models show more uniform cross-sector performance — they generalize better across industry-specific terminology and meeting structures — while weaker models are more sensitive to domain shift. GPT-3.5-turbo "fails for the Utilities sector due to input length exceeding its 16K token limit" (Section 4.2), a hard failure mode rather than a gradual degradation.

• QA pair extraction shows the most sector variability, with no clear best-performing sector — performance depends on model, language, and sector jointly in ways the aggregate Table 12 does not fully disaggregate.

---

Effect of Input Length (Figure 2c, Table 11)

The length-stratified results reveal sharp boundaries in model capability:

• GPT-3.5-turbo shows catastrophic failure above 15K tokens. Table 11 reports: for Set4 (15–20K), GPT-4-Judge scores drop to 19.00 (summarization), 14.36 (QA pair extraction), 11.16 (question answering), and overall 6.19. For Set5 (>20K), all scores are 0 — the 16K context window simply cannot accommodate the input, so the model produces nothing or truncated output. This is the canonical "cliff" behavior when context capacity is exceeded.

• Qwen2-72B-Instruct shows task-specific length sensitivity. Figure 2c and Table 11 reveal that its QA pair extraction performance declines steadily with increasing length (GPT-4-Judge: 56.48 at Set1 → 61.36 at Set2 → 61.37 at Set3 → 62.49 at Set4 → 59.15 at Set5 — not monotonic but generally flat-to-declining). Meanwhile, its summarization and question answering performance hold relatively stable across the same length range (summarization GPT-4-Judge: 70.29 at Set1 → 75.70 at Set5). The paper notes this indicates "a reduced capability to maintain performance with longer inputs in this specific task" (Section 4.2) — it's not a general long-context deficit, but a specific interaction between length and question detection.

• Qwen2.5-72B-Instruct and GPT-4o show stable performance across all lengths for summarization and question answering. Qwen2.5-72B-Instruct's summarization GPT-4-Judge ranges narrowly from 70.40 (Set1) to 76.79 (Set5); question answering ranges from 67.07 to 78.75. These models' 128K context windows are more than sufficient for even the longest meetings (max ~25K tokens), so no truncation occurs. However, QA pair extraction still shows length effects: Qwen2.5-72B-Instruct GPT-4-Judge drops from 60.93 (Set1) to 68.59 (Set5) — yes, it slightly improves with length, but the pattern is noisy and the per-length differences should be interpreted cautiously given the small bin sizes (58 meetings in Set5).

• General pattern: summarization and question answering degrade gracefully or not at all with increasing length for models with sufficient context windows, while QA pair extraction is the task most sensitive to document length.

---

RAG-Based Question Answering Comparison (Figure 3)

The RAG experiment on 50 meetings using Qwen2.5-72B-Instruct produces the following results (Figure 3):

• For documents >15K tokens: Baseline 1 (batched, full-context) outperforms all other variants, including Baseline 2 (sequential, full-context) and all RAG configurations (top-1, top-3, top-5). This is the central finding: providing the entire transcript in one prompt yields better QA performance than retrieving the 5 most similar 1,024-token chunks, despite RAG having access to up to 5,120 tokens of focused content vs. the full transcript's 15,000+ tokens.

• For documents 10–15K tokens: Baseline 2 > RAG (top-5) > RAG (top-3) > RAG (top-1). Larger retrieved contexts improve RAG performance, but full-context sequential answering (one question per call) still outperforms RAG. Baseline 1 (batched) is not separately compared in this range in the text, but Figure 3 suggests it underperforms Baseline 2 for this length bin.

• For documents <10K tokens: RAG (top-5) surpasses both non-RAG baselines. For short meetings, retrieval provides focused, noise-free context that helps more than having the full (but still relatively short) transcript available.

• RAG (top-1) performs worst universally, confirming that single-chunk retrieval is insufficient — answers to financial questions are distributed across multiple non-contiguous transcript segments.

The paper does not report the specific numeric GPT-4-Judge scores underlying these comparisons — Figure 3 is a bar chart without a companion table in the main text. The exact performance deltas between Baseline 1 and RAG (top-5) at >15K tokens are therefore not directly quotable from the paper's text. The finding is qualitatively clear from Figure 3, but readers seeking exact numbers must extract them visually from the plot.

---

GPT-4-Judge Validation: Qwen-plus-Judge Consistency (Section 4.3, Table 6)

Table 6 presents the full results using Qwen-plus as the alternative judge, with identical prompt templates (Figures 8–9). The key finding: rankings and trends are substantially preserved. Qwen2.5-72B-Instruct remains top overall; GPT-4o and Qwen2-72B-Instruct show similar performance; Qwen2-7B-Instruct and GLM4-9B-Chat form the middle tier; GPT-3.5-turbo and LLaMA3.1-8B-Instruct trail. The paper states that "the performance trends in Table 6 align closely with those in Table 5" and that this "further demonstrates that Qwen-Plus-Judge is a reliable alternative evaluator" (Section 4.3).

The alignment of rankings between two different judge models (OpenAI's GPT-4 and Alibaba's Qwen-plus) provides evidence against the concern that GPT-4-Judge systematically favors GPT-family outputs. If such self-bias were present, Qwen-plus would likely rank Qwen models higher relative to GPT-4o. The fact that both judges produce the same ordering — with the open-source Qwen2.5-72B-Instruct leading GPT-4o under both judges — is strong evidence that the scores reflect genuine output quality rather than judge-model family effects.

Specific numeric scores are available in Table 6 but not reproduced in the main text excerpt.

---

Human Evaluation and Fleiss' Kappa (Section 4.4, Tables 7–8)

The human validation on 100 randomly selected meetings yields the paper's most methodologically significant finding:

• Table 7 compares GPT-4-Judge ratings (mapped from 0–100 to 1–5 by dividing by 20) against average human annotator ratings for GPT-4o's outputs across the three tasks. The paper states that the ratings align, but specific numeric comparisons require consulting Table 7 directly (not fully reproduced in the text excerpt).

• Table 8 reports Fleiss' Kappa scores. The paper computes two values: (1) agreement among all six evaluators (five humans + GPT-4-Judge) and (2) agreement among the five human annotators only. The result: the Kappa score including GPT-4-Judge is higher than the human-only Kappa. This means GPT-4-Judge's ratings are not an outlier relative to human judgments — they fall within the range of human inter-annotator agreement and actually increase overall consistency when included.

The paper's interpretation is noteworthy: GPT-4-Judge applies a consistent internal rubric, while human annotators exhibit variance from fatigue, differing interpretations, or inconsistent standards. The finding that GPT-4-Judge increases rather than decreases overall agreement suggests it captures the shared signal in human quality judgments without introducing idiosyncratic biases.

Caveat on scope: The human evaluation covers only GPT-4o outputs (not all seven models) and is limited to 100 meetings (one-sixth of the full dataset). The claim that GPT-4-Judge is more consistent than individual humans is therefore demonstrated for this specific model's outputs, these specific annotators, and this specific evaluation rubric. Generalization to other model outputs or annotator pools would require additional validation.

---

Ablation Studies and Robustness Checks

BLEU/ROUGE metrics vs. alignment + judge metrics (Appendix D, Table 9). The paper reports BLEU-4 and ROUGE-1/2/L in addition to the primary alignment and GPT-4-Judge metrics. Results show that different metrics produce different model rankings: Qwen2-72B-Instruct achieves the best BLEU/ROUGE on summarization, while Qwen2.5-72B-Instruct leads on both QA pair extraction and question answering under BLEU/ROUGE, and also leads across all tasks under GPT-4-Judge. This divergence confirms that lexical overlap and semantic quality are measuring different constructs — the benchmark does not rely on any single metric for its conclusions.

Qwen-plus-Judge vs. GPT-4-Judge (Section 4.3, Table 6). As discussed above, the alternative judge produces consistent rankings with GPT-4-Judge, providing evidence against systematic judge-model family bias. This is effectively an ablation of the judge model choice — the results are robust to which specific LLM serves as evaluator.

Human evaluation with Fleiss' Kappa (Section 4.4, Tables 7–8). In addition to validating GPT-4-Judge, the human evaluation serves as a robustness check on the entire evaluation methodology: it confirms that the five quality criteria (Coverage, Redundancy, Readability, Accuracy, Consistency) capture human-recognizable quality dimensions, and that the scores assigned by automated judges correspond to what trained financial analysts would assign.

Input length ablation via GPT-3.5-turbo's 16K limit. GPT-3.5-turbo is included as a deliberate "ablation" on context window capacity. Its catastrophic failure on Set4 and Set5 (zero scores, inability to process Utilities sector meetings) demonstrates what happens when context window is insufficient — the degradation is sharp and total, not gradual. This validates that the benchmark's length range (meetings up to 25K+ tokens) genuinely stresses context capacity for models with limited windows, and that the strong performance of 128K-context models on the same long meetings reflects genuine long-context capability rather than an easy task.

RAG retrieval depth ablation (Figure 3). The comparison of RAG (top-1), RAG (top-3), and RAG (top-5) against full-context baselines shows that retrieval depth matters — more chunks improve RAG performance — but that even top-5 retrieval underperforms full-context processing for documents >15K tokens. This is effectively an ablation on how much context is provided for QA: from 1,024 tokens (top-1) to 5,120 tokens (top-5) to the full document (10K–25K tokens). The non-monotonic finding (full-context outperforms all chunked variants only above a length threshold) reduces concern that the full-context advantage is simply an artifact of model architecture rather than a genuine benefit of access to distributed information across long transcripts.

Question batching vs. sequential answering (Figure 3). For documents >15K tokens, Baseline 1 (all questions in one prompt) outperforms Baseline 2 (one question per prompt). This is effectively an ablation on task batching strategy, and the result supports the paper's choice of batched QA as the primary approach for long meetings: providing all questions together allows the model to answer them in the context of the full transcript, rather than re-processing the same long document for each question.

Cost transparency. The paper reports that "the total cost for OpenAI API calls was about $2,500, while experiments with other LLMs use eight NVIDIA A100/80G GPUs" (Section 4.2, footnote 6). While not a traditional ablation, this transparency allows readers to assess the practical feasibility of reproducing or extending the evaluation.

---

Critical Assessment

Claim 1: "Even the most advanced LLMs currently available struggle with the tasks in M³FinMeeting, revealing substantial room for performance improvement."

Assessment: Well-supported, but the "room for improvement" is task-specific. The evidence is unambiguous for QA pair extraction: Qwen2.5-72B-Instruct's recall of 45.65% means it misses more than half of gold-standard questions, and no model surpasses this threshold. This is a clear performance ceiling with substantial headroom.

For summarization, the picture is more nuanced. The alignment metrics (P/R/F1 all <30%) suggest severe difficulty with topic segmentation and coverage. However, the GPT-4-Judge scores exceed 70 for the best models — indicating that models produce summaries a judge deems high-quality despite poor structural alignment. Whether this means the models are "struggling" depends on what one cares about: if a user wants a readable, approximately accurate summary, current models perform reasonably well; if they need a summary organized identically to how a human analyst would structure it (same topic boundaries, same coverage), models fail badly. The benchmark's value is in revealing this distinction, but the claim of "struggle" applies more to the structural alignment dimension than to holistic quality.

For question answering, models demonstrably do not struggle when questions are provided — F1 >90% for most models on question repetition, and GPT-4-Judge scores in the 70s–80s. The "struggle" is confined to the autonomous question-discovery aspect of meeting understanding, not the answering aspect.

The paper would be strengthened by quantifying the ceiling more precisely — what would a perfect score look like? Human performance on the same tasks is not reported (the human evaluation in Section 4.4 rates model outputs for quality, not humans performing the tasks themselves). Without knowing whether trained financial analysts achieve 60% or 95% recall on QA pair extraction, it's hard to interpret how far current models are from expert-level performance. The "room for improvement" could be modest (if even humans max out at 60–70% due to task ambiguity) or enormous (if humans achieve near-perfect extraction).

---

Claim 2: "Qwen2.5-72B-Instruct significantly outperforms other LLMs" on M³FinMeeting.

Assessment: Supported under GPT-4-Judge and Qwen-plus-Judge, but not uniformly across all metrics. Qwen2.5-72B-Instruct leads on GPT-4-Judge overall scores (Table 5), on Qwen-plus-Judge overall scores (Table 6), and on BLEU/ROUGE for QA pair extraction and question answering (Table 9). However, Qwen2-72B-Instruct achieves better BLEU/ROUGE on summarization (Table 9), and the alignment F1 comparisons show task-by-language variation — Qwen2.5-72B-Instruct is not uniformly best on precision/recall/F1 in every language-task combination (Table 10). The "significantly" qualifier lacks a statistical test — no confidence intervals or significance values are reported, so we cannot assess whether Qwen2.5-72B-Instruct's margin over GPT-4o or Qwen2-72B-Instruct is reliable or could be explained by variance across the 600-meeting test set.

A missing analysis: The paper doesn't explore why Qwen2.5-72B-Instruct outperforms GPT-4o on this benchmark. Is it because Qwen models are trained on more multilingual financial data? Because Qwen2.5 has architectural advantages for long-context dialogue? Because GPT-4o's training data underrepresents meeting-transcript-style text? Without this analysis, the ranking is a leaderboard entry rather than a diagnostic finding.

---

Claim 3: The benchmark "fills significant gaps in existing benchmarks, which often rely on static sources like news articles and earnings reports."

Assessment: Strongly supported by Table 1 and the task design. The paper's literature review (Table 1) comprehensively documents that no prior financial benchmark meets all three criteria: (1) meeting transcript source material, (2) multilingual support across EN/ZH/JA, and (3) tasks requiring both question discovery and answer generation from long-form dialogue. The RAG experiment (Figure 3) further validates that meeting-length financial documents pose challenges distinct from shorter, structured financial texts — the finding that full-context processing outperforms retrieval at >15K tokens would not emerge from a benchmark using news articles or earnings reports where the relevant information is more localized.

A limitation on "fills": The benchmark fills the gap for evaluation, but the paper does not release training data for these tasks. The meetings are annotated for testing only. This is appropriate for a benchmark paper, but means the gap in training resources for financial meeting understanding remains unfilled. Models evaluated on M³FinMeeting are all pretrained or instruction-tuned on other corpora; there is no evidence about how much fine-tuning on meeting transcripts would improve performance.

---

Claim 4: The evaluation methodology (GPT-4-Judge + alignment metrics + human validation) provides reliable quality assessment.

Assessment: The most thoroughly validated aspect of the paper. The triple validation — dual LLM judges (Tables 5–6), human evaluation on subset with Fleiss' Kappa (Tables 7–8), and cross-metric comparison showing BLEU/ROUGE diverges from GPT-4-Judge (Table 9) — provides strong evidence that the primary reported scores measure something real and human-recognizable. The Fleiss' Kappa result (GPT-4-Judge increases inter-annotator agreement) is particularly strong — it suggests the automated judge is not merely an approximation of human judgment but may actually provide more stable evaluations than small groups of human annotators.

Caveats on the human validation scope:

• Only GPT-4o outputs are evaluated (100 meetings), not all seven models. If GPT-4-Judge favors GPT-4o's outputs specifically (self-bias), human evaluation of only GPT-4o outputs would not detect it — the Fleiss' Kappa would remain high because both GPT-4-Judge and humans are evaluating the same (potentially favored) outputs. The Qwen-plus-Judge comparison partially mitigates this by showing rank-order preservation across all models, but a full human evaluation of outputs from multiple model families would be stronger.
• The mapping from 1–100 to 1–5 (dividing by 20) assumes linear correspondence between the two scales, which may not hold — GPT-4-Judge might use the 0–100 range differently than humans use 1–5, and the division-by-20 mapping could distort agreement.
• The five human annotators appear to be from the same annotator pool (trained financial analysts working on the project) rather than independent external evaluators. Their agreement patterns may reflect shared training rather than independent judgment, which could inflate the human-only Kappa and understate the relative benefit of adding GPT-4-Judge.

---

Claim 5: The RAG experiment demonstrates that full-context processing is superior to chunked retrieval for long financial meetings.

Assessment: Supported but with important scope limitations. Figure 3 shows that Baseline 1 (batched full-context) exceeds RAG (top-5) for documents >15K tokens, using Qwen2.5-72B-Instruct and OpenAI Embeddings for retrieval. This finding is:

• Specific to this model: A model with weaker long-context attention might show a different crossover point.
• Specific to this retrieval method: Alternative chunking strategies (different chunk sizes, overlap, hybrid retrieval, re-ranking) could improve RAG performance. The paper uses fixed 1,024-token chunks and cosine similarity on OpenAI Embeddings — a reasonable but basic RAG pipeline.
• Specific to this task: QA on meeting transcripts, where answers are distributed across non-contiguous sections. For tasks where answers are more localized (e.g., factoid QA on news articles), RAG might perform differently.
• Evaluated on a small subset: Only 50 meetings (10 per length bin), with no statistical test on the performance differences.

The finding is a valid counterexample to the assumption that RAG is universally preferable for long documents, but it does not establish that "full-context beats RAG for financial meetings" as a general claim — it establishes that it does for this model, this task, and this retrieval pipeline.

---

Missing Experiments That Would Strengthen the Paper

• Human performance on the tasks themselves. What accuracy do trained financial analysts achieve on QA pair extraction and summarization? Without this ceiling, the "room for improvement" is unquantified. If humans achieve 60% recall on QA pair extraction, then current models at 45% are within striking distance. If humans achieve 95%, the gap is enormous and qualitatively different.

• Fine-tuned models. All evaluations are zero-shot. The paper could include at least one fine-tuned baseline — e.g., a model fine-tuned on ECTSum earnings call summaries and tested on M³FinMeeting summarization — to establish how much domain-adaptive training helps. This would address the question: is the performance gap due to insufficient model capability, or insufficient domain-specific training data?

• Cross-model human evaluation. The human validation covers only GPT-4o outputs. Extending it to include outputs from at least one other model family (e.g., Qwen2.5-72B-Instruct) would allow testing whether GPT-4-Judge's agreement with humans holds across model families or whether it degrades for non-OpenAI outputs.

• Statistical significance. No confidence intervals, standard deviations, or pairwise significance tests are reported for any model comparison. For a 600-meeting test set, the standard error on aggregate GPT-4-Judge scores would be computable (e.g., via bootstrap) and would clarify whether the Tier 1 vs. Tier 2 distinctions are reliable.

• Ablation on the embedding similarity threshold (0.75). The alignment metrics depend critically on this threshold. An ablation showing how P/R/F1 change with thresholds of 0.65, 0.75, and 0.85 would clarify whether the "all models below 30%" finding is robust or sensitive to threshold choice. If lowering the threshold to 0.65 substantially increases F1, the models might be producing section summaries that are semantically related to gold sections but with different granularity — a segmentation problem rather than a comprehension problem.

• Spearman/Pearson correlation between metrics. The paper shows that BLEU/ROUGE, alignment F1, and GPT-4-Judge produce different model rankings, but doesn't quantify how strongly they correlate across individual meetings. A correlation matrix would reveal whether the metrics measure related or independent quality dimensions, which would help users interpret the multi-metric reporting.

• Analysis of why Japanese meetings yield higher scores. The paper notes this pattern but only speculates vaguely about "adhering to instructions more effectively." A qualitative error analysis on a sample of English vs. Japanese transcripts — e.g., examining whether Japanese meeting discourse contains more explicit question markers or clearer topic transitions — would convert this observation into an actionable finding about cross-lingual meeting understanding.

• Speaker-attributed evaluation. Meeting transcripts contain speaker labels, but the evaluation does not test whether models can attribute statements to the correct speaker. A sub-analysis checking whether QA pair extraction errors are concentrated in multi-speaker exchanges (vs. single-speaker monologues) would reveal whether speaker tracking is a specific failure mode.

---

Genuine Weaknesses

1. Small test set for some stratified cells. The per-sector and per-length analyses split 600 meetings into small bins: Utilities has only 8 meetings, Real Estate has 13, Set5 (>20K) has 58. Performance estimates for these bins have high variance, and the paper reports point estimates without uncertainty. Apparent sector differences (e.g., Energy outperforming Consumer Discretionary) could be noise from small samples.

2. No inter-annotator agreement reported for ground-truth creation. The paper emphasizes annotation quality control heavily (training, guidelines, reviewer feedback) but provides no quantitative agreement metrics (Cohen's Kappa, Krippendorff's alpha) for the summarization segmentation or QA extraction tasks. For a benchmark that rests on human-created references, this is a notable gap — we cannot assess how much of the "model error" is actually intrinsic task ambiguity that human annotators would also disagree on.

3. Single chunking and retrieval strategy for RAG. The RAG comparison uses one chunk size (1,024 tokens), one embedding model (OpenAI text-embedding-3-small), and one retrieval method (cosine similarity top-k). The finding that RAG underperforms full-context processing is specific to this configuration — it may not hold with optimized chunking (smaller chunks with overlap, semantic chunking, or section-aware splitting), better retrieval models, or hybrid retrieval + re-ranking. The paper presents the RAG comparison as a diagnostic, not a comprehensive RAG ablation, but readers might overgeneralize the result.

4. No analysis of transcript quality effects. The transcripts are Whisper-generated with manual correction, but different meetings likely have different residual error rates (background noise, accented speech, overlapping speakers). If some meetings have lower-quality transcripts than others, performance differences across sectors or languages could partially reflect transcription quality rather than meeting complexity. The paper does not analyze or control for this.

5. The GPT-4-Judge may be evaluating surface features rather than factual accuracy. The five criteria include Accuracy, but GPT-4-Judge does not have access to external knowledge beyond the provided transcript. If the transcript contains an error or the model hallucinates a plausible-sounding financial claim, GPT-4-Judge might not detect it — it can check whether the summary is consistent with the transcript, but not whether the transcript itself is factually correct or whether the summary adds unsupported claims not present in the transcript. The human validation (Section 4.4) partially addresses this (human annotators can also be fooled by plausible-sounding hallucinations), but the paper doesn't analyze hallucination rates directly.

6. No comparison to smaller or non-LLM baselines. The paper evaluates only LLMs with context windows of 16K+. Including a strong extractive baseline (e.g., LexRank for summarization, or a fine-tuned BERT-based QA model with sliding window for question answering) would establish what portion of the performance is due to LLM-specific capabilities vs. simpler approaches. Without this, we can't tell whether M³FinMeeting tasks require deep financial reasoning or could be solved adequately by non-LLM methods.

7. The computational cost analysis is incomplete. The paper reports OpenAI API costs (~$2,500) and GPU hardware for open-source models, but doesn't report total FLOPs, inference time, or per-meeting latency. For a benchmark designed to evaluate practical deployability, knowing that Qwen2.5-72B-Instruct achieves the best scores is only half the picture — how long does it take to process a 60-minute meeting, and at what cost per meeting?

6. Limitations and Trade-offs

The Difficulty Estimation Cost Dominates the Reported Efficiency Gains

The assumption or constraint. The entire compute-optimal allocation framework rests on knowing each prompt's difficulty before deciding how to spend the inference budget. The paper's method for estimating difficulty — generating 2,048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive relative to the budgets being optimized. The acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline ~4× efficiency gain over best-of-N is computed after difficulty is already known, without amortizing the cost of learning it. Generating 2,048 samples to estimate difficulty costs more than the largest test-time budgets studied (256–512 generations). In a realistic deployment, total cost equals difficulty estimation cost plus strategy execution cost, and the former could dominate the latter — potentially eliminating or even reversing the claimed efficiency advantage. Without a cheap difficulty estimator, the compute-optimal policy is a theoretical upper bound, not a practical deployment strategy.

What evidence exists in the paper. The paper explicitly flags this cost (Section 3.2) and shows that predicted (PRM-based, non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4, 8), but neither variant accounts for the cost of generating the 2,048 samples needed to produce those bin assignments. The cost is never included in any budget calculation or efficiency comparison.

Mitigation status. Not addressed. The paper acknowledges the issue and suggests "training models to directly predict difficulty of a question" (Section 8) as future work, but no such model is developed or evaluated. The reported efficiency figures should be understood as upper bounds conditional on free difficulty estimation — they are not realized deployment gains.

---

Hard Problems Remain Fundamentally Unsolved Across All Methods

The assumption or constraint. All test-time compute methods studied in this paper — PRM-guided search, iterative revisions, and their compute-optimal combinations — operate by amplifying existing capabilities of the base model. They cannot create capabilities that are absent. This is not a hidden assumption; it is a structural property: if the base model's pass@1 on a problem class is near zero, no amount of search or revision can help, because there are simply no correct solutions in the proposal distribution to find or refine.

The consequence. The paper's approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For the hardest questions (difficulty bin 5), accuracy is near zero across all methods and all budgets — the scaling curves are essentially flatlines (Figure 3 right, Figure 7 right). The FLOPs-matched comparison (Section 7, Figure 9) shows that for these hard problems, pretraining a larger model is almost always the better investment, and test-time compute can actually produce negative relative performance compared to the ~14× larger model baseline (down to −52.9% for PRM search on hard questions at R ≫ 1). For deployments where the problem distribution skews toward genuinely hard problems outside the base model's reach, compute-optimal test-time scaling provides essentially zero benefit regardless of budget.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all search methods. Figure 7 (right) shows bin 5 accuracy at roughly 2–3% irrespective of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line flat near 0–5% across all conditions, below the ~14× larger model's greedy performance (stars) at all R values. The paper acknowledges this transparently in the Section 7 takeaway box: "test-time compute can amplify existing capability but cannot create it."

Mitigation status. The paper does not attempt to solve this — it is a fundamental boundary condition, not a bug. The authors correctly characterize when test-time compute is preferable (easy-to-medium problems, low R regimes) and when pretraining is necessary (hard problems), which is itself a valuable contribution. However, practitioners whose problem distribution includes substantial hard examples should understand that M³FinMeeting-style adaptive allocation provides no help on those cases.

---

The Revision Model Has a Systematic Correct-to-Incorrect Reversion Problem

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). During training, the model never sees a situation where the current answer in context is already correct. At inference time, however, the revision chain may produce correct intermediate answers, and the model — having never been trained on what to do in this situation — will often incorrectly "revise" a correct answer into a wrong one.

The consequence. The paper reports that approximately 38% of correct answers produced during a revision chain get converted back to incorrect answers in the subsequent revision step (Section 6.1). This means the revision model is actively harming its own output on a substantial fraction of problems: even when it stumbles onto the right answer, it has a 38% chance of undoing that progress. The paper's mitigation — majority voting or verifier-based selection across the entire chain of revisions rather than always taking the last revision — is a patch, not a fix. It requires generating longer chains (more compute) and relies on the selection mechanism to identify which revision in the chain is correct. In practice, this means the revision model wastes a significant fraction of its generation budget producing counterproductive revisions that actively degrade answer quality.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The ReST^EM experiment (Appendix K, Figure 16) provides additional evidence that revision training is fragile: an attempt to further optimize the revision model using RL-style training caused performance to degrade substantially with sequential revisions (fully sequential dropping to ~33.5% vs. ~38.5% at the optimal ratio), suggesting the revision approach is sensitive to training methodology in ways that amplify the reversion problem.

Mitigation status. Partially addressed via majority voting or verifier-based selection across the chain, but no principled solution (such as training the model to recognize when no revision is needed, or including "already correct" trajectories in the training data) is developed. The paper acknowledges the issue but treats the within-chain selection workaround as sufficient for the reported results — in practice, the workaround adds computational overhead (more revisions must be generated and all must be scored) and still fails when the selection mechanism chooses the wrong revision.

---

Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. The PRM is trained via Monte Carlo rollouts from the base model (Section 5.1) and is imperfect — it assigns high scores to some incorrect solutions and low scores to some correct ones. When search algorithms aggressively optimize against the PRM's scores, they find solutions that score highly under the verifier but are actually incorrect — a standard reward hacking / Goodhart's law phenomenon.

The consequence. Over-optimization prevents unbounded improvements from additional test-time compute and can even cause performance degradation at high budgets. Beam search degrades easy-problem performance at high budgets (Figure 3, right, bin 1 accuracy decreasing from ~78% to ~77% as budget increases). Lookahead search — the most aggressive optimizer — paradoxically performs worst overall (Figure 3, left), because its more accurate scoring of individual steps makes it better at exploiting verifier blind spots. The compute-optimal policy partially mitigates this by routing easy problems away from aggressive search (using best-of-N instead), but it does not solve the underlying verifier quality problem. The compute-optimal approach is fundamentally bounded by how well the PRM can distinguish correct from incorrect solutions, and further scaling of test-time compute will require better verifiers, not better search algorithms or allocation policies.

What evidence exists in the paper. Figure 3 (right) shows beam search degrading on bin 1 at high budgets. Figure 3 (left) shows lookahead search underperforming all other methods despite being the most powerful optimizer. Appendix M (Figure 29 and qualitative examples) shows search producing degenerate outputs — repetitive low-information steps, overly short 1–2 step solutions — that score highly under the PRM but are clearly wrong. The PRM vs. ORM comparison (Appendix F, Figure 14) shows that even the better-trained PRM still has substantial room for improvement (ORMs score lower, but both are far from perfect).

Mitigation status. The compute-optimal policy mitigates over-optimization by avoiding aggressive search where the verifier is most vulnerable (easy problems), but this is a routing strategy, not a solution. The paper does not explore improvements to verifier robustness — adversarial training, ensemble methods, KL-constrained search — that would raise the over-optimization ceiling. The finding that verifier quality is the primary bottleneck (more so than search algorithm sophistication) is itself a useful insight, but the limitation remains: the entire test-time compute scaling framework is capped by verifier reliability, and the current training approach (Monte Carlo rollouts with soft labels) produces verifiers that are good enough for moderate compute budgets but not for the aggressive optimization that would be needed to extract more value from larger budgets.

---

Single Benchmark, Single Model Family Limits Generality

The assumption or constraint. All experiments are conducted on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model (Section 4). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs," but provide no evidence that the findings generalize to other benchmarks, other model families, or other task types.

The consequence. Several aspects of the paper's findings could be model-specific or benchmark-specific. The PRM's over-optimization behavior depends on PaLM 2-S*'s specific output distribution and error patterns — a model with different calibration properties might exhibit qualitatively different difficulty-dependent scaling curves (e.g., beam search might help on easy problems rather than hurt them). The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of high-school competition math problems requiring symbolic reasoning and computation — it is unclear whether the central findings (beam search degrades easy problems; revisions help easy problems; medium problems benefit most from search; hard problems are unsolved) generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than deductive reasoning. The finding that test-time compute can substitute for pretraining at up to ~14× parameter scale may not hold for model families with different scaling properties or for tasks with different difficulty distributions.

What evidence exists in the paper. None — there is no multi-benchmark evaluation and no multi-model-family evaluation. The paper evaluates only PaLM 2-S* (with the 14× larger variant used only for the FLOPs-matched comparison, also within the PaLM 2 family). The authors acknowledge this as a limitation implicitly through their "believe this model is representative" framing, but do not test it.

Mitigation status. Not addressed. The authors do not claim universality and are transparent about the scope (Section 4: "all experiments use PaLM 2-S*"), but a practitioner deciding whether to adopt compute-optimal test-time scaling for a different model family or task domain has no empirical basis for confidence. The difficulty-dependent patterns (easy → best-of-N, medium → beam search, hard → no method helps) are presented as general principles rather than PaLM-2-MATH-specific observations, which may or may not hold under replication. This is a gap that future work must fill — the paper provides a framework and methodology for studying test-time compute allocation, but the specific policy recommendations (which strategy to use at which difficulty level) may not transfer without re-derivation on the target model and task.

---

Sequential Revisions Impose Latency That Parallel Methods Avoid

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial — each revision depends on the previous one and cannot be parallelized — while parallel best-of-N can be executed simultaneously given sufficient hardware. The compute-optimal policy often favors sequential-heavy strategies on easy problems (Figures 7, 8), where the model's initial attempts are roughly correct and just need refinement.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer in wall-clock time than one that runs 128 parallel samples simultaneously. For latency-sensitive applications — interactive assistants, real-time decision support, user-facing chatbots — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. The FLOPs-matched comparison (Section 7) compares total computation but does not account for the fact that sequential computation cannot be parallelized across hardware, making the test-time compute approach potentially much slower than the pretraining-scaled approach even when FLOPs are matched.

What evidence exists in the paper. This is a conceptual limitation that the paper does not measure or discuss. There are no latency measurements, no wall-clock time comparisons, and no analysis of how the sequential-to-parallel tradeoff interacts with deployment latency requirements. The paper treats "budget" purely in terms of generation count, which abstracts away the critical practical distinction between latency (serial computation) and throughput (parallel computation).

Mitigation status. Not addressed. The paper does not discuss latency, throughput, or the practical constraints of serial vs. parallel computation in deployment. A practitioner deploying a compute-optimal system would need to separately account for latency budgets, which might shift the optimal policy substantially toward parallel strategies even where sequential strategies are more generation-efficient. This is particularly relevant for the revision model, where the optimal policy on easy problems is almost purely sequential (Figure 7, right: bin 1 performance is essentially flat across all ratios because easy problems are trivially solvable, but the paper still recommends sequential revisions as the most compute-efficient choice — a recommendation that ignores latency entirely).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not shift any research paradigm — it does not propose a new model architecture, training objective, or inference algorithm. What it accomplishes is something quieter but, for the specific subfield of financial NLP evaluation, more practically useful: it reveals a capability bottleneck that was invisible under all prior benchmark designs and provides an evaluation instrument precise enough to measure it. The bottleneck is question detection in long-form financial dialogue — the finding that today's most advanced long-context LLMs, even when they answer provided questions with near-perfect fidelity (F1 > 90% on question answering), miss more than half of the financially meaningful questions when asked to discover them autonomously in meeting transcripts (recall < 46% across all models, Table 5). Two things make this finding landscape-shifting for the financial NLP community: first, prior benchmarks could not have discovered it because they uniformly supply questions rather than requiring their discovery; second, it reframes what "financial language understanding" means in a way that realigns evaluation with actual analyst workflows.

A reframing, not a paradigm shift. The conceptual move is from evaluating models as answerers (given a question, can you produce the right answer?) to evaluating them as analysts (given a document, can you identify what questions were asked and what the answers were?). This is not a new problem — information extraction and open-domain QA have decades of history — but M³FinMeeting is the first benchmark to instantiate this framing specifically for financial meeting transcripts, with the domain complexity (GICS sector terminology, multi-party dialogue, 60+ minute durations) that makes the question-discovery challenge genuinely hard. The paper's contribution is the demonstration that this framing reveals a sharp capability dissociation (answer generation >> question detection) that should reorder research priorities in financial NLP: improving question detection in dialogue is now clearly the highest-leverage target for making meeting understanding systems practically useful, ahead of improving answer quality or long-context retrieval.

Resolving prior contradictions. The paper does not directly resolve a contradiction in the literature — financial meeting understanding was essentially unevaluated before M³FinMeeting. But it does explain why the gap existed: the financial NLP community gravitated toward benchmarks sourced from structured, monologic documents (earnings reports, news, filings) because those were easier to annotate at scale and easier to evaluate with exact-match metrics. Meeting transcripts — long, noisy, multi-speaker, loosely structured — were avoided precisely because they resist clean evaluation. M³FinMeeting demonstrates that the multi-metric approach (alignment metrics + LLM-judge + human validation) can produce credible evaluation even for these messy text types, which should lower the barrier for others to construct meeting-understanding benchmarks in adjacent domains (healthcare grand rounds, legal depositions, corporate strategy sessions).

Which research directions become more attractive. The paper's findings redirect investment along two axes:

1. Toward question detection in dialogue, away from incremental improvements in structured-document QA. If current models already achieve >90% F1 on answering provided questions about meetings (Table 5), further work on better answer generation for financial QA is likely to hit diminishing returns on meeting-understanding tasks specifically. The bottleneck is upstream — finding the questions — and this paper provides a clean benchmark (QA pair extraction) for measuring progress on that bottleneck. Research on dialogue structure parsing, financial intent classification in spoken language, and long-context information extraction becomes directly relevant in a way it wasn't when all financial QA benchmarks supplied the questions.

2. Toward full-context processing, away from RAG-as-default for long financial documents. The RAG experiment (Figure 3, Section 4.2) provides domain-specific counter-evidence to the prevailing assumption that chunked retrieval is the right scaling strategy for long documents. For meetings exceeding ~15K tokens, providing the full transcript in a single prompt outperforms retrieving the top-5 embedding-similar chunks, using a standard RAG pipeline. This doesn't invalidate RAG — the finding is specific to this model (Qwen2.5-72B-Instruct), this task (financial meeting QA), and this retrieval configuration — but it does establish that the "always RAG for long documents" heuristic has exceptions worth studying systematically. Research on when and why full-context processing outperforms retrieval, and on hybrid approaches that combine both, becomes more motivated.

What changes for the financial NLP ecosystem. The paper positions M³FinMeeting as a diagnostic evaluation rather than a training resource, and this choice has ecosystem implications. If the benchmark gains adoption, financial LLM developers will have a standardized way to demonstrate meeting understanding capability — a use case that was previously unevaluable. This could accelerate the development of meeting-specialized models (fine-tuned on meeting transcripts) in the same way that FinQA accelerated numerical reasoning research or LongBench accelerated long-context research. The paper's detailed per-language, per-sector, and per-length breakdowns provide a richer diagnostic signal than aggregate leaderboard scores, which should encourage model developers to report stratified performance rather than gaming a single number.

What does not change. The paper does not solve financial meeting understanding — it reveals how far current models are from it on the hardest sub-task (question discovery). It does not provide training data or methods for closing that gap. It does not establish whether the performance ceiling is near (e.g., if humans also achieve only 60–70% recall on QA pair extraction) or far (if humans achieve near-perfect extraction). These unknowns mean the benchmark diagnoses a problem without prescribing a solution, which limits its immediate practical impact but maximizes its research-stimulating value.

---

Follow-Up Research This Work Enables

Training a question-detection classifier on M³FinMeeting-style data. The paper's central finding — that question detection, not answer generation, is the dominant failure mode — immediately suggests a targeted research direction: can we train a dedicated question-detection model (a classifier or span-extractor) that identifies financially substantive questions in meeting transcripts, separate from the answer-generation LLM? M³FinMeeting provides gold-standard question annotations (Table 4 statistics) that could serve as training data for such a model. A strong follow-up would: (1) fine-tune a moderate-sized encoder (e.g., a multilingual BERT variant) on the M³FinMeeting QA pair extraction annotations to identify question spans; (2) use this classifier as a preprocessing step before feeding detected questions + full transcript to a long-context LLM for answering; (3) compare the pipeline (detect-then-answer) against end-to-end QA pair extraction (current approach) and against the question answering baseline where gold questions are provided. This experiment would quantify how much of the 45+ percentage-point recall gap can be closed by decomposing question detection and answer generation into separate specialized components — and how much of the gap reflects genuinely hard ambiguity in identifying financial questions in dialogue that no current approach can resolve.

Cross-model and cross-domain replication of the question-detection bottleneck. The paper's findings are based on seven LLMs evaluated on financial meeting transcripts. A critical follow-up question is whether the question-detection/answer-generation dissociation generalizes across model architectures, scales, and domains. A strong replication study would: (1) evaluate a broader set of models (including models not in the Qwen/GPT families, such as Claude, Gemini, Mistral, and smaller fine-tuned models) on M³FinMeeting's QA pair extraction vs. question answering tasks; (2) construct an analogous benchmark in a different meeting-heavy domain — medical case conferences, legal depositions, or academic seminars — with the same task structure (extract QA pairs vs. answer provided questions); (3) test whether the dissociation magnitude (F1 QA answering > 90% vs. recall QA extraction < 50%) is specific to financial meetings or reflects a general property of long-form multi-speaker dialogue understanding. If the dissociation replicates across domains, it would establish question detection as a fundamental bottleneck in spoken-language understanding, not just a quirk of financial terminology. If it fails to replicate in some domains, the domain-specific features that make financial question detection easier or harder would become an informative object of study.

Calibrating the embedding similarity threshold and exploring alternative alignment methods. The paper's alignment-based evaluation (precision/recall/F1) depends on a cosine similarity threshold of 0.75 with OpenAI's text-embedding-3-small embeddings — a hyperparameter chosen without ablation. A methodological follow-up would systematically vary this threshold (e.g., 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90) and report how the precision/recall/F1 curves shift for each task and model. The goal is to establish: (1) whether the "all models below 30% F1" finding is robust or an artifact of a strict threshold; (2) what threshold maximizes agreement with human judgments of structural alignment (using the existing human evaluation data from Section 4.4); (3) whether different embedding models (e.g., text-embedding-3-large, E5, BGE, or a financial-domain fine-tuned embedder) produce systematically different alignment patterns. Additionally, exploring alternative alignment methods — such as using an LLM to judge pairwise section/question alignment rather than embedding similarity, or using optimal transport-based matching that enforces one-to-one alignment rather than many-to-many — would clarify whether the low F1 scores reflect genuine model failure or limitations of the alignment metric. This research would strengthen the benchmark's methodology and provide guidance for future meeting-understanding benchmarks in other domains.

Combining M³FinMeeting-style evaluation with fine-tuning on meeting transcripts. The paper evaluates models exclusively in zero-shot settings. A natural extension is to test whether fine-tuning on meeting-style data — even if not on M³FinMeeting itself — substantially closes the performance gap. A strong experiment would: (1) take an open-source model (e.g., Qwen2.5-7B-Instruct) and fine-tune it on available meeting-summarization or QA datasets (ECTSum's bullet-point summaries, AMI/ICSI meeting corpora, or a synthetically generated dataset of meeting transcripts with QA pairs); (2) evaluate the fine-tuned model on M³FinMeeting alongside the zero-shot baselines reported in Table 5; (3) measure whether fine-tuning improves QA pair extraction recall specifically (the bottleneck) or whether gains are concentrated on summarization and question answering (where zero-shot performance is already stronger). If fine-tuning substantially improves QA pair extraction, it would suggest that the question-detection deficit is largely a domain-adaptation problem solvable with modest training data. If fine-tuning helps summarization and QA but not QA pair extraction, it would suggest that question detection requires fundamentally different capabilities (dialogue structure parsing, intent recognition) that current fine-tuning approaches don't impart.

Exploring the human performance ceiling on M³FinMeeting tasks. The paper reports model performance but not human performance on the tasks themselves — the human evaluation in Section 4.4 rates model outputs for quality, not humans performing summarization and QA pair extraction from scratch. Establishing human performance on M³FinMeeting would transform the benchmark's interpretability. A critical follow-up would: (1) recruit trained financial analysts (similar to the original annotators) to perform summarization and QA pair extraction on a random subset of 50–100 meetings, without access to the gold annotations; (2) evaluate their outputs using the same alignment metrics and GPT-4-Judge framework; (3) report both the human-human agreement (how much do different analysts' summaries and QA extractions align with each other?) and the human-model gap. If human analysts achieve 80–90% recall on QA pair extraction, the 45% model ceiling represents a large and meaningful gap. If human analysts achieve only 60–70% (because question identification in meeting dialogue is genuinely ambiguous even for experts), then current models are closer to the practical ceiling than the raw numbers suggest, and research should focus on resolving ambiguity rather than improving detection per se. The human ceiling is also essential for the summarization alignment metrics: if even human annotators produce section summaries with <50% F1 when compared to another human's segmentation (i.e., topic boundary identification is inherently subjective), then the model-vs-gold alignment F1 being <30% is less alarming.

Stress-testing the RAG finding with improved retrieval and different models. The paper's RAG experiment (Figure 3) uses one embedding model, one chunk size (1,024 tokens), and one model (Qwen2.5-72B-Instruct) on a 50-meeting subset. The finding that full-context outperforms RAG at >15K tokens is provocative but fragile. A systematic stress-test would: (1) vary chunk size (512, 1,024, 2,048, 4,096 tokens) and chunking strategy (fixed-size, semantic/section-aware splitting, sliding window with overlap); (2) test multiple retrieval methods (sparse BM25, dense embeddings from multiple models, hybrid retrieval, re-ranking with a cross-encoder); (3) test on models with different long-context capabilities (a weaker model like LLaMA3.1-8B might benefit more from RAG than Qwen2.5-72B, while a stronger model might show an even larger full-context advantage); (4) expand the evaluation to the full 600-meeting dataset to get statistically reliable per-length-bin comparisons. The goal is to establish the boundary conditions under which full-context processing beats retrieval for meeting-length documents — and to identify whether improved retrieval can close the gap at >15K tokens or whether the distributed nature of meeting information fundamentally requires full-context attention.

---

Practical Applications and Downstream Use Cases

Financial analyst workflow augmentation for earnings call season. During quarterly earnings seasons, financial analysts at investment firms must process dozens of earnings calls per week — each 60–90 minutes, each requiring a summary of key points, extraction of the Q&A exchanges that revealed new information, and compilation of answers to standardized questions (revenue guidance changes, margin pressures, competitive dynamics). Currently, this is done by junior analysts listening to calls and taking manual notes — a process consuming 2–3 hours per call. M³FinMeeting's task suite directly mirrors this workflow. The paper's results suggest that a system using Qwen2.5-72B-Instruct (the best-performing model overall, with GPT-4-Judge scores of ~73–84 depending on language and task, Table 5) could produce a first-pass meeting summary and QA extraction that an analyst then reviews and corrects, rather than starting from scratch. Even at current performance levels (QA pair extraction recall <46%), the system would surface nearly half of the substantive Q&A exchanges automatically, allowing the analyst to focus verification effort on the missing half rather than scanning the full transcript. The RAG finding (full-context processing outperforms retrieval at >15K tokens for QA, Figure 3) provides guidance for deployment architecture: for typical earnings calls (~10K–15K tokens), feed the full transcript to the model in one pass rather than chunking.

Cross-lingual meeting intelligence for global financial institutions. Investment firms, rating agencies, and corporate development teams operating across Asian and Western markets routinely handle meetings in English, Chinese, and Japanese — the three languages M³FinMeeting supports. Current practice relies on human translators or language-specialist analysts, creating bottlenecks and inconsistencies. The paper's per-language performance breakdown (Table 10, Figure 2a) provides actionable guidance: Japanese meetings yield the highest model performance across all tasks (Qwen2.5-72B-Instruct overall GPT-4-Judge 84.10 in JA vs. 69.49 in ZH and 74.48 in EN), suggesting that automated meeting understanding can be deployed with higher confidence for Japanese-language meetings than for Chinese. The finding that the three Qwen models "perform similarly in the Summarization task across all languages" but diverge most in QA pair extraction (Section 4.2) implies that summarization is the most immediately deployable cross-lingual capability, while QA extraction requires more language-specific development. A practical deployment would prioritize Japanese → English → Chinese for automated QA extraction, and treat summarization as equally reliable across all three.

Meeting intelligence for regulated industry compliance monitoring. In heavily regulated sectors — financial services, pharmaceuticals, energy — meetings with external parties (clients, regulators, partners) must be documented for compliance, with questions raised and commitments made tracked for audit purposes. M³FinMeeting's GICS sector coverage (Table 12) shows that Utilities, Energy, and Real Estate meetings actually yield the highest model performance despite having the highest average token counts (Utilities: 17,290 avg tokens, Real Estate: 15,270 — Table 2), with Qwen2.5-72B-Instruct achieving overall GPT-4-Judge scores of 78.78 (Utilities), 79.10 (Real Estate), and 80.30 (Energy). This is practically significant: the longest, most technically dense meetings in the most regulated sectors are also where the best models perform strongest, suggesting that automated compliance documentation — tracking what questions were asked by regulators and what commitments were made in response — is more feasible for these high-stakes meetings than for shorter, less-structured meetings in Consumer Discretionary or Communication Services (where model scores are lower, ~73). A compliance-focused deployment could use M³FinMeeting's QA pair extraction task as a direct template: extract all question-answer exchanges, flag those containing commitment language or regulatory inquiries, and route them for human review with the model's extraction as a starting point. The finding that "the performance gaps among sectors for GPT-4o, Qwen2-72B-Instruct, and Qwen2.5-72B-Instruct are much smaller compared to GPT-3.5-turbo and LLaMA3.1-8B-Instruct" (Section 4.2, Figure 2b) tells practitioners that tier-1 models generalize across sectors well enough to serve as a uniform platform — you don't need sector-specific models for reasonable baseline performance, though fine-tuning might still improve per-sector accuracy.

Cost-efficient evaluation-as-a-service for financial LLM benchmarking. The paper's evaluation methodology — dual LLM judges (GPT-4 and Qwen-plus) validated against human annotators with Fleiss' Kappa (Tables 6–8) — provides a template for ongoing, automated benchmarking of financial LLMs as new models are released. The finding that GPT-4-Judge's agreement with human annotators is higher than human inter-annotator agreement (Table 8) is practically important: it means that for the specific tasks and criteria in M³FinMeeting, automated evaluation using GPT-4-Judge is not merely a cheap approximation of human judgment but may actually be more reliable than small panels of human evaluators. A firm maintaining an internal leaderboard of financial LLM performance could run M³FinMeeting evaluation on each new model release with confidence that the GPT-4-Judge scores are tracking human-recognizable quality, without the recurring cost and latency of human evaluation. The reported ~$2,500 in OpenAI API costs for the full 7-model × 600-meeting evaluation (Section 4.2, footnote 6) establishes a concrete budget baseline: benchmarking a single new model costs roughly$350 in API calls for GPT-4-Judge evaluation, plus GPU time for the model's own inference. For a financial institution evaluating quarterly model updates, this is a trivial expense relative to the cost of deploying an underperforming model.