LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

🎯 Pitch

Standard LMM benchmarks like ChartQA and VQAv2 already have over 20% training data overlap with leading models, making their scores unreliable. This paper introduces LIVEBENCH, a dynamic evaluation fed from real-time news and forums, where GPT-4o leads open-source rivals by over 6 points—revealing a real-world generalization gap that static benchmarks completely miss.

1. Executive Summary

This paper conducts a reality check on the evaluation landscape for Large Multimodal Models (LMMs), introducing LMMS-EVAL, a unified and standardized benchmark suite covering over 50 tasks and more than 10 models to ensure transparent, reproducible comparisons. Recognizing that simultaneous wide coverage, low cost, and zero contamination form an impossible trilemma, the authors contribute two complementary solutions: LMMS-EVAL LITE, a pruned evaluation set that reduces cost while maintaining result alignment with full-set evaluations (achieving over 0.87 correlation across benchmarks through k-center coreset selection), and Multimodal LIVEBENCH, a dynamically updated benchmark that sources questions from continuously refreshing news and forum websites to assess zero-shot generalization while preventing data contamination. The contamination analysis reveals that benchmarks like ChartQA and VQAv2 suffer over 20% overlap with LLaVA training data, while LIVEBENCH results show GPT-4o achieving 92.0 overall accuracy compared to the best open-source model Qwen2-VL-72B at 85.9, establishing that commercial models maintain substantial real-world generalization advantages that static benchmarks fail to capture.

2. Context and Motivation

The Core Problem: LMM Evaluation Is Fragmented, Unstandardized, and Potentially Misleading

The fundamental problem this paper addresses is that we lack reliable, standardized, and transparent ways to evaluate Large Multimodal Models (LMMs). As the field rapidly transitions from text-only LLMs to vision-language models like GPT-4V, Gemini, Claude, and open-source alternatives such as LLaVA and Qwen-VL, the evaluation infrastructure has failed to keep pace. This gap is not merely academic — it has direct consequences for how we decide which models work, where we invest development effort, and whether public benchmarks actually reflect real-world capability.

The paper identifies three specific, interacting problems that together form what the authors call the evaluation trilemma:

Problem 1: Wide-coverage evaluation is expensive and often non-reproducible. Evaluating a single LMM on the dozens of benchmarks now available (AI2D for science diagrams, ChartQA for chart reading, DocVQA for document understanding, MMMU for multi-disciplinary reasoning, MathVista for math, and many more) requires assembling custom inference pipelines per model, handling different data formats, different output postprocessing conventions, and different metrics. The authors point out (Section 2.1) that "model publishers come up with custom evaluation pipelines, which often differ significantly in data preparation, output postprocessing, and metrics calculation, hindering transparency and reproducibility." This means that a score reported for LLaVA on MMBench might not be directly comparable to a score reported for Qwen-VL on the same benchmark, because one model's evaluation used perplexity-based answer selection while another used generation-based matching. Wide coverage — evaluating across many dimensions to get a complete picture — compounds this overhead linearly with the number of datasets.

Problem 2: Reducing evaluation cost typically narrows coverage or introduces contamination risk. The natural response to expensive evaluation is to reduce the number of benchmarks, use smaller test sets, or rely on a handful of representative datasets. But this creates a tension: cut too aggressively and you lose the broad diagnostic signal that identifies specific model weaknesses; cut too conservatively and the evaluation remains prohibitively expensive for rapid model iteration (e.g., during ablations or architecture search). The Hugging Face OpenLLM leaderboard for text-only models exemplifies this tradeoff — it is economical but "prone to overfitting and contamination" because models are specifically trained against its fixed set of benchmarks.

Problem 3: Static benchmarks are increasingly contaminated. As LMMs are trained on ever-larger web-scale datasets — Qwen-VL uses 1.4 billion pretraining samples, CogVLM uses 1.5 billion — the likelihood that benchmark test data appears in pretraining or fine-tuning corpora grows substantially. The paper demonstrates this concretely in Section 4.1, showing that ChartQA has a 68.64% image overlap and 26.52% text overlap with LLaVA training data, while VQAv2 and COCO each have over 45% image overlap. This is not accidental: "the re-annotation and conversion of large web and academic datasets into training materials frequently lead to issues of overlap and contamination" (Section 4.1). A benchmark score that partly reflects memorization of test data, rather than genuine visual reasoning, is misleading — it artificially inflates perceived model capability and masks real weaknesses.

The trilemma is that these three problems constrain each other (Figure 1): you can have wide coverage and low cost, but the static benchmark data will be contaminated (the OpenLLM problem). You can have zero contamination and wide coverage, but it will be expensive (the Chatbot Arena problem — requiring tens of thousands of human preference judgments). You can have zero contamination and low cost, but you sacrifice coverage (testing only a narrow slice of capability). The paper's core premise is that this trilemma cannot be fully broken, but the tradeoffs within it can be substantially improved.

Why This Problem Matters

Pacing model development. The paper explicitly frames this with the opening line: "Good benchmarks guide AI development" (Section 1). When evaluation signals are unreliable or prohibitively expensive, developers cannot efficiently iterate. They cannot tell whether a new architecture, training data strategy, or prompting technique genuinely improves capability or merely overfits to contaminated test sets. The fragmentation of evaluation pipelines means that comparing results across published papers requires heroic assumptions about methodological consistency. This slows the entire field down.

The gap between benchmarks and real user experience. Section 4.2 begins with a telling observation: "While open-source models often outperform commercial ones like GPT-4V in benchmarks, they fall short in real user experience." If benchmarks systematically overstate open-source model capability relative to closed-source alternatives — either because of contamination or because static test sets fail to capture the generalization demands of real-world use — then the research community is optimizing for the wrong targets. Researchers tout benchmark victories that don't translate to production systems, users adopt and abandon tools based on misleading signals, and resource allocation decisions (commercial vs. open-source investment) are made on flawed premises.

A systemic threat to scientific progress. Contamination is not just a nuisance error term — it systematically biases results in a way that compounds as models scale. Larger training data pools increase contamination probability. If contamination goes unmeasured (and the paper's analysis in Section 4.1 shows many benchmarks have never been systematically audited for overlap with training corpora), then apparent progress on reasoning and understanding may partially reflect progress on memorization. The field could collectively over-invest in architectures and training recipes that improve test-set performance through exposure rather than through genuine visual-linguistic reasoning. This is a classic "Goodhart's law" scenario where the measure becomes the target and ceases to be a good measure.

Where Existing Approaches Fall Short

The paper surveys and critiques several categories of existing work:

Model-by-model, dataset-by-dataset evaluation (Section 2.1). This is the status quo ante that LMMS-EVAL aims to replace. In this approach, each research group writes custom scripts for each model and each benchmark they want to evaluate on. The authors characterize this as producing "much overhead" and requiring users to "manually launch each individual script to preprocess the datasets, inference models, and calculate final scores." More critically, the lack of standardization creates non-comparability: two evaluations of the "same" model on the "same" benchmark can produce different scores because they differ in chat template handling, output parsing strategy (PPL-based vs. generation-based), or metric implementation. The paper gives the concrete example (Section 2.1) of SEED-Bench evaluations: Li et al. (2023c) extract answers by comparing output probabilities among choices (PPL-based), counting an answer correct if the ground truth has the lowest perplexity; Liu et al. (2023a) instead use generation-based evaluation where the model must explicitly output the correct option letter. The same model evaluated under both protocols can yield systematically different scores.

Human-preference arenas (Chatbot Arena, WildVision). The LMSys Chatbot Arena (Chiang et al., 2024) and AI2 WildVision (Lu et al., 2024b) represent an alternative design philosophy: rather than using fixed test sets, these platforms collect real-time human preference judgments on model outputs. This solves contamination (the prompts are live user interactions, not static benchmarks) and provides wide coverage (users naturally test diverse capabilities). However, the paper identifies two critical shortcomings: it is "expensive to gather tens of thousands of human preferences" and "noisy traffic" makes "consistent comparisons tough." Human evaluation introduces its own inconsistency — different raters have different standards, and the same user's judgment varies with context and fatigue. These arenas sit at the "wide coverage, zero contamination, high cost" corner of the trilemma.

Static multi-task benchmarks (MME, MMBench, MMMU, SEED-Bench, MathVista, AI2D, etc.). The paper acknowledges that these benchmarks (many of which LMMS-EVAL incorporates) represent meaningful progress toward holistic evaluation. MMBench evaluates reasoning and perception across multiple dimensions. MMMU tests multi-disciplinary understanding at expert level. MathVista probes mathematical reasoning from visual inputs. However, the paper's own contamination analysis in Figure 4 reveals that many of these benchmarks have significant training data overlap: ChartQA (68.64% image overlap, 26.52% text overlap), VQAv2 (46.21% image overlap), AI2D (25.97% text overlap), SeedBench (13.84% text overlap). The contamination is often structural rather than incidental — training datasets used for LLaVA development explicitly include the training splits of benchmarks like ChartQA, VQAv2, and COCO, and the test splits are not always adequately separated.

Lite benchmark efforts (Perlitz et al., 2024; Vivek et al., 2024; Polo et al., 2024). In the LLM evaluation space, several works have attempted to reduce evaluation cost by selecting representative subsets of benchmarks. The paper surveys these in Appendix A: Perlitz et al. (2024) use stratified random sampling; Vivek et al. (2024) employ anchor points for clustering; Polo et al. (2024) use Item Response Theory (IRT) to create embeddings for data points. The paper's approach (coreset selection via k-center clustering) builds on this lineage but is novel in applying it to the multimodal evaluation context, where both image and text features must be considered jointly for representative selection.

Decontamination efforts (Brown et al., 2020; Shi et al., 2024; Yang et al., 2023a). Prior work in LLM evaluation has explored methods to detect and mitigate data contamination, primarily through n-gram overlap analysis (Brown et al., 2020), similar embedding removal (Shi et al., 2024), or influence functions (Koh and Liang, 2020). However, the paper notes that "the issue of data contamination in benchmarks for LMMs remains relatively unexplored." The multimodal setting introduces unique challenges that text-only methods don't address: images can be near-duplicates even when pixel-different, visual concepts can be tested through different-but-related images, and question text can vary while testing identical underlying capabilities.

How This Paper Positions Itself

The paper positions itself not as solving the trilemma — the authors are explicit that "we cannot break this impossible triangle" (Section 2.2) — but as improving all three tradeoff frontiers simultaneously through three complementary contributions:

LMMS-EVAL addresses the fragmentation problem by providing a unified evaluation infrastructure that standardizes data preprocessing, model inference, output parsing, and metric calculation across 50+ tasks and 10+ model families. This directly attacks the reproducibility and comparability gaps by ensuring that all models are evaluated under identical conditions. As the authors put it, "we believe there is no best setup but one needs to fix one when comparing results across different models" (Section 2.1). The framework follows the design philosophy of LM-EVAL-HARNESS (Gao et al., 2023) but extends it to the multimodal setting, enabling one-command evaluation across multiple models and datasets with automatic logging for transparency.

LMMS-EVAL LITE addresses the cost-coverage tradeoff by applying coreset selection to prune unnecessary data instances while maintaining evaluation quality. The key insight is that many benchmark datasets contain redundancy — similar questions testing similar capabilities — and identifying representative subsets can preserve the overall diagnostic signal at a fraction of the computational cost. The paper operationalizes this through k-center clustering on joint CLIP + BGE-M3 embeddings of image-text pairs, selecting data points that maximize coverage of the full dataset's feature space. This sits at the "wide coverage, low cost" edge of the trilemma (trading off some contamination resistance in exchange for efficiency).

LIVEBENCH addresses the contamination problem by abandoning static evaluation entirely. Rather than fixing test questions in advance (which can be memorized during training), LIVEBENCH continuously scrapes fresh content from news websites and online forums, generates new question-answer pairs using commercial models (Claude-3.5-Sonnet for information extraction and QA generation), and evaluates LMMs on questions about events that occurred after the models' training data cutoff. This sits at the "zero contamination, low cost" edge (accepting some coverage limitations in exchange — the questions are necessarily about current events and the pipeline's output quality may vary). The paper explicitly frames this as a complement to, not replacement for, static benchmarks: "LIVEBENCH aims to evaluate models' zero-shot generalization ability on the most recent events" (Section 4.2), targeting a capability dimension that static benchmarks cannot assess.

The paper's position is thus integrative rather than adversarial: it doesn't claim that any prior approach is wrong, but rather that no single approach can satisfy all evaluation desiderata, and that the community needs a suite of complementary tools — standardized wide-coverage evaluation (LMMS-EVAL), efficient diagnostic evaluation (LMMS-EVAL LITE), and contamination-resistant generalization testing (LIVEBENCH) — to get a complete picture of multimodal model capability. This is a pragmatic, engineering-oriented contribution philosophy: identify the binding constraints, build tools that relax them, and release everything open-source so the community can adopt and extend.

The authors also position their work as a reality check — the paper's title is deliberate. They are not introducing a new model or training technique but rather stepping back to examine whether the evaluation infrastructure that the LMM community relies on is actually adequate for the decisions it informs. The systematic contamination audit (Section 4.1, Figure 4), the documentation of evaluation pipeline inconsistencies (Section 2.1), and the LIVEBENCH results showing commercial models substantially outperform their benchmark scores (Table 3) collectively argue that the field has been operating with at least partially distorted feedback signals. The paper's contributions are designed to correct these distortions.

3. Technical Approach

3.1 Reader Orientation

The paper builds three complementary evaluation tools — a unified benchmarking framework (LMMS-EVAL), an efficient subset selector (LMMS-EVAL LITE), and a dynamically-generated test set (LIVEBENCH) — that together address the fundamental trilemma in multimodal model evaluation where wide coverage, low cost, and zero contamination cannot all be achieved simultaneously. The "shape" of the solution is not a single system but a toolkit philosophy: instead of trying to break the trilemma with one perfect benchmark, the authors accept its impossibility and provide specialized instruments for different evaluation scenarios, with the understanding that practitioners will use them in combination depending on their specific needs (e.g., LMMS-EVAL LITE for rapid model development iterations, LIVEBENCH for contamination-free generalization testing, and the full LMMS-EVAL for comprehensive capability audits).

3.2 Big-Picture Architecture (Diagram in Words)

The overall system can be understood as three independent pipelines that share a common evaluation interface:

LMMS-EVAL Core Framework — a standardized evaluation infrastructure that abstracts away model-specific and dataset-specific differences, providing a single entry point where any supported model can be evaluated on any supported dataset with consistent preprocessing, inference, output parsing, and metric calculation. This is the backbone that makes all evaluations reproducible and comparable.
LMMS-EVAL LITE Selector — a dataset pruning pipeline that takes the full benchmark datasets from LMMS-EVAL, embeds each image-text pair into a joint feature representation (using CLIP for images and BGE-M3 for text), then runs a greedy k-center clustering algorithm to select a subset of data points that maximally represent the full dataset's diversity. The output is a reduced test set (typically 300–700 instances per dataset, from original sizes of 2,500–31,784) that preserves the relative ranking and absolute scores of models evaluated on the full set.
LIVEBENCH Generation Pipeline — a continuously-running data factory that scrapes screenshots from over 60 news websites and forums, uses Claude-3.5-Sonnet to extract information and generate question-answer pairs at four cognitive levels (based on Bloom's Taxonomy), applies automated quality checking and reformatting, and produces a fresh evaluation set of 100–300 questions per month. This pipeline is designed to be automated and low-cost, trading off some question quality for sustainability and contamination resistance.

All three pipelines share a common evaluation interface: models receive multimodal inputs (image + text question), produce free-form or structured responses, and are scored by a judge model (GPT-4o for LIVEBENCH, ground-truth matching for static benchmarks) using pre-defined criteria or exact-match/accuracy metrics. The shared interface means that once a model is integrated into LMMS-EVAL, it can be evaluated across any of the supported benchmarks or LIVEBENCH question sets without additional engineering work.

3.3 Roadmap for the Deep Dive

First, we examine the LMMS-EVAL standardization layer (Section 3.4.1), which is the architectural foundation that makes everything else possible — how the framework normalizes evaluation across models and datasets, why this matters for comparability, and what specific design choices (data parallel inference, automatic logging, unified dataset interface) enable scaling to 50+ tasks.
Second, we unpack the coreset selection methodology behind LMMS-EVAL LITE (Section 3.4.2), which is the paper's most algorithmically substantive contribution: the k-center clustering objective, the joint embedding representation using CLIP + BGE-M3, the greedy approximation algorithm, and the experimental validation showing that scores on the reduced set correlate with full-set scores at $r > 0.87$ across most benchmarks.
Third, we detail the contamination detection methodology (Section 3.4.3) that motivated LIVEBENCH: the 8-gram text overlap analysis, the novel SEED-tokenizer-based image overlap detection, and the three categories of contamination identified (duplicate images, similar images, similar questions).
Fourth, we walk through the LIVEBENCH curation pipeline (Section 3.4.4) from end to end: website selection, information extraction via Claude-3.5-Sonnet, the four-category question generation schema based on Bloom's Taxonomy, the checker-finalizer-scorer quality control chain, and the GPT-4o-based evaluation protocol.
Fifth, we cover the score aggregation methodology (Section 3.4.5) used in LMMS-EVAL LITE to produce a single summary signal from heterogeneous benchmark metrics, explaining the per-dataset normalization to a 0–100 scale and why simple averaging would be inappropriate.
Sixth, we examine the evaluation standardization details (Section 3.4.1 continues with specific metrics) including the distinction between perplexity-based and generation-based answer extraction, the handling of chat templates, and the strategy for supporting both open-source models (via data or model parallelism) and API-based models.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper with contributions spanning evaluation infrastructure, dataset construction, and empirical contamination analysis. The core technical idea is that no single evaluation method can simultaneously satisfy the three desiderata of wide coverage, low cost, and zero contamination, so the field needs complementary tools optimized for different tradeoffs along these dimensions. The paper provides three such tools, each with a distinct technical approach.

3.4.1 LMMS-EVAL: Standardized Evaluation Infrastructure

The first contribution is an engineering system that abstracts away the heterogeneity in LMM evaluation to enable reproducible, comparable, and scalable benchmarking. The system follows the architecture of LM-EVAL-HARNESS (Gao et al., 2023) — a widely-adopted framework for text-only language model evaluation — and extends it to the multimodal setting where inputs include both images and text, and models can be open-source (requiring GPU inference with parallelism strategies) or closed-source (accessed via API).

The evaluation overhead problem. Section 2.1 of the paper characterizes the status quo ante: each model publisher writes custom evaluation scripts for each benchmark, leading to several forms of inconsistency. First, data preprocessing differs: different scripts may resize images differently, apply different normalization, or handle text prompts with different formatting. Second, output parsing differs: "Li et al. (2023c) extracts model answers by comparing the output probabilities among the choices... However, Liu et al. (2023a) use the generation-based evaluation. An answer is counted as correct only if the model's generation matches the option letter." Third, metric calculation differs: the same benchmark may be scored with different aggregation functions depending on the script. These differences mean that published scores for the "same" benchmark are not directly comparable across papers, undermining the scientific value of benchmarking.

Unified dataset interface. LMMS-EVAL preprocesses all datasets into a common format before evaluation begins. The paper states: "We preprocess and handle all the data needed during evaluation, ensuring a single data source is used across different models for a standardized evaluation." This means each dataset (AI2D, ChartQA, DocVQA, MMMU, etc.) is converted to a standardized representation where each instance contains: (1) a question or prompt string, (2) one or more images (for multimodal benchmarks) or an indicator that the question is text-only, (3) ground-truth answer(s) in a known format (multiple-choice options, free-form answer strings, or structured outputs), and (4) metadata indicating the evaluation metric type (accuracy, CIDEr, ANLS, F1, etc.). This abstraction layer means that when a new model is added to the framework, it automatically works with all supported datasets without requiring per-dataset code changes.

Model inference abstraction. The framework provides a unified model interface that handles both open-source and API-based models. For open-source models (LLaVA variants, Qwen-VL, InternVL, InstructBLIP, Idefics2, etc.), the framework supports two parallelism strategies depending on model size: data parallelism (replicating the model across multiple GPUs and distributing different evaluation instances to each GPU) for models under 72B parameters, and pipeline parallelism (splitting the model layers across GPUs) for larger models. The paper reports in Figure 2 that evaluations were conducted "using 8×A100 GPUs with flash attention enabled" and that "for models larger than 72B, we use pipeline parallelism to load a single model across different GPUs." For API-based models (GPT-4V, GPT-4o, Gemini, Claude, Qwen-VL-Max), the framework handles API authentication, rate limiting, and response parsing through a common client interface.

Chat template handling. A subtle but important standardization choice is the handling of chat templates. Instruction-tuned models are trained with specific prompt formats (system messages, role markers, special tokens), and deviating from these formats at evaluation time can degrade performance in ways that confound model comparison. The paper states: "For a fair comparison, we also respect the chat template of the models if they are instruction-tuned." This means that when evaluating, say, LLaVA-NeXT-Vicuna-7B, the framework uses the Vicuna chat template that the model was fine-tuned with, while evaluation of Qwen-VL-Chat uses the Qwen-specific template. By handling this automatically rather than requiring each evaluator to manually format prompts, LMMS-EVAL removes a source of cross-study variance.

Automatic logging for reproducibility. The framework automatically logs "the evaluation setup, model generations, and score breakdown" to enable post-hoc verification and analysis. This addresses a common complaint in the benchmarking literature: published papers often report only aggregate scores without providing model outputs, making it impossible to audit the evaluation or understand failure modes. The paper emphasizes that detailed outputs are "automatically logged for future analysis," which is particularly important for multi-task benchmarks like MMMU where understanding which sub-disciplines or question types a model struggles with requires granular access to per-instance results.

Scaling to 50+ tasks. Table 25 in Appendix F lists the full set of supported datasets, which span diverse task domains: Science (AI2D, ScienceQA), Chart Understanding (ChartQA), Document Understanding (DocVQA, InfoVQA, MultiDocVQA), Captioning (COCO, Flickr30k, NoCaps, TextCaps), Visual QA (GQA, OKVQA, TextVQA, VQAv2, VizWizVQA), Referring Expression (RefCOCO, RefCOCO+, RefCOCOg), Multi-task Benchmarks (MMBench, MME, MMMU, CMMMU, SEED-Bench, MM-Vet), Math (MathVista, MathVerse), Hallucination Detection (POPE, HallusionBench), OCR (OCRBench), Web Understanding (VisualWebBench, WebSRC), and GUI Understanding (ScreenSPOT). The ground-truth types include multiple-choice, short answer, free-form, yes/no, captioning, and referring expressions. Each requires different evaluation metrics: accuracy for classification, CIDEr for captioning, ANLS for document understanding, F1 for hallucination detection, and exact match for text recognition. The framework standardizes all of these into a common evaluation pipeline.

Model coverage. Table 26 in Appendix F lists 27 model variants across 10+ model families supported in the initial release: LLaVA-1.5 (7B, 13B), LLaVA-NeXT (Vicuna-7B/13B, Mistral-7B, LLaMA3-8B, Yi-34B, Qwen-72B/110B), LLaVA-OV (0.5B, 7B, 72B, with and without SI), InstructBLIP (7B, 13B), Qwen-VL-Chat (7B), Fuyu-8B, Idefics2-8B, MiniCPM-V-2.8B, XComposer-4KHD (8B), InternVL-1.5 (26B), plus API-based models (GPT-4V, GPT-4o, Gemini 1.0/1.5 Pro/Flash, Claude-3 Haiku/Sonnet/Opus). This coverage is important because it demonstrates the framework's generality across model architectures (LLaMA-based, Vicuna-based, internally-developed), model sizes (0.5B to 110B for open-source), and deployment modes (local GPU vs. cloud API).

Concrete results demonstrating standardization value. Table 1 in the main paper shows selected results across 8 benchmarks for 22 model variants, all evaluated through the same pipeline. This table is only possible because of the unified framework — comparing, say, LLaVA-1.5-7B and Qwen-VL-Chat on MathVista would be misleading if the two models were evaluated under different parsing conventions. The framework ensures that both models receive identical prompts, with answers extracted using the same method, scored by the same metric. The resulting numbers (e.g., LLaVA-1.5-7B achieving 26.7 on MathVista while LLaVA-OV-72B achieves 67.5) can be directly compared because the evaluation protocol is identical.

Why this design over alternatives? The alternative would be to continue the status quo of per-model, per-benchmark evaluation scripts. The paper argues this produces "much overhead" (Section 2.1) and non-comparable results. A natural question is why the authors didn't simply rely on the existing LM-EVAL-HARNESS framework. The answer, implied by the paper's description, is that LM-EVAL-HARNESS was designed for text-only models and doesn't handle multimodal inputs (images), multimodal model architectures (vision encoders, projector layers, different image preprocessing requirements), or multimodal-specific metrics (captioning evaluation, visual grounding metrics). LMMS-EVAL extends the harness design pattern to the multimodal domain while preserving the key abstraction benefits: one-command evaluation, unified data preprocessing, consistent metrics, and automatic logging.

3.4.2 LMMS-EVAL LITE: Coreset Selection for Efficient Evaluation

The second contribution addresses the cost side of the evaluation trilemma through dataset pruning via coreset selection. The core insight is that many benchmark instances are redundant — they test similar capabilities on similar images — and identifying a representative subset can substantially reduce evaluation cost while preserving the signal about relative model quality.

The formal objective. Section 3 defines the problem of constructing a lite benchmark set. Let the full benchmark be represented as:

$D = \{(x_i, y_i)\}_{i=1}^n$

where $x_i$ is the input to the model (image + text prompt) and $y_i$ is the ground-truth answer. Given a model $f$ , the model's response to instance $x_i$ is $\hat{y}_i = f(x_i)$ , and a scoring function $S(y_i, \hat{y}_i)$ evaluates the correctness of the response. The goal is to select a subset $V \subset D$ such that:

$\min_{V: |V| \ll |D|} \left| \frac{1}{|D|} \sum_{i=1}^{|D|} S(y_i, \hat{y}_i) - \frac{1}{|V|} \sum_{i=1}^{|V|} S(y_i, \hat{y}_i) \right|$

where $|D|$ is the size of the full dataset and $|V|$ is the size of the reduced set.

What it computes: the absolute difference between the model's average score on the full benchmark and its average score on the subset. The optimization minimizes this difference by choosing which instances go into $V$ . If the selected subset produces aggregate scores close to the full set, we can use it as a cheap proxy for the full evaluation.

Why this form: the objective directly measures what we care about — evaluation reliability. An alternative would be to maximize the diversity of selected instances (e.g., via stratified sampling), but that doesn't guarantee score alignment. The k-center formulation ensures that selected points are representative in the feature space that matters for model evaluation, which the paper demonstrates leads to better score correlation than alternatives like random sampling or k-means clustering.

Reduction to k-center clustering. The paper notes that this objective "is equivalent to solving the k-Center problem," which seeks to identify a subset of data points whose maximum distance to any point in the full set is minimized. Let $\Delta(x_i, x_j)$ be a distance metric between data points in some embedding space. The k-center objective selects $k = |V|$ centers from the $n = |D|$ data points such that:

$\min_{C \subset D, |C| = k} \max_{i \in D} \min_{j \in C} \Delta(x_i, x_j)$

where $C$ is the set of selected center points, $k$ is the desired subset size, and $\Delta$ is the distance function.

What it computes: the worst-case distance from any data point in the full set to its nearest selected center. Minimizing this maximum distance ensures that every point in the full dataset is "close to" some selected representative, which in turn ensures that the subset covers the full distribution of difficulty levels, visual concepts, and question types present in the original benchmark.

Why this form: the k-center objective provides a worst-case guarantee — no point in the full dataset is farther than some bounded distance from a selected center. This is stronger than average-case objectives like k-means, which might leave some regions of the feature space completely unrepresented. Since we care about the subset's performance across the full range of model capabilities, the worst-case coverage property is more important than average reconstruction error.

The greedy approximation algorithm. Since "solving the k-Center problem is NP-hard," the paper uses "a greedy algorithm to achieve a 2-OPT solution efficiently." The algorithm is detailed in Algorithm 1 (Appendix D.4):

Initialization: Start with an empty set of centers $s = \emptyset$ .
First center: Select a random point as the first center.
Iterative selection: While $|s| < k$ :
- For each unselected point $i \in D \setminus s$ , compute its minimum distance to any already-selected center: $d_i = \min_{j \in s} \Delta(x_i, x_j)$ .
- Select the point with the maximum $d_i$ : $u = \arg\max_{i \in D \setminus s} d_i$ .
- Add $u$ to the center set: $s = s \cup \{u\}$ .
Output: Return $s$ as the selected subset.

What this computes: each iteration greedily adds the point that is furthest from all previously selected centers. This ensures that each new center covers a region of the feature space that is currently least represented.

Why the greedy algorithm: while it doesn't guarantee global optimality (the problem is NP-hard), the greedy algorithm provably achieves a 2-approximation to the optimal k-center solution — the maximum distance in the greedy solution is at most twice the optimal maximum distance. This is the best possible approximation ratio for this problem unless P=NP. Moreover, the greedy algorithm runs in $O(nk)$ time, making it computationally tractable even for large benchmarks (the largest dataset, Flickr30k, has 31,784 instances).

Embedding construction for multimodal data. The critical design choice is how to embed each data point — an image-text pair — into a feature vector on which distances can be measured. The paper's approach is conceptually straightforward but novel in its application to multimodal benchmark pruning:

Image embedding: Each image is encoded using CLIP (Radford et al., 2021), producing a dense vector representation that captures visual semantics.
Text embedding: Each text prompt or question is encoded using BGE-M3 (Chen et al., 2024a), a multilingual embedding model that captures semantic content of the text.
Concatenation: The two embeddings are concatenated into a single joint feature vector. The paper doesn't specify the dimensionality or weighting, but the concatenation approach ensures that both visual and textual information contribute to the distance metric.

An alternative embedding approach. Appendix D.1 describes an alternative embedding strategy that did not make it into the main LMMS-EVAL LITE configuration but was tested during development: training a small LLaVA-Qwen 1.8B model (following the training recipe of Liu et al., 2023a) and using its last hidden states as embeddings. Specifically, "the last hidden states for all tokens were averaged into a single vector to serve as the feature vector for each data point." This approach has the theoretical advantage of producing embeddings that are aligned with the model family being evaluated (since the embedding model shares architecture with the evaluated models). However, the correlation results in Table 5 show that CLIP+BGE embeddings generally perform comparably or better than LLaVA embeddings, and the CLIP+BGE approach is simpler (no model training required).

Correlation validation. To validate that the selected subset preserves relative model rankings and absolute scores, the paper evaluates "six versions of LLaVA" on both the full benchmark and the lite subset, then computes the correlation between full-set scores and lite-set scores. Table 2 reports these correlations for several benchmarks, comparing the paper's k-center approach against two baselines:

Quire: an active learning method that selects "the most informative and representative points" (Appendix A).
k-means: the standard clustering algorithm that minimizes within-cluster variance.

The results show that the k-center approach consistently achieves high correlations: AI2D achieves 0.98, Flickr30k achieves 0.91, SeedBench achieves 0.87, and TextVQA achieves 0.99. The comparison with k-means is particularly instructive — on Flickr30k, k-center (0.91) substantially outperforms k-means (0.79), while on SeedBench the two methods tie at 0.87. The Quire baseline shows dramatically worse performance on some benchmarks (AI2D: 0.45, SeedBench: 0.27), suggesting that informativeness-based selection (Quire's design principle) doesn't generalize well to this context.

Full correlation results. Table 5 in Appendix D.1 provides comprehensive correlation results across all 15 lite benchmarks, using both LLaVA embeddings and CLIP+BGE embeddings. The CLIP+BGE approach achieves correlations above 0.90 on 12 out of 15 benchmarks, with the lowest being SeedBench at 0.87 and InfoVQA at 0.94. The consistency of these high correlations across diverse task domains (chart understanding, document QA, captioning, referring expressions, visual QA, multi-task benchmarks) is the key empirical evidence that coreset selection works for multimodal evaluation.

Lite set construction specifics. The paper constructs LMMS-EVAL LITE by selecting 15 datasets "across different task domains for broad coverage" and applying the selection method only to datasets with "over 1500 data points." Larger datasets (Flickr30k: 400 selected from 31,784; DocVQA: 400 from 5,349; SeedBench: 700 from 17,990) are aggressively pruned, while smaller datasets (LLaVA-W: 60 from 60; MMMU: 900 from 900; CMMMU: 900 from 900) are retained in full. The resulting lite set contains 9,134 instances, compared to 90,223 in the full set — a roughly 10× reduction in evaluation instances. For MME, which has 2,374 instances, the paper notes that "due to low correlation between the original and lite set scores, we retain the full version" — an important admission that the method doesn't work uniformly across all benchmarks. The specific MME correlation failures are not detailed, but this demonstrates a principled approach: validate the pruning empirically and retain full datasets where it doesn't work.

An extended lite version. Appendix D.3 and Table 7 describe an extended LMMS-EVAL LITE that incorporates more datasets (COCO, VQAv2, OKVQA, VizWiz-VQA, GQA, MM-Bench cn/en), bringing the total from 15 to 22 benchmarks while expanding the lite set from 9,134 to 13,734 instances. This demonstrates the modularity of the approach — new datasets can be added to the lite set by applying the same coreset selection methodology, and the paper provides the full configuration in Table 7 as a reference for practitioners.

Evaluation cost comparison. Figure 2 shows that the full LMMS-EVAL evaluation of LLaVA models on all datasets requires substantial compute time, motivating the need for lite evaluation. The exact time values are not readable from the figure description, but the visual contrast between the full and lite bars demonstrates the computational savings. The paper notes that these evaluations "were conducted using 8×A100 GPUs with flash attention enabled," providing the hardware context.

Why coreset selection over stratified sampling? One natural alternative would be stratified random sampling: divide the dataset into categories (e.g., by question type, difficulty, visual domain) and sample proportionally from each. The paper doesn't explicitly compare against this, but the comparison against random baselines (implied by the Quire results) and k-means suggests that the embedding-based approach captures non-obvious structures that simple categorical stratification would miss — for example, two questions about different types of charts might test similar visual reasoning skills even though they fall in different nominal categories. The joint CLIP+BGE embedding space captures these semantic similarities.

Why CLIP+BGE over other embedding choices? The paper doesn't provide an extensive ablation of embedding choices, but the motivation is implicit: CLIP is the standard image encoder in vision-language research and produces embeddings that are well-aligned with how current LMMs represent images (since many LMMs use CLIP-based vision encoders). BGE-M3 is a state-of-the-art multilingual text embedding model. The concatenation approach avoids the need for a specialized multimodal embedding model (which would need to be trained specifically for this purpose) while still capturing both visual and textual similarity.

3.4.3 Contamination Detection Methodology

The paper's contamination analysis in Section 4.1 is both a standalone contribution (revealing widespread contamination in existing benchmarks) and the motivation for LIVEBENCH (showing that static benchmarks are fundamentally vulnerable to this problem). The methodology has two components: text overlap detection and image overlap detection.

Text overlap detection. The approach follows established practice from the LLM literature (Brown et al., 2020; Team, 2023a; Touvron et al., 2023) but is adapted for the multimodal evaluation context:

N-gram extraction: All text in both the training data and the benchmark data is tokenized into 8-gram sequences. The paper states: "Typically, an 8 ∼ 13 n-grams range is used, but we consistently use 8 n-grams for simplicity."
Meaningless n-gram filtering: Any n-gram that appears more than 10 times in the training data is excluded from the overlap analysis. These are presumably common phrases, function words, and boilerplate text that would produce false-positive overlap signals.
Overlap ratio computation: For each n-gram in a benchmark instance, the system checks whether it appears in the training data (subject to the meaningless n-gram filter). The overlap ratio for a benchmark instance is the fraction of its n-grams that appear in the training data.
Additional filtering: The paper mentions excluding n-grams "exceeding a predefined threshold" when matching against the meaningless n-gram set, though the exact threshold is not specified.

What this detects: verbatim text copying between training and test data. If a benchmark question's exact wording appears in the training corpora (even as part of a larger document), the 8-gram overlap will capture it.

Limitation: 8-gram matching only catches exact string matches. Paraphrased test questions, translated questions, or questions testing the same knowledge with different wording would not be detected. The paper acknowledges this implicitly by showing that text overlap rates (Table 4) are generally lower than image overlap rates — text is easier to modify than images while preserving the underlying evaluation target.

Image overlap detection — the novel contribution. This is the more technically interesting component because "determining image overlap is a more challenging task" than text overlap (Section 4.1). The paper's key insight is to use a tokenizer-based approach rather than the more common embedding-similarity approach:

"Instead of computing similarity in the embedding space, we empirically find that using the pretrained SEED-tokenizer leads to meaningful separation in detecting the overlap."

The procedure works as follows:

Image tokenization: Each image is passed through the SEED-tokenizer (Ge et al., 2023), which converts it into a 1-D sequence of 32 discrete tokens. These tokens are analogous to the token IDs produced by a text tokenizer, but they represent visual concepts rather than linguistic units.
8-gram construction: An 8-gram lookup table is constructed from these image token sequences, exactly paralleling the text overlap analysis.
Overlap detection: If an 8-gram (8 consecutive image tokens) appears in both a training image and a benchmark image, those images are flagged as overlapping. The paper notes that "the occurrence of 8-gram overlap can be interpreted as approximately 1/4 of the image overlapping," since 8 tokens out of 32 represents 25% of the image's token sequence.

Why tokenization over embedding similarity? The paper doesn't elaborate extensively on this design choice, but the reasoning can be inferred. Embedding similarity (e.g., cosine similarity between CLIP embeddings) would require setting a similarity threshold to declare "overlap," and "selecting an appropriate threshold applicable to all datasets is difficult." Different datasets have different degrees of visual diversity; a threshold that works for charts (where similar visual structures are common) might not work for natural images. The tokenization approach provides a discrete decision criterion: an overlap is either present (the 8-gram matches) or absent (it doesn't). No threshold tuning is required.

Why SEED-tokenizer specifically? SEED-tokenizer is designed to produce discrete visual tokens that can be fed into language models, making it a natural fit for this n-gram-based approach. The discrete tokenization means that visually similar image regions map to similar (or identical) token sequences, while dissimilar regions map to different token sequences. The 8-gram unit captures local visual patterns (approximately one-quarter of the image) that indicate meaningful content overlap rather than coincidental low-level similarity.

Contamination results. Table 4 presents the overlap statistics across 21 benchmarks. The most contaminated benchmarks (with respect to LLaVA-NeXT training data) are:

ChartQA: 68.64% image overlap, 26.52% text overlap — the most severely contaminated benchmark, because ChartQA's training set was explicitly included in LLaVA's training data.
COCO2014: 46.05% image overlap, 22.19% text overlap.
VQAv2: 46.21% image overlap, 2.90% text overlap.
POPE: 42.20% image overlap, 0.00% text overlap — but this is because "POPE uses image sources from COCO," so the image overlap reflects the COCO images rather than POPE-specific data.
DocVQA: 36.08% image overlap, 4.06% text overlap.
AI2D: 6.09% image overlap, 25.97% text overlap — showing that text contamination can be severe even with low image overlap.

Most benchmarks maintain relatively low contamination (below 10% for both image and text), but the paper's key finding is that some of the most widely-used benchmarks are heavily contaminated, and this contamination has been largely unexamined in the LMM literature.

Three contamination categories. The paper's analysis identifies three distinct patterns of contamination, illustrated in Figure 5 and Figure 7:

Duplicate images: Identical images appearing in both training and benchmark data. Examples are found in ChartQA and MM-Vet. This is the clearest form of contamination.
Similar images: Visually similar (but not pixel-identical) images across training and benchmark, detected through shared image token n-grams. The paper gives examples from NoCaps, ChartQA, and MM-Vet, noting that "such similarities could lead to semantically similar questions."
Similar questions: Recurring question structures even when images differ. The text n-gram analysis captures these cases. The paper presents examples from MathVista where "though not necessarily contamination or overlapping cases, the two images are both testing similar domain knowledge and may help the model to answer questions in the benchmarks."

The third category is particularly subtle — it's not exactly "contamination" in the sense of memorized test answers, but it represents a form of evaluation leakage where training on questions with the same structure gives models an unfair advantage on benchmark questions, even if the specific numbers or images differ.

Why this methodology matters beyond the paper. The contamination analysis serves two purposes in the paper's argument. First, it empirically demonstrates that static benchmarks are vulnerable to contamination, motivating the need for LIVEBENCH's dynamic approach. Second, it provides a concrete, reproducible methodology that other researchers can use to audit their own benchmarks and training datasets — the paper open-sources the detection tools, making this analysis repeatable for any model-training-data combination.

3.4.4 LIVEBENCH: Dynamic Benchmark Generation Pipeline

The third contribution is the most architecturally complex: a pipeline that automatically generates fresh evaluation questions from continuously-updating web content, creating a benchmark that is inherently contamination-resistant because the test data didn't exist when models were trained. The pipeline has five stages: data collection, information extraction, QA generation, quality control (checker + finalizer), and scoring.

Design philosophy. The paper is explicit about the tradeoff: "the quality of our QA may still fall below that of human-curated answers, as we are aiming to build a dynamic evaluation pipeline that strikes a balance between cost and broad coverage." This is the "low-cost, zero-contamination" corner of the trilemma — accepting some quality reduction in exchange for sustainability and timeliness.

Data Collection from the Web (Stage 1). The paper selects "over 60 news outlets" as source websites, listed in Appendix E.1 (Table 28). The sources span multiple categories:

General News: BBC, CNN, WSJ, Reuters, CCTV
Business/Finance: Bloomberg Economics/Industries/Technology/Politics/Opinion
Technology: Andreessen Horowitz, Hacker News, Reddit, Crunchbase News
Domain-specific: BBC Sport, BBC Innovation, BBC Culture, BBC Travel, WSJ Science (including sub-categories for Archaeology, Biology, Environment, Physics, Space)
Regional coverage: WSJ Africa, Americas, Asia, China, Europe, Middle East, India, Oceania, Russia, UK

The diversity of sources is intentional — it ensures that LIVEBENCH questions span multiple domains (politics, economics, technology, science, sports, culture), languages (the website list includes Japanese sources like Asahi Shimbun, Chinese sources like Xinhua and CCTV, and English sources), and cognitive demands (breaking news requires different reasoning than feature stories or financial data).

Information Extraction (Stage 2). This is the first of three stages powered by Claude-3.5-Sonnet, chosen presumably for its strong multimodal understanding capabilities. The process captures screenshots of news website homepages, then performs a three-step extraction via prompted LLM calls:

OCR extraction: The model extracts all text from the website screenshot. The prompt (Table 13) instructs: "Please extract the text from the website as detailed as possible. Only output the text extracted from the website, do not include any other information."
Image analysis: The model identifies "meaningful images in this screenshot" and extracts "relevant information about these images, such as the environment depicted, the actions and expressions of the people, and the connection between these images and the corresponding text." This step captures the multimodal nature of news — headlines are often paired with photographs that carry additional information beyond the text.
Newsworthiness identification: The model is asked to specify "what makes this website different from other websites? What is special about its news? Since it is a news website, where is the 'new' aspect reflected?" The prompt explicitly instructs against generalized answers: "Do not provide a generalized answer; you need to give detailed responses based on the specific content of each news article and the accompanying illustrations." This step identifies the temporally-specific elements that distinguish current news from stale information, which is critical for creating questions that test zero-shot generalization rather than general world knowledge.

The prompt also provides an example of the expected depth: "For example, if the news is about a software update, what conveniences will this update bring to people? How can people use these new features? Perhaps there are also some drawbacks? You need to come up with your own questions worth pondering about the website and describe in as much detail as possible your understanding of what is 'new' on the website."

QA Generation (Stage 3). The extracted information is fed to the "quiz model" (also Claude-3.5-Sonnet, though the paper doesn't explicitly confirm this; it's implied by the pipeline description) to generate question-answer pairs. The prompt (Table 9) instructs the model to act as "a quizmaster who designs questions based on a provided image that would challenge adults to think critically."

Four cognitive levels based on Bloom's Taxonomy. The question generation is structured around four categories of increasing cognitive demand, derived from Bloom's Taxonomy (Bloom et al., 1956):

Concrete Recognition (Comprehension and Remembering): Questions at this level require recalling facts and explaining concepts. Examples: "What are the key points in this news story?" and "How would you explain the main event reported here?" At this level, models may need OCR capabilities to extract information from screenshots and basic comprehension to summarize it.
Real-world Application: Questions require applying extracted knowledge to practical tasks. Examples: "Please present this news in Arabic and output it in markdown format," "Organize all the news on this page in the form of an HTML table, including the title, release time, and keywords," "Sort out the exchange rate data and plot them using the Julia language," and "Can you give me an example of this update in Python?" This category tests whether models can operationalize their understanding — transforming information from one format to another, generating code, or translating content.
Analytical Understanding: Questions require breaking down information to understand relationships and deeper meanings. Examples: "What are the factors that led to this event?" and "How does this event relate to other current issues?" This category probes causal reasoning and contextual understanding.
Divergent Thinking & Creation: The highest cognitive level, requiring generation of new ideas and synthesis. Examples: "How could you create a new headline that captures the essence of the event differently?" and "If you were the reporter, how would you approach this story to provide a unique angle?" This category explicitly tests creativity and perspective-taking.

Question format flexibility. The prompt encourages the quiz model to "try to be innovative" and produce "difficult questions, as well as multiple-choice questions, fill-in-the-blank questions, or even image-text matching questions, and sequencing questions." However, the finalizer stage (described below) may reformat these into more standardized forms.

Scoring criteria generation. The quiz model is also instructed to generate scoring criteria alongside each question: "Each question is scored as $0 \sim 10$ , and the correct answers should be scored as $10$ . Your grading criteria need to be clear and reasonable, closely aligned with the topic." These criteria are used later by the GPT-4o judge to assign scores to model responses.

Quality Control: Checker and Finalizer (Stage 4). After initial QA generation, two additional models refine and validate the questions:

The Checker model is responsible for "refining the questions and answers, restructuring them to ensure the questions are more answerable, verifiable, and challenging." Its prompt (Table 11) gives it specific instructions:

It must verify that the assigned category (Concrete Recognition, Analytical Questions, Divergent Thinking, Real-world Assistance) matches the question content. If it doesn't match, the checker must either "modify the question to correspond to the subtask" or "modify the subtask to correspond to the question" — but with a preference for modifying the question rather than the category: "If you feel the original question's subtask does not match the question, modify the question to match the subtask instead of rewriting the subtask."
It must ensure questions are "answerable, checkable, and challenging."
It must avoid political bias: "try to avoid questions with any political bias when asking questions. The question should focus on understanding and thinking about the image, not on political opinions."
If a question is fundamentally unanswerable or poor quality, the checker can "provide a new question and answer."

The Finalizer model handles formatting and scoring criteria refinement. Its prompt (Table 12) specifies eight requirements:

Criteria should be in "natural language" not dict/json format.
Bullet points, numbered lists, or YAML format are acceptable; Python-like format is not.
Answers should be in natural language if possible, not dict format (unless the question explicitly requires it).
Scoring criteria must be in English (though content can be in other languages).
Scoring criteria must be "rational and facilitate the accurate assessment of responses."
The full score must be exactly 10 points and "must directly relate to the specific answer."
The question must be "clear and unambiguous."
The answer must be "correct and reasonable."

The finalizer also handles edge cases: "For some extremely hard open-ended questions where answers may vary, hitting all points perfectly may not be realistic. In such cases, you can relax the criteria slightly. For example, if there are five possible points in an answer, but answering three adequately could merit full points." And critically: "DO NOT CHANGE the question to multiple-choice questions. If the original question is multiple-choice, you need to change it to another type of question." This suggests a preference for open-ended evaluation formats that test generation capabilities rather than recognition.

QA Scorer (Stage 4 continued). A separate scoring model evaluates each QA pair on three dimensions (Table 10):

Authenticity (5 points): Whether "the information is directly observable in the image or can be reasonably inferred with strong evidence."
Logical Coherence (3 points): Whether "the answer logically follows from the question and maintains consistency with the image context."
Clarity and Precision (2 points): Whether "the question and answer are clearly articulated and precisely address specifics of the image."

Each QA pair receives a score from 1 to 10 based on these criteria. The paper then collects "approximately 500 questions each month and selects 100 to 300 for the final LIVEBENCH problem set, based on those that exceed a certain score threshold." The specific threshold is not reported. Additionally, the authors "manually review the questions to remove any that are inappropriate," adding a human-in-the-loop quality filter.

Evaluation Metrics (Stage 5). Once the question set is finalized, models are evaluated using GPT-4o as the primary judge model. The evaluation prompt (Table 14) provides the judge with:

The original image
The question
The model's generated response
The ground truth answer
The question-specific scoring criteria

The judge assigns a score from 0 to 10 and provides an explanation (in JSON format). The per-question scores are then scaled to an accuracy metric ranging from 0 to 100 (presumably by dividing the raw score by 10 and multiplying by 100, though the exact formula isn't specified).

Multi-judge validation. The paper notes that "Claude-3.5-Sonnet and Gemini 1.5 Pro serve as alternative judge models" to validate the GPT-4o judgments, though the main reported results in Table 3 use GPT-4o. This addresses the concern that using an LLM as judge might introduce systematic biases (e.g., GPT-4o might prefer responses that match its own generation style).

Why this pipeline design? The five-stage pipeline with multiple quality control steps represents a deliberate engineering tradeoff. A simpler design — generate questions directly from webpages with a single LLM call — would be cheaper and faster, but would produce lower-quality questions (hallucinated facts, unanswerable questions, inconsistent scoring criteria). The checker and finalizer stages add cost but serve as quality filters, removing obviously flawed questions before they reach the final benchmark. The human review step adds further cost but catches edge cases that automated quality checks miss (inappropriate content, subtle factual errors). The monthly cadence (500 questions generated, 100-300 selected) balances freshness against quality control costs.

Why Bloom's Taxonomy? The four-level cognitive structure is not just a categorization convenience — it's a deliberate attempt to assess capabilities at different depths, from surface-level extraction to creative synthesis. This aligns with the paper's broader goal of comprehensive evaluation: a model that performs well on Concrete Recognition but poorly on Analytical Understanding has a specific and diagnosable weakness (inability to reason causally from visual-textual evidence) that would be invisible in a benchmark that only tested fact extraction.

Why news and forums? The choice of dynamically-updated news websites as the data source solves the contamination problem in a specific way: because the news is continuously updated, questions about events from September 2024 could not have been in models trained on data with a cutoff before that date. This is the "zero-contamination" guarantee — the test data literally didn't exist when the models were trained. Forums like Reddit and Hacker News add a different dimension: user-generated content that tests understanding of informal, conversational, and sometimes technically specialized information that news articles may not cover.

Example LIVEBENCH questions. Tables 21-24 provide representative examples of each question type, giving concrete illustrations of what the pipeline produces:

Concrete Recognition (Table 21): "Analyze the ongoing tennis matches displayed on the webpage, detailing the players involved, their current scores, and the tournaments they are part of." The ground truth is a detailed list of specific matches, scores, and tournaments. This tests OCR capability, information extraction, and structured output generation.
Real-world Application (Table 22): "Create an HTML table summarizing the improvements in 'ML Benchmarks' shown in the image. The table should include the benchmark names, gpt4o scores, and o1 scores. Ensure proper HTML structure and formatting, and include basic styling for better readability." This tests the ability to extract data from an image, transform it into a specific structured format (HTML with CSS styling), and produce syntactically correct code.
Analytical Understanding (Table 23): "Analyze the scene depicted in the image associated with the Haiti gang 'massacre' article. Describe the environment, the actions of the people, and the emotions conveyed. How do these elements support the narrative described in the article?" This tests visual interpretation, emotional inference, and the ability to connect visual evidence to textual narrative.
Divergent Thinking (Table 24): "Evaluate the potential impact of AI on the IT job market, considering both the negative and positive effects described in the provided image and text. Discuss the short-term and long-term impacts, and suggest strategies for IT professionals to adapt to these changes." This is substantially more open-ended, requiring synthesis of extracted information with external knowledge about economics and technology trends, plus generation of normative recommendations.

3.4.5 Score Aggregation for LMMS-EVAL LITE

A critical design detail that affects the usability of LMMS-EVAL LITE is how scores from different benchmarks are combined into a single summary signal. The paper recognizes that "since different datasets and benchmarks come up with their own metrics, it is not reasonable to simply calculate the average score." Direct averaging would be inappropriate because different benchmarks use different scales and metrics: accuracy is 0-100%, CIDEr can range from 0 to 10, ANLS is between 0 and 1, and F1 is also 0-100%. Averaging raw CIDEr (where 1.0 might be excellent) with raw accuracy (where 90% is excellent) would produce a meaningless composite.

The paper's solution is straightforward:

Per-dataset normalization: Each model's score on each dataset is normalized to a 0-100 range. For metrics that are naturally bounded (accuracy, F1), this is already in percentage form. For unbounded metrics (CIDEr), the normalization procedure is not explicitly detailed but can be inferred: the maximum possible or maximum observed score is used as the reference point, and all scores are expressed as a percentage of that maximum.
Simple average: The normalized scores are averaged to produce an aggregated score.

What this computes: a single number that summarizes overall model capability across the benchmark suite, with each benchmark contributing equally to the final score (after normalization). This is analogous to the practice in the HELM framework (Liang et al., 2022) and other holistic evaluation suites.

Why this form: the per-dataset normalization ensures that no single benchmark dominates the aggregate simply because its metric has a larger numeric range. Equal weighting across benchmarks is a neutral default — it doesn't privilege any particular capability dimension. The paper doesn't claim this weighting is optimal, but it provides a simple, interpretable aggregate that developers can use for quick comparisons during model development.

Validation. Figure 3 demonstrates the effectiveness of this aggregation by comparing the aggregated scores from the full LMMS-EVAL set with the aggregated scores from LMMS-EVAL LITE across multiple models. The visual alignment of the bars (full vs. lite) for each model confirms that the lite set preserves not just per-benchmark rankings but also the overall composite signal.

3.4.6 Evaluation Metrics and Answer Extraction Methods

The paper handles a diverse set of evaluation metrics that span classification, generation, and structured output tasks. While the metrics themselves are dataset-specific and pre-existing (not innovations of this paper), the standardization of their application is a key technical contribution.

Multiple-choice evaluation. For benchmarks with discrete answer options (MMMU, MMBench, SEED-Bench, ScienceQA, AI2D, MathVista), the framework supports two answer extraction methods:

Perplexity-based (PPL-based): The model's output logits for each answer option are compared, and the option with the lowest perplexity (or, equivalently, the highest probability) is chosen as the prediction. This doesn't require the model to actually generate the answer text — it only requires access to the logits. The paper references Li et al. (2023c) as an example of this approach.
Generation-based: The model generates free-form text, and the framework parses whether the generated text matches one of the answer options (typically by exact string match on the option letter, e.g., "(A)" or "A"). The paper references Liu et al. (2023a) as using this approach.

The paper takes a pragmatic stance on which method is "correct": "We believe there is no best setup but one needs to fix one when comparing results across different models." The key contribution is fixing one — all models are evaluated with the same extraction method, making scores comparable.

Captioning evaluation. For captioning benchmarks (COCO, Flickr30k, NoCaps, TextCaps), the metric is CIDEr (Consensus-based Image Description Evaluation), which compares generated captions against multiple reference captions using TF-IDF weighted n-gram matching. The framework handles the tokenization and scoring automatically.

Visual QA evaluation. For VQA benchmarks (VQAv2, GQA, OKVQA, TextVQA, VizWizVQA), the metric is typically accuracy or exact match — the generated answer is compared against one or more ground-truth answers, with fuzzy matching for numerical answers and case-insensitive matching for text answers.

Document understanding evaluation. For DocVQA, the metric is ANLS (Average Normalized Levenshtein Similarity), which computes the edit distance between the generated answer and the ground truth, normalized by the maximum of the two string lengths, then averaged across instances. This is more forgiving than exact match — minor OCR errors or formatting differences don't result in zero credit.

Referring expression evaluation. For RefCOCO/RefCOCO+/RefCOCOg, the evaluation involves generating bounding box coordinates or segmentation masks. The paper uses CIDEr as the evaluation metric for referring expressions, which is somewhat unusual (CIDEr is typically for captioning) and suggests that the referring expression tasks were framed as text generation tasks in this framework.

Hallucination detection evaluation. For POPE, the metric is F1 score (harmonic mean of precision and recall), reflecting the binary classification nature of the task (is the object present in the image? yes/no).

3.4.7 Model Integration and Inference Details

The framework's ability to support diverse model architectures is itself a technical achievement, given the heterogeneity in LMM design. Each model family has different requirements for image preprocessing, prompt formatting, and output generation:

LLaVA family: Uses CLIP-based vision encoders with a linear projector to the LLM embedding space. Requires specific image preprocessing (resolution, normalization) and uses chat templates based on the underlying LLM (Vicuna, Mistral, LLaMA-3, Yi, Qwen).

Qwen-VL: Uses its own vision encoder and supports variable image resolutions. The model has built-in support for bounding box generation via special tokens.

InstructBLIP: Uses a Q-Former architecture to bridge vision and language, with different preprocessing requirements than LLaVA.

InternVL: Scales vision foundation models and has specific requirements for dynamic resolution handling.

Idefics2: Based on the Idefics architecture with specific image token handling.

API-based models: GPT-4V, GPT-4o, Gemini, Claude, Qwen-VL-Max are accessed via HTTP APIs with image encoding (base64 or URL-based) and prompt construction following each provider's specifications.

The framework abstracts these differences behind a common model.generate(image, prompt) interface, where the implementation handles model-specific preprocessing. This is the standard design pattern for evaluation harnesses, but implementing it correctly for 10+ model families with 30+ variants requires substantial engineering effort.

Inference parallelism. Section 3 describes two parallelism strategies:

Data parallel (default): Model weights are replicated across all available GPUs, and evaluation instances are distributed among GPUs. Each GPU processes its assigned instances independently. This is efficient for models that fit on a single GPU (up to ~72B parameters with 8×A100-80GB).
Pipeline parallel: For models larger than 72B (LLaVA-NeXT-110B), the model is split across multiple GPUs, with each GPU holding a subset of layers. This uses the GPipe algorithm (Huang et al., 2019). The tradeoff is that pipeline parallelism introduces idle time (pipeline bubbles) and reduces throughput compared to data parallelism, but it enables evaluation of models that wouldn't fit on a single GPU.

Flash attention. All evaluations use Flash Attention, which reduces memory usage and speeds up attention computation — important for handling the long context lengths that multimodal models sometimes produce (descriptions of complex images, multi-step reasoning chains).

3.4.8 LIVEBENCH Case Analysis Protocol

The paper includes extensive qualitative analysis of LIVEBENCH results (Tables 15-20 in Appendix E.5), which demonstrates a systematic approach to understanding model failures:

Each case study presents: (1) the question, (2) a specific model's incorrect response, (3) the GPT-4o judge's score and rationale, and (4) GPT-4o's own correct response for comparison. This structured format allows the reader to directly compare model outputs and understand the nature of errors.

The case studies reveal specific failure modes:

OCR failure with non-English text: LLaVA-1.5-7B completely fails on Japanese news content, producing "repeated nonsense sentences" instead of actual headlines (Table 15).
Numerical extraction and arithmetic errors: LLaVA-NeXT-OV-72B-Chat misreads closing prices and performs incorrect arithmetic, while GPT-4o correctly extracts and averages Bitcoin price data (Table 20).
Entity hallucination: LLaVA-NeXT-OV-72B-Chat incorrectly matches tennis players to opponents and hallucinates player names (e.g., calling Qinwen Zheng "Qiang Wang") (Table 18).
Failure to follow instructions: LLaMA-3.2-Vision-11B-Instruct provides a detailed image description when asked to summarize news article content, indicating poor instruction following despite adequate visual understanding (Table 19).

These cases serve as qualitative validation that LIVEBENCH questions genuinely test capabilities that simpler benchmarks miss — specifically, the ability to handle unstructured, real-world visual-textual information with high precision.

3.4.9 Summary of Design Choices and Their Justifications

CLIP + BGE-M3 for embeddings (rather than a trained multimodal embedder): Simpler to deploy (no model training required), produces high-quality correlations (0.87-0.99), and the concatenation approach captures both visual and textual similarity without requiring a specialized multimodal embedding model.

Greedy k-center over k-means for coreset selection: The worst-case coverage guarantee of k-center is more appropriate than k-means' average-case optimization, because we care about preserving performance across all capability levels, not just the average case. Empirically, k-center outperforms k-means on some benchmarks (Flickr30k: 0.91 vs. 0.79).

SEED-tokenizer over embedding similarity for image overlap detection: The discrete tokenization avoids the need to set per-dataset similarity thresholds, making the approach more general and less sensitive to hyperparameter choices.

Claude-3.5-Sonnet for LIVEBENCH generation: The paper doesn't explicitly justify this choice (beyond presumably Claude's strong multimodal capabilities), but the multi-stage pipeline (generation → checking → finalizing) provides redundancy that reduces dependence on any single model's quality.

GPT-4o for LIVEBENCH evaluation: GPT-4o is chosen as the primary judge "due to its popularity and high-throughput API," with Claude and Gemini as alternatives for validation. This is a practical choice rather than a theoretically motivated one — GPT-4o is widely available and generally produces reasonable judgments.

Per-dataset normalization followed by equal-weight averaging for LMMS-EVAL LITE aggregation: Simple, interpretable, doesn't privilege any capability dimension. The paper doesn't claim optimality, only practical utility for rapid development feedback.

Full retention of MME (no pruning) despite size: The paper found low correlation between MME full and lite scores, so it retains the full 2,374-instance set rather than risk misleading evaluation signals. This demonstrates a principled empirical approach — validate before pruning, and don't prune when validation fails.

4. Key Insights and Innovations

Innovation 1: The Evaluation Trilemma as an Explicit Organizing Principle

The paper's most conceptually distinctive contribution is the articulation and operationalization of the evaluation trilemma — the claim that wide coverage, low cost, and zero contamination cannot be simultaneously achieved in any single benchmark for LMMs. While individual tensions between these desiderata have been implicitly recognized before (the Hugging Face OpenLLM leaderboard is known to be cheap but contamination-prone; Chatbot Arena is uncontaminated but expensive), the paper is the first to name this as a structural impossibility, characterize the tradeoffs explicitly, and build a coordinated toolkit that addresses each side of the triangle through separate, complementary instruments rather than attempting to optimize all three simultaneously.

This framing represents a fundamental conceptual shift from how the field has approached benchmark design. Prior work in LMM evaluation — whether static multi-task suites like MMBench and MMMU, human-preference arenas like WildVision, or lite benchmark efforts like those of Perlitz et al. (2024) and Polo et al. (2024) — implicitly assumed that a single evaluation approach could (or should) serve multiple purposes. MMBench aims to be both comprehensive and practical. Chatbot Arena trades cost for authenticity. The paper's trilemma framing makes explicit what was previously latent tension: these tradeoffs are not engineering limitations to be solved with better design, but fundamental constraints arising from the relationship between test data, training data, and evaluation cost. You cannot have a fixed test set that is simultaneously comprehensive (requiring many diverse instances), cheap to evaluate (requiring few instances), and uncontaminated (requiring data that didn't exist during training). The tension is structural.

What makes this more than a taxonomy is its operational payoff: recognizing the trilemma leads directly to the paper's architectural decision to build three separate tools rather than one unified benchmark. Each tool optimizes a different pair of desiderata while accepting weakness on the third — LMMS-EVAL LITE for wide coverage + low cost (accepting contamination), LIVEBENCH for zero contamination + low cost (accepting narrower coverage), and the full LMMS-EVAL for wide coverage + zero contamination (accepting high cost as the necessary price of comprehensiveness). This is a design philosophy that generalizes beyond this specific paper: future evaluation efforts can explicitly position themselves within the trilemma, stating which tradeoffs they accept, rather than claiming or implying a nonexistent Pareto-optimal solution. In this sense, the trilemma functions as a diagnostic framework for the entire evaluation landscape, not merely a justification for the paper's own contributions.

The paper's evidence for the trilemma is distributed across its empirical findings: Figure 4 demonstrates that contamination is widespread (justifying why "wide coverage + low cost" is problematic in practice, not just theory), Figure 2 shows the computational cost of comprehensive evaluation (justifying why "wide coverage + zero contamination" is expensive), and the LIVEBENCH results in Table 3 show that dynamic evaluation reveals capability gaps invisible in static benchmarks (justifying why "low cost + zero contamination" has value even if coverage is narrower). The trilemma is not asserted axiomatically — it is demonstrated through the systematic documentation of each constraint.

This contribution is incremental in its components but fundamental in its synthesis. The individual observations about cost, contamination, and coverage are not new in themselves. What is new is the unified framing that reveals their interaction as an impossibility result, and the architectural response that treats the trilemma as a design constraint to be navigated rather than a problem to be solved.

Innovation 2: SEED-Tokenizer-Based Image Contamination Detection as a Tunable-Free Method

The paper's second distinctive contribution is a methodological innovation in contamination detection: using the SEED-tokenizer's discrete visual token sequences to detect image overlap through n-gram matching, bypassing the threshold-tuning problem that plagues embedding-similarity-based approaches. This is a novel combination of two existing techniques — SEED-tokenization (Ge et al., 2023) and n-gram overlap analysis (Brown et al., 2020) — applied to a specific bottleneck in multimodal evaluation that the field had not systematically addressed. While text-only contamination detection through n-gram overlap is well-established in the LLM literature, and image similarity detection through embedding cosine similarity is standard in computer vision, the paper identifies and solves a specific failure mode at their intersection: "selecting an appropriate threshold applicable to all datasets is difficult" when using continuous similarity scores.

The key insight is that discretization removes a hyperparameter. By converting continuous image representations into 32 discrete tokens, the SEED-tokenizer transforms the image overlap problem from "how similar do two images need to be to count as contaminated?" (requiring a similarity threshold) into "do any 8-token sequences match between training and test images?" (a binary decision with no threshold). This is not merely an engineering convenience — it addresses a genuine scientific problem. Different benchmark datasets have different visual diversity: an embedding similarity of 0.8 might indicate genuine contamination in a chart dataset (where visual structures are highly constrained) but might indicate normal visual variation in a natural image dataset. A per-dataset threshold approach would require calibration on held-out contamination data that generally doesn't exist (since contamination is exactly what we're trying to detect). The SEED-tokenizer approach achieves dataset-independent detection through the structure of the tokenization itself — similar image regions map to similar token sequences, dissimilar regions diverge, and the 8-gram length provides a natural granularity for detecting partial image overlap.

The paper's evidence for this method's effectiveness is both quantitative and qualitative. Table 4 provides systematic contamination statistics across 21 benchmarks, revealing that ChartQA has 68.64% image overlap with LLaVA training data — a finding that would be difficult to establish robustly with embedding similarity alone, given ChartQA's domain-specific visual style. Figure 5 provides qualitative examples of three contamination categories (duplicate images, similar images, similar questions) that were identified by the tokenizer-based approach, demonstrating that the method surfaces both obvious and subtle contamination patterns. The paper's finding that POPE has 42.20% image overlap despite 0.00% text overlap (because "POPE uses image sources from COCO," which is in the training data) is a specific insight that would be difficult to capture with text-only analysis and demonstrates the value of multimodal contamination detection.

This contribution is fundamental in problem identification but incremental in solution technique. The problem — image contamination in LMM benchmarks has been largely unexamined — is genuinely important and under-addressed. The solution — repurposing SEED-tokenizer for overlap detection — is a creative synthesis of existing tools rather than a new method from scratch. The contribution's significance lies less in the technical novelty of the detection approach and more in its demonstration that contamination is pervasive and systematically biased in the LMM evaluation landscape. The numbers in Figure 4 — ChartQA at 68.64% image overlap, VQAv2 at 46.21%, COCO at 46.05% — are the real payload. They establish that researchers evaluating LMMs on these benchmarks without contamination correction are measuring, in part, training data memorization rather than genuine visual reasoning. The methodological contribution enables this empirical finding, but the finding itself is what changes how the field should interpret existing benchmark results.

Innovation 3: Coreset Selection for Multimodal Benchmark Pruning via Joint Visual-Textual Embedding

The paper's third distinctive contribution is the application of k-center coreset selection to multimodal benchmark evaluation, using a joint CLIP + BGE-M3 embedding space to represent image-text pairs and selecting representative subsets that preserve both absolute scores and relative model rankings. While coreset selection for efficient evaluation has been explored in the text-only LLM space — Perlitz et al. (2024) use stratified random sampling, Vivek et al. (2024) use anchor points, Polo et al. (2024) use Item Response Theory — the paper is among the first (and to the authors' knowledge, the first systematically validated application) to extend this approach to multimodal benchmarks where both visual and textual features must be jointly considered for representativeness.

The conceptual contribution is the recognition that multimodal benchmark instances are not just text-with-pictures — they are joint entities where the interaction between visual content and textual question determines the evaluation signal. A naive approach would embed only the text (missing that two questions with similar wording but different images test different capabilities) or only the image (missing that two similar images with different questions test different skills). The paper's solution — concatenating independently-sourced CLIP image embeddings and BGE-M3 text embeddings — is straightforward in retrospect but addresses the core challenge: the embedding space must capture both what the model sees (visual content) and what it's asked to do (the textual task), because the evaluation signal depends on their combination.

The evidence for this approach's effectiveness is the correlation validation in Table 2 and Table 5. The key finding is not just that the lite set correlates with the full set (that would be true for any reasonable pruning method) but that the k-center approach achieves correlations above 0.90 on 12 out of 15 benchmarks and outperforms both k-means (on Flickr30k: 0.91 vs. 0.79) and Quire (on AI2D: 0.98 vs. 0.45). The failure of Quire — an active learning method designed for selecting informative examples — is particularly instructive. It suggests that representativeness, not informativeness, is the right criterion for benchmark pruning: the goal is not to find the hardest or most diagnostic instances (which Quire targets), but to find instances that collectively span the distribution of difficulty levels and capability types present in the full set. This is a non-obvious insight that matters for anyone building lite benchmarks.

Figure 3 provides the complementary validation: aggregated scores from LMMS-EVAL LITE closely track aggregated scores from the full LMMS-EVAL across multiple models. The fact that the lite set preserves not just per-benchmark rankings but also the overall composite signal means it can serve as a drop-in replacement for full evaluation during model development — precisely the use case the paper targets ("provide useful and low-cost signals during model training and ablations").

This contribution is incremental in technique but important in validation scope. The k-center algorithm is standard; its application to CLIP+BGE embeddings is a natural extension. What makes the contribution distinctive is the comprehensiveness of the validation: 15 benchmarks spanning chart understanding, document QA, captioning, referring expressions, visual QA, and multi-task suites, with correlations validated across six model variants. This systematic demonstration that coreset selection works across diverse multimodal evaluation contexts — rather than on one or two cherry-picked benchmarks — is what transforms the approach from an interesting idea to a reliable tool. The negative result on MME (where "low correlation between the original and lite set scores" led to retaining the full set) further strengthens credibility: the paper doesn't claim universal applicability and is transparent about where the method fails.

Innovation 4: The LIVEBENCH Pipeline as a Practical Countermeasure Against Contamination Through Temporal Freshness

The fourth contribution is the design and deployment of a fully automated, continuously-updating benchmark generation pipeline that achieves contamination resistance not through better detection or filtering, but through temporal exclusion: by sourcing test questions from news and forum content published after model training cutoffs, the benchmark guarantees that test data could not have been in training corpora regardless of how aggressive the data collection was. This is a fundamentally different approach to contamination than prior work — rather than trying to measure, remove, or correct for contamination in existing benchmarks (Brown et al., 2020; Shi et al., 2024; Yang et al., 2023a), LIVEBENCH makes contamination structurally impossible through the temporal relationship between training data and test data.

The conceptual move is recognizing that contamination resistance is not a property of the benchmark content but of the benchmark's relationship to model training timelines. A static benchmark can be contaminated even if its content is perfectly original, because models can be inadvertently trained on it. A dynamic benchmark sourcing content from websites that are continuously updated ensures that any given test question references events that occurred after the training data was collected. This shifts the contamination problem from "how do we keep test data out of training data?" (a data governance challenge that is increasingly impossible as training corpora scale) to "how do we ensure test data postdates training data?" (a temporal constraint that is enforceable through benchmark design).

The paper's evidence for LIVEBENCH's value comes from Table 3, which reveals a specific and important pattern: GPT-4o achieves 92.0 overall accuracy on LIVEBENCH-2024-09, while the best open-source model (Qwen2-VL-72B) achieves 85.9 — a 6-percentage-point gap. On Concrete Recognition, the gap is even larger: GPT-4o scores 91.7 vs. Qwen2-VL-72B's 86.7. The paper explicitly contrasts this with static benchmarks where "open-source models often outperform commercial ones" and argues that LIVEBENCH "requires models to demonstrate strong zero-shot generalization abilities, as they must interpret continuously updated content from news and forum websites, highlighting the unique advantages of these commercial models." This is not merely a "GPT-4o is better" finding — it is evidence that existing static benchmarks systematically underrepresent the real-world generalization capabilities that commercial models possess and that open-source models lack. If static benchmarks show open-source models outperforming commercial ones, but dynamic benchmarks show the reverse, then static benchmarks are not just noisy — they are systematically biased in a specific direction, likely due to contamination and the simplicity of fixed-format evaluation.

The four-category cognitive hierarchy based on Bloom's Taxonomy is an additional conceptual contribution embedded within the pipeline design. By structuring questions at four levels (Concrete Recognition, Real-world Application, Analytical Understanding, Divergent Thinking & Creation), LIVEBENCH provides not just a single accuracy number but a capability profile that reveals whether a model's strength is in surface-level extraction or deeper reasoning. Table 3 shows that models differ in their relative strengths across categories: GPT-4o is strong across all four (91.7, 93.8, 94.8, 87.6), while LLaVA-OV-72B-Chat is notably weaker at Concrete Recognition (62.0) than at Analytical Understanding (87.8) — an inversion that might reflect OCR limitations rather than comprehension failures. This diagnostic granularity is built into the evaluation structure itself, not added as post-hoc analysis.

This contribution is fundamental as a design paradigm but has clear practical limitations. The paradigm — use temporal freshness as the contamination countermeasure, accept automated question generation as a cost-quality tradeoff — is genuinely novel in the LMM evaluation space and potentially generalizable to other domains where continuously-updating data sources exist (financial reports, sports statistics, scientific publications, legal filings). The limitations are equally important: the pipeline's dependency on Claude-3.5-Sonnet for question generation creates a potential bias (questions are generated by a commercial model that may have specific strengths or blind spots), the quality of automatically-generated questions "may still fall below that of human-curated answers," and the scope is necessarily limited to domains with frequent, structured updates (news, forums) rather than more stable capability areas (scientific reasoning, mathematical problem-solving). The paper is transparent about these tradeoffs, framing LIVEBENCH as a complement to, not replacement for, static benchmarks.

Innovation 5: Empirical Evidence That Static Benchmarks Systematically Overstate Open-Source LMM Capabilities Relative to Real-World Performance

The paper's fifth contribution — more an empirical finding than a methodological innovation — is the systematic demonstration that the relationship between static benchmark performance and real-world generalization is not monotonic across model types. Specifically, the LIVEBENCH results in Table 3 combined with the contamination analysis in Figure 4 provide converging evidence that commercial models like GPT-4o and Claude-3.5-Sonnet possess generalization capabilities that existing static benchmarks fail to measure, while open-source models' benchmark scores are partially inflated by data contamination and task-specific optimization.

This finding is significant because it invalidates a common interpretation of benchmark results in the LMM community. If LLaVA-NeXT-110B achieves higher scores than GPT-4V on MME (a claim the paper implies is plausible based on published results), the natural interpretation is that LLaVA-NeXT-110B is the more capable model overall. But if the same model substantially underperforms on LIVEBENCH questions that require interpreting novel, unstructured web content, then the MME score is measuring something different than "general visual-linguistic capability" — it may be measuring memorization of benchmark-specific patterns, optimization against known evaluation formats, or contamination-amplified performance on frequently-tested concepts. The paper provides evidence for all three mechanisms: the contamination analysis (Section 4.1) shows direct data overlap, the evaluation pipeline inconsistencies (Section 2.1) suggest that different answer extraction methods can advantage different models, and the LIVEBENCH case studies (Tables 15-20) reveal specific failure modes (OCR with non-English text, numerical extraction errors, entity hallucination) that don't appear in structured benchmarks.

What makes this more than a "GPT-4o is better" claim is the diagnostic structure it enables. The paper's framework allows researchers to ask not just "which model is better?" but "in what ways is a model's benchmark performance misleading about its real-world capability?" The four-category LIVEBENCH breakdown provides a template for this analysis: a model that performs well on Concrete Recognition but poorly on Analytical Understanding may have strong OCR but weak reasoning; a model that excels at Divergent Thinking but struggles on Real-world Application may be creative but imprecise. The contamination analysis adds another dimension: a benchmark with high image overlap (ChartQA: 68.64%) is measuring more memorization than a benchmark with low overlap (MMBench: 2.77%). Together, these diagnostics allow for disaggregated capability assessment — understanding not just whether a model is "good" but which specific capabilities it genuinely possesses versus which benchmark scores are artifacts of training data exposure.

The paper's score normalization and aggregation methodology for LMMS-EVAL LITE (Section 3) is relevant here as an operational enabler of this insight. By providing a lightweight way to track per-dataset performance across many models, the lite benchmark makes it feasible to identify cases where a model's performance pattern is anomalous — e.g., a model that performs exceptionally well on one benchmark but average on others might be contaminated on that specific benchmark, and this would be visible in the lite evaluation signals without requiring a full 50-task evaluation run.

This contribution is fundamental as an empirical corrective to the field's evaluation practices, even though it is presented as an observation rather than a formal theorem. The combination of contamination data and LIVEBENCH results constitutes a reality check — the paper's title is itself the insight. The field has been operating on the assumption that static benchmark rankings reflect genuine capability differences. The paper demonstrates, with systematic evidence, that this assumption is violated in specific, quantifiable ways. This does not mean static benchmarks are useless — the paper uses them extensively — but it means their interpretation requires contamination awareness, benchmark-specific skepticism, and complementary evaluation through contamination-resistant methods like LIVEBENCH. The insight is not that one evaluation method should replace another, but that no single evaluation method provides a complete or unbiased picture, and that triangulating across methods with different failure modes is necessary for reliable capability assessment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses the LMMS-EVAL benchmark suite, which encompasses over 50 tasks spanning diverse domains including science diagrams (AI2D, 3,088 test instances), chart understanding (ChartQA, 2,500 test instances), document QA (DocVQA, 5,349 validation instances; InfoVQA, 2,801 validation instances), captioning (Flickr30k, 31,784 test instances; COCO, 40,504 validation instances; NoCaps, 4,500 validation instances; TextCaps, 3,166 validation instances), visual QA (TextVQA, 5,000 validation instances; GQA, 12,578 test instances; VQAv2, 214,354 validation instances; OKVQA, 5,046 validation instances; VizWizVQA, 4,319 validation instances), referring expressions (RefCOCO, RefCOCO+, RefCOCOg with multiple splits totaling over 30,000 instances), multi-task benchmarks (MMBench with en-dev and cn-dev splits totaling ~8,700 instances; MME with 2,374 test instances; MMMU with 900 validation instances; CMMMU with 900 validation instances; SEED-Bench with 17,990 test instances; SEED-Bench-2 with 24,371 test instances; MM-Vet with 218 test instances), math and science reasoning (MathVista with 1,000 testmini instances; ScienceQA with 4,241 test instances), hallucination detection (POPE with 9,000 test instances), and others (Table 25 for full listing). LIVEBENCH uses a dynamically-generated dataset of 100–300 questions per month sourced from over 60 news websites and forums (Table 28), with the September 2024 set as the primary reported evaluation. The contamination analysis in Section 4.1 examines overlap between benchmark test data and LLaVA-NeXT training data across over 20 benchmarks.
Base model(s). The LMMS-EVAL framework supports over 10 model families with approximately 30 variants, spanning both open-source and API-based models (Table 26). Open-source models include LLaVA-1.5 (7B, 13B), LLaVA-NeXT (Vicuna-7B/13B, Mistral-7B, LLaMA3-8B, Yi-34B, Qwen-72B/110B), LLaVA-OV (0.5B, 7B, 72B, with and without SI variants), InstructBLIP (Vicuna-7B/13B), Qwen-VL-Chat (7B), Fuyu-8B, Idefics2-8B, MiniCPM-V-2.8B, XComposer-4KHD (8B), and InternVL-1.5 (26B). API-based models include GPT-4V, GPT-4o, Gemini-1.0-Pro, Gemini-1.5-Flash, Gemini-1.5-Pro, Claude-3-Haiku/Sonnet/Opus, Claude-3.5-Sonnet, Qwen-VL-Plus, and Qwen-VL-Max. The model selection spans 0.5B to 110B parameters, covering architectures from multiple research groups (LLaMA-based, Vicuna-based, internally-developed) to ensure the framework's generality. For LIVEBENCH (Table 3), 14 models are evaluated including GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, Gemini-1.5-Pro/Flash, Qwen2-VL-7B/72B, InternVL2-8B, LLaMA-3.2-V-11B-Instruct, LLaVA-OV-0.5B/7B/7B-Chat/72B-Chat, and LLaVA-1.5-7B.
Metrics. LMMS-EVAL uses dataset-specific metrics aligned with prior work: accuracy for multiple-choice benchmarks (MMMU, MMBench, SEED-Bench, ScienceQA, AI2D, MathVista, RealWorldQA); CIDEr for captioning benchmarks (COCO, Flickr30k, NoCaps, TextCaps); ANLS (Average Normalized Levenshtein Similarity) for document understanding (DocVQA); exact match or accuracy for visual QA (TextVQA, GQA, VQAv2, OKVQA, VizWizVQA); F1 score for hallucination detection (POPE); and dataset-specific metrics for referring expressions and other tasks (Table 25). For LIVEBENCH, GPT-4o serves as the primary judge model, assigning scores from 1 to 10 per question based on pre-generated scoring criteria and ground-truth answers, with Claude-3.5-Sonnet and Gemini 1.5 Pro as alternative judges for validation. The final LIVEBENCH scores are scaled to 0–100 accuracy. For LMMS-EVAL LITE aggregation, per-dataset scores are first normalized to a 0–100 range, then averaged with equal weighting to produce a single composite signal.
Baselines. The paper compares its evaluation methodology against multiple baseline approaches: (1) Per-model, per-benchmark custom evaluation scripts (Section 2.1) as the status quo ante, characterized as producing non-comparable results due to inconsistent data preprocessing, output parsing, and metric calculation. (2) Quire (Huang et al., 2010) and k-means (Lloyd, 1982) as alternative coreset selection methods for lite benchmark construction (Table 2). (3) Human-preference arenas (LMSys Chatbot Arena by Chiang et al., 2024; AI2 WildVision by Lu et al., 2024b) as alternative evaluation paradigms that prioritize zero contamination and wide coverage but at high cost (Section 2.2). (4) Existing static benchmarks (MME, MMBench, MMMU, etc.) as the standard against which LIVEBENCH's dynamic approach is compared, with the explicit observation that "open-source models often outperform commercial ones like GPT-4V in benchmarks, they fall short in real user experience" (Section 4.2). (5) For the contamination analysis, the LLaVA-NeXT training data (Liu et al., 2023a) serves as the reference corpus against which benchmark overlap is measured (Figure 4, Table 4).
Generation budget / compute accounting. For LMMS-EVAL, compute is measured in wall-clock evaluation time on 8×A100 GPUs with flash attention enabled (Figure 2). The paper distinguishes between data parallel inference (model weights replicated across GPUs, used for models up to ~72B parameters) and pipeline parallelism (model layers split across GPUs using Huang et al., 2019, used for models larger than 72B). For LMMS-EVAL LITE, efficiency is measured by the reduction in evaluation instances — the lite set contains 9,134 instances compared to 90,223 in the full set, representing roughly a 10× reduction (Table 6). For LIVEBENCH, cost is amortized through automated pipeline design: approximately 500 questions are generated monthly using Claude-3.5-Sonnet, with 100–300 selected after quality filtering and manual review (Section 4.2.1). The evaluation cost per model on LIVEBENCH is the GPT-4o API cost for judging responses plus model inference cost on the selected question set.
Cross-validation / statistical protocol. For LMMS-EVAL, the paper standardizes evaluation by fixing a single protocol — same data preprocessing, same output parsing method, same metrics — across all models to ensure comparability, following the principle that "one needs to fix one when comparing results across different models" (Section 2.1). For LMMS-EVAL LITE validation, the paper evaluates "six versions of LLaVA" (Liu et al., 2023a) on both the full benchmarks and the lite subsets, then computes correlation coefficients between full-set and lite-set scores (Table 2, Table 5). The paper explicitly notes that the evaluation serves as an experimental validation rather than a train-test split — the same models are used both for selecting representative points (via the embedding space) and for validating score alignment, which is appropriate since the selection method (k-center clustering) does not use model performance scores during subset construction. For LIVEBENCH, the evaluation uses GPT-4o as the primary judge with Claude-3.5-Sonnet and Gemini 1.5 Pro as alternative judges (Appendix E.3), and the paper includes qualitative case analyses (Tables 15-20) showing judge rationales alongside ground-truth comparisons. The contamination analysis uses an 8-gram overlap detection approach with a meaningless n-gram filter (excluding n-grams appearing more than 10 times in training data) and no statistical significance testing.

Main Quantitative Results

LMMS-EVAL Standardized Benchmark Results

Table 1 presents selected results across 8 benchmarks for 22 model variants evaluated through the unified LMMS-EVAL pipeline. Key findings include:

LLaVA-OV-72B (SI) achieves the highest scores among open-source models on most benchmarks, reaching 85.1 on AI2D, 84.9 on ChartQA, 93.5 on DocVQA, 93.7 on LLaVAW, 66.5 on MathVista, 2269.0 on MME, 57.4 on MMMU, and 73.8 on RealworldQA. The SI variant (with SI likely indicating a specific training or inference configuration) generally matches or slightly exceeds the non-SI variant.
Model scale consistently improves performance within the LLaVA family. Moving from LLaVA-OV-0.5B to LLaVA-OV-7B to LLaVA-OV-72B, MathVista scores increase from 34.8 → 63.2 → 67.5, MMMU scores from 31.4 → 48.8 → 56.8, and MME scores from 1478.0 → 1998.0 → 2261.0. The gains are largest between 0.5B and 7B, with diminishing returns from 7B to 72B on some benchmarks (e.g., AI2D: 81.4 → 85.6 for OV-7B vs. OV-72B, only a 4.2-point gain).
Instruction tuning and architecture choices matter significantly at similar parameter counts. At the 7–8B scale, models vary dramatically: Xcomposer4K-HD (8B) achieves 57.3 on MathVista and 2189.8 on MME, while InstructBLIP-Vicuna-7B achieves only 23.4 and 1508.7 respectively. On ChartQA, Xcomposer4K-HD reaches 80.6 while InstructBLIP-Vicuna-7B reaches only 12.5, a 68-point gap that demonstrates how model architecture and training data composition can dominate parameter count in determining capability.
The full results in Appendix F.1 (Table 27) extend coverage to additional benchmarks, showing LLaVA-NeXT-34B reaching 83.98 ANLS on DocVQA, LLaVA-1.5-13B achieving 78.26 accuracy on VQAv2, and LLaVA-NeXT-Mistral-7B reaching 72.16 ANLS on DocVQA (substantially outperforming LLaVA-1.5-7B at 28.08). These results demonstrate the value of the unified framework: without standardized evaluation, comparing a LLaVA-1.5 model evaluated with one pipeline against a LLaVA-NeXT model evaluated with another would confound model differences with methodological differences.

Contamination Analysis Results

Figure 4 and Table 4 present the systematic contamination audit across over 20 benchmarks against LLaVA-NeXT training data. The headline findings are:

Severe contamination in widely-used benchmarks: ChartQA shows 68.64% image overlap and 26.52% text overlap — the highest of any benchmark. COCO2014 val shows 46.05% image overlap and 22.19% text overlap. VQAv2 shows 46.21% image overlap (2.90% text). DocVQA shows 36.08% image overlap (4.06% text). These numbers mean that nearly half of the test images in these benchmarks have near-duplicate versions in the training data.
Text contamination can be high even with low image overlap: AI2D shows only 6.09% image overlap but 25.97% text overlap, indicating that question text and answer structures are being memorized even when the specific diagram images are novel. Similarly, NoCaps has only 2.53% image overlap but 19.98% text overlap, and SEED-Bench has 1.11% image overlap but 13.84% text overlap.
Most benchmarks maintain relatively low contamination: CMMMU (2.89% image, 1.11% text), MMBench (2.77% image, 0.81–7.97% text), MME (1.60% image, 1.39% text), MMMU (2.67% image, 3.56% text), MathVista (9.90% image, 7.70% text), and ScienceQA (0.35% image, 1.54% text) all fall below 10% on both dimensions, suggesting these benchmarks are substantially less compromised.
POPE's contamination is structural: At 42.20% image overlap but 0.00% text overlap, the contamination arises because "POPE uses image sources from COCO" (Appendix C) — the images are contaminated through their source dataset even though POPE's specific question format is novel. This demonstrates how contamination can propagate through benchmark construction: a new benchmark built on an existing image dataset inherits that dataset's contamination, even if the questions are newly written.
Three contamination categories identified qualitatively (Figure 5, Figure 7): (1) Duplicate images — identical images in both training and benchmark, found in ChartQA and MM-Vet. (2) Similar images — visually near-identical images detected through shared SEED-tokenizer 8-grams, found in NoCaps, ChartQA, and MM-Vet. (3) Similar questions — recurring question structures that "though not necessarily contamination or overlapping cases... are both testing similar domain knowledge and may help the model to answer questions in the benchmarks," found in MathVista.

LMMS-EVAL LITE Validation Results

Figure 3 demonstrates the aggregate score alignment between the full LMMS-EVAL and LMMS-EVAL LITE across multiple LLaVA model variants. The visual comparison shows that the lite set preserves the relative ordering of models and their approximate absolute scores, with the weighted average percentage scores on the x-axis showing strong correspondence between full (blue) and lite (red) bars for each model.

Table 2 reports the per-benchmark correlation between full-set scores and lite-set scores, comparing the paper's k-center approach against Quire and k-means baselines:

k-center (Lite) achieves correlations above 0.87 on all reported benchmarks: AI2D (0.98), Flickr30k (0.91), SeedBench (0.87), TextVQA (0.99). The high TextVQA correlation (0.99) indicates near-perfect preservation of model rankings, while the lower SeedBench correlation (0.87) suggests that some capability dimensions in SeedBench are more sensitive to instance selection.
k-center substantially outperforms Quire on several benchmarks: On AI2D, k-center achieves 0.98 while Quire achieves 0.45; on SeedBench, 0.87 vs. 0.27; on Flickr30k, 0.91 vs. 0.97 (Quire slightly better here). The catastrophic failure of Quire on AI2D and SeedBench indicates that informativeness-based selection — Quire's design principle — is inappropriate for benchmark pruning, because the most "informative" instances may be outliers that don't represent typical evaluation behavior.
k-center modestly outperforms k-means on Flickr30k (0.91 vs. 0.79) but performs similarly on other benchmarks. The advantage on Flickr30k suggests that worst-case coverage (k-center's guarantee) matters more than average reconstruction (k-means' objective) for datasets with high visual diversity.

Table 5 in Appendix D.1 provides the complete correlation results across all 15 lite benchmarks using both LLaVA embeddings and CLIP+BGE embeddings. With CLIP+BGE embeddings, 12 of 15 benchmarks achieve correlations above 0.90, with the lowest being SeedBench at 0.87 and InfoVQA at 0.94. The LLaVA embedding approach achieves similarly high correlations on most benchmarks but slightly lower on some (SeedBench: 0.77 vs. 0.87). The paper notes that for MME, "due to low correlation between the original and lite set scores, we retain the full version" — the exact correlation is not reported, but this negative result demonstrates principled methodology: validate before pruning, and don't prune when validation fails.

Figure 2 provides the cost motivation for LMMS-EVAL LITE by visualizing evaluation time on 8×A100 GPUs with flash attention. The comparison between full-set and lite-set evaluation times across LLaVA model variants demonstrates the computational savings achieved by reducing evaluation instances from 90,223 to 9,134 — roughly a 10× reduction. The exact time values are not numerically specified in the text, but the figure's bar chart format makes the relative savings visually apparent.

LIVEBENCH Evaluation Results

Table 3 presents the LIVEBENCH-2024-09 results for 14 models across four cognitive categories (Concrete Recognition, Analytical Understanding, Divergent Thinking & Creation, Real-world Application) and an Overall score:

GPT-4o achieves the highest overall accuracy at 92.0, followed by GPT-4o-mini at 91.9, Claude-3.5-Sonnet at 90.3, and Qwen2-VL-72B at 85.9. The gap between the best commercial model (GPT-4o) and the best open-source model (Qwen2-VL-72B) is 6.1 percentage points.
The Concrete Recognition category shows the largest gap between commercial and open-source models. GPT-4o scores 91.7 while the best open-source model (Qwen2-VL-72B) scores 86.7, a 5.0-point gap. The weakest open-source model (LLaVA-1.5-7B) scores only 9.4, indicating near-total failure on concrete recognition tasks — these likely involve OCR-heavy questions where LLaVA-1.5-7B's text recognition capabilities are insufficient.
Some open-source models show surprising category-specific strengths and weaknesses. LLaVA-OV-72B-Chat scores 62.0 on Concrete Recognition (relatively weak) but 87.8 on Analytical Understanding (competitive with commercial models) — an inversion that suggests strong reasoning capabilities coupled with weaker OCR or information extraction. LLaMA-3.2-V-11B-Instruct shows a different pattern: 51.9 on Concrete Recognition (below average) but 74.7 on Real-world Application (above many larger models), suggesting strength in following instructions for practical tasks.
Model scale improves performance monotonically within the LLaVA-OV family, from 0.5B (32.4 overall) to 7B (64.9) to 72B (75.0). The 0.5B model already achieves 25.1 on Concrete Recognition — substantially better than LLaVA-1.5-7B's 9.4, suggesting that the LLaVA-OV architecture provides better out-of-the-box OCR capabilities even at very small scales.
Gemini models occupy the middle tier between open-source and GPT-4o, with Gemini-1.5-Pro at 84.5 overall and Gemini-1.5-Flash at 81.6. Both outperform the best open-source model (Qwen2-VL-72B at 85.9) on Concrete Recognition (85.4 and 77.1 vs. 86.7) but the differences are narrower than the gap to GPT-4o.
The Divergent Thinking & Creation category shows the most compressed range among the top models, with GPT-4o (94.8), Claude-3.5-Sonnet (95.3), GPT-4o-mini (95.3), and Qwen2-VL-72B (89.0) clustering tightly. LLaVA-OV-72B-Chat drops to 83.8, and LLaVA-1.5-7B recovers somewhat from its Concrete Recognition failure to 45.4. This category may be more forgiving of OCR errors since it emphasizes creative synthesis over precise extraction.

The case studies in Tables 15–20 provide qualitative validation of these quantitative patterns. Table 15 shows LLaVA-1.5-7B producing "repeated nonsense sentences" when asked to sort Japanese news headlines, receiving a GPT-4o score of 0. Table 20 shows LLaVA-NeXT-OV-72B-Chat making arithmetic errors when calculating Bitcoin average closing prices — it misidentifies the current price (62242) as a closing price and produces an average of 15,556.8 instead of the correct 61,754. Table 18 shows LLaVA-NeXT-OV-72B-Chat hallucinating tennis player names (calling Qinwen Zheng "Qiang Wang") and incorrectly matching players to opponents. Table 19 shows LLaMA-3.2-Vision-11B-Instruct providing a detailed image description instead of summarizing the news article's main points, indicating instruction-following failure. In all cases, GPT-4o produces correct, well-structured responses and the judge model identifies specific error types with detailed rationales.

Score Aggregation Results

Figure 3 compares aggregated scores from the full LMMS-EVAL set with aggregated scores from LMMS-EVAL LITE across multiple LLaVA model variants. The bars for each model show strong alignment between full-set and lite-set scores, with the relative ordering of models preserved. The paper does not report the numerical aggregated scores or correlation coefficients, but the visual evidence supports the claim that LMMS-EVAL LITE "provides reliable and aligned results with the time-consuming full-set evaluation" (Section 3).

Table 6 provides the dataset composition of LMMS-EVAL LITE: 15 datasets reduced from 90,223 total instances to 9,134 instances. The largest reductions come from Flickr30k (31,784 → 400, a 79× reduction) and SeedBench (17,990 → 700, a 26× reduction), while smaller datasets are retained in full (LLaVA-W: 60 → 60, MMMU: 900 → 900). The lite set maintains coverage across all task domains: Doc & Infographic Understanding, Image Understanding & Captioning, Visual Question Answering, Math & Science, Visual Dialogue, and Multi-discipline.

Table 7 presents the extended LMMS-EVAL LITE with additional datasets (COCO, VQAv2, OKVQA, VizWiz-VQA, GQA, MM-Bench cn/en), expanding from 15 to 22 benchmarks and from 9,134 to 13,734 total instances. This demonstrates the modular extensibility of the coreset selection approach.

Ablation Studies and Robustness Checks

Embedding choice for coreset selection (Table 5, Appendix D.1): The paper compares two embedding strategies for k-center clustering: LLaVA-Qwen-1.8B embeddings (trained following Liu et al., 2023a, using averaged last hidden states) and CLIP + BGE-M3 concatenation. CLIP+BGE achieves higher correlations on most benchmarks: SeedBench (0.87 vs. 0.77), AI2D (0.98 vs. 0.94), while LLaVA embeddings are slightly better on Flickr30k (0.99 vs. 0.91) and TextCaps (0.98 vs. 0.96). The overall pattern suggests CLIP+BGE is a robust default that doesn't require training a specialized embedding model, while being generally competitive with or superior to the model-specific alternative.

Alternative coreset selection methods (Table 2): Comparing k-center against Quire (active learning) and k-means (variance-minimizing clustering) reveals that k-center's worst-case coverage objective is better suited for benchmark pruning. Quire fails dramatically on some benchmarks (AI2D: 0.45, SeedBench: 0.27) despite strong performance on others (Flickr30k: 0.97, TextVQA: 0.99), indicating that informativeness-based selection is unreliable across task domains. K-means performs comparably to k-center on most benchmarks but notably worse on Flickr30k (0.79 vs. 0.91), suggesting that average-case coverage leaves some regions of highly diverse datasets underrepresented.

Negative result on MME pruning (Section 3): The paper reports that "for MME, due to low correlation between the original and lite set scores, we retain the full version." This demonstrates that coreset selection is not universally applicable — MME's 2,374 instances may test a sufficient diversity of capabilities that aggressive pruning degrades the evaluation signal. The paper does not report the exact correlation or analyze why MME specifically fails, which is a limitation but also an honest acknowledgment of the method's boundary conditions.

Extended lite benchmark (Table 7, Appendix D.3): The paper validates that the coreset methodology extends to additional datasets by constructing an extended LMMS-EVAL LITE with 22 benchmarks and 13,734 instances. The correlations for the additional datasets are not reported, so it's unclear whether they achieve the same alignment quality as the original 15 benchmarks. However, the methodological demonstration that the approach scales to new datasets is valuable.

Multi-judge validation for LIVEBENCH (Appendix E.3): The paper uses GPT-4o as the primary judge model but notes that "Claude-3.5-Sonnet and Gemini 1.5 Pro serve as alternative judge models." While the paper does not report a systematic comparison of judge agreement or a statistical analysis of inter-judge reliability, the inclusion of alternative judges provides a basic robustness check against single-judge bias.

Manual review in LIVEBENCH curation (Section 4.2.1): The paper mentions that after automated quality scoring, the authors "manually review the questions to remove any that are inappropriate," providing a human-in-the-loop quality filter. The scope and criteria of this manual review are not detailed, but it represents an acknowledgment that fully automated benchmark generation requires human oversight to catch edge cases that automated quality metrics miss.

Contamination analysis granularity (Figure 4, Table 4): The paper reports both image and text overlap separately, revealing that contamination can be high on one dimension while low on the other (e.g., AI2D: 6.09% image but 25.97% text; POPE: 42.20% image but 0.00% text). This granularity is important because it distinguishes different contamination mechanisms — image memorization vs. question pattern memorization — and demonstrates that analyzing only one modality would give an incomplete picture.

N-gram length and filtering choices (Section 4.1): The paper uses 8-grams for both text and image token overlap analysis, noting that "typically, an 8~13 n-grams range is used, but we consistently use 8 n-grams for simplicity." The meaningless n-gram filter (excluding n-grams appearing more than 10 times in training data) is an important preprocessing step that prevents common phrases from generating false-positive overlap signals. The paper does not ablate these choices (different n-gram lengths, different frequency thresholds), so the sensitivity of the contamination results to these hyperparameters is unknown.

Critical Assessment

Claim 1: "LMMS-EVAL provides a unified and standardized multimodal benchmark framework that ensures transparent and reproducible evaluations"

What was demonstrated: The paper successfully built and deployed an evaluation framework covering 50+ tasks and 10+ model families with 30+ variants. Table 1 and Table 27 demonstrate that the framework produces consistent, comparable results across diverse benchmarks and models. The framework's design — unified data preprocessing, standardized output parsing, automatic logging — addresses the specific inconsistencies the paper identifies (PPL-based vs. generation-based answer extraction, differing chat template handling).

What was not demonstrated: The paper does not quantitatively demonstrate that the standardization actually matters for result reliability. The natural experiment would be: evaluate the same model on the same benchmark using LMMS-EVAL vs. using the original custom evaluation scripts, and show the score differences. Alternatively: show that model rankings change meaningfully depending on evaluation protocol choices (PPL vs. generation), demonstrating that standardization is necessary for valid comparisons. Without such a demonstration, the paper's claim that existing evaluations are "not directly comparable" rests on qualitative argument rather than quantitative evidence.

Missing evidence: The framework's reproducibility claims would be strengthened by showing that independent runs of the same evaluation produce identical results (or quantifying variance), and that the logged outputs enable an external researcher to reproduce the reported scores. The paper mentions automatic logging but doesn't provide examples or demonstrate reproducibility.

Claim 2: "LMMS-EVAL LITE provides an efficient benchmark set with reliable and aligned results with the time-consuming full-set evaluation"

What was demonstrated: The correlation validation in Table 2 and Table 5 provides strong evidence that lite-set scores track full-set scores across multiple benchmarks and models. The per-benchmark correlations (0.87–0.99) are generally high, and Figure 3 shows aggregate score alignment. The comparison against Quire and k-means demonstrates that k-center is an appropriate algorithm choice.

What was not demonstrated: Several important validation dimensions are missing. First, the paper evaluates correlation using only LLaVA model variants — the same model family used to construct the embeddings (via CLIP+BGE). It is unknown whether the lite set preserves rankings for models with substantially different architectures (e.g., would Qwen-VL or InternVL rankings be as well-preserved as LLaVA rankings?). The embeddings are model-agnostic (CLIP+BGE), which is promising, but the validation is model-family-specific. Second, the paper validates score correlation but not failure mode preservation — does the lite set identify the same specific weaknesses (e.g., "Model X struggles on chart reading but excels at captioning") as the full set? If the lite set preserves aggregate scores but masks capability-specific patterns, its diagnostic value is reduced. Third, the paper does not report the absolute score differences between full and lite evaluations — only correlations. A correlation of 0.98 is excellent, but if the lite set systematically overestimates scores by 5 percentage points relative to the full set, that matters for absolute performance claims.

The MME failure: The acknowledgment that MME was not pruned due to low correlation is honest but raises questions. Why does MME fail when other benchmarks succeed? Is it a property of MME's question structure (binary yes/no, two sub-categories of cognition and perception), its instance count (2,374 — moderate), or something about the embedding space? Without analysis of this failure case, users cannot predict whether their own benchmarks would be amenable to coreset selection.

Missing cost analysis: The paper motivates LMMS-EVAL LITE by evaluation cost but never quantifies the computational savings in concrete terms. Figure 2 shows relative bar heights without numerical labels. What is the actual GPU-hour cost of evaluating LLaVA-OV-72B on the full set vs. the lite set? The 10× instance reduction is clearly stated, but GPU-hours depend on sequence lengths, batch sizes, and parallelism strategies. A concrete cost analysis would help practitioners decide whether the lite set's efficiency gains justify any residual correlation loss.

Claim 3: "LIVEBENCH provides a low-cost and zero-contamination evaluation through continuously updating content from news and forum websites"

What was demonstrated: Table 3 shows that LIVEBENCH produces plausible, discriminative evaluations — model rankings are sensible (GPT-4o > Qwen2-VL-72B > LLaVA-1.5-7B), scores span a wide range (9.4 to 94.8), and the four cognitive categories reveal capability profiles that differ across models (e.g., LLaVA-OV-72B-Chat's relative weakness at Concrete Recognition vs. Analytical Understanding). The contamination analysis in Figure 4 and Table 4 convincingly demonstrates that many existing static benchmarks have significant data overlap with training corpora, establishing the need for contamination-resistant alternatives.

What was not demonstrated: Several aspects of the LIVEBENCH claims require additional evidence. First, "zero-contamination" is claimed but not proven. The temporal argument — questions about September 2024 news couldn't be in training data with earlier cutoffs — is reasonable, but it assumes that similar questions about similar events haven't appeared in training data. A model might have been trained on news articles from 2023 that contain structurally similar content (e.g., "Analyze the ongoing tennis matches displayed on the webpage") even if the specific player names and scores differ. This is "similar questions" contamination (the third category identified in Section 4.1.1), which LIVEBENCH doesn't eliminate — it only eliminates exact duplication. Second, "low-cost" is claimed relative to human-preference arenas like Chatbot Arena, but no cost comparison is provided. What is the Claude-3.5-Sonnet API cost for generating 500 questions, the GPT-4o cost for judging model responses, and the human review cost? How does this compare to the cost of human preference collection? Third, question quality is not systematically evaluated beyond the automated scoring rubric in Stage 4. The paper acknowledges that "the quality of our QA may still fall below that of human-curated answers" but doesn't quantify this gap — e.g., what fraction of questions are removed during manual review, what types of errors survive the automated pipeline, and how often do human evaluators disagree with GPT-4o judgments?

The pipeline dependency on commercial models: LIVEBENCH's generation pipeline uses Claude-3.5-Sonnet for information extraction and QA generation, and GPT-4o for evaluation. This creates a circularity concern: the benchmark is generated by commercial models and judged by commercial models, which could advantage models from the same families (GPT-4o judging GPT-4o's own responses, or Claude-3.5-Sonnet generating questions that favor Claude's strengths). The paper partially addresses this by using GPT-4o as judge (different from Claude-3.5-Sonnet, the generator) and by mentioning alternative judges, but the fundamental dependency remains: LIVEBENCH can only be produced using commercial models, and it tends to show commercial models outperforming open-source ones. The observed 6-point GPT-4o advantage could reflect genuine generalization capability, benchmark construction bias, or (most likely) some combination of both.

Category validity: The four-category Bloom's Taxonomy structure is a design choice that isn't empirically validated. Do the Concrete Recognition questions actually test surface-level extraction, or do they sometimes require deeper reasoning? Do the Divergent Thinking questions actually test creativity, or do they test the model's ability to produce plausible-sounding open-ended responses that GPT-4o judges favorably? The paper provides example questions (Tables 21–24) that appear face-valid, but face validity is not sufficient — a systematic study of whether different question categories actually measure distinct capabilities in LMMs would require factor analysis or similar psychometric validation.

Claim 4 (implicit): "Static benchmarks systematically overstate open-source model capability relative to real-world performance"

What was demonstrated: The contamination analysis (Figure 4) shows that many widely-used benchmarks have significant training data overlap, which could inflate open-source model scores (since open-source models are often trained on the same public datasets that benchmarks derive from). LIVEBENCH (Table 3) shows a 6.1-point gap between GPT-4o (92.0) and the best open-source model (85.9) that is larger than gaps on some static benchmarks.

What was not demonstrated: The paper does not directly compare LIVEBENCH rankings against static benchmark rankings for the same set of models. For the claim to be validated, one would need to show that: (a) open-source models outrank or match GPT-4o on specific static benchmarks (e.g., MME, MMBench), and (b) these same models substantially underperform GPT-4o on LIVEBENCH. The paper asserts that "open-source models often outperform commercial ones like GPT-4V in benchmarks" (Section 4.2) but never provides a concrete example — e.g., "LLaVA-NeXT-110B achieves X on MME vs. GPT-4V's Y, but on LIVEBENCH it achieves A vs. GPT-4o's B." Without this direct comparison, the claim that static benchmarks overstate open-source capability relative to LIVEBENCH rests on the reader's prior knowledge rather than the paper's evidence.

An alternative explanation: The LIVEBENCH gap could reflect the fact that commercial models are genuinely more capable, and static benchmarks also show this gap, but the paper's benchmark selection (focusing on benchmarks where contamination is low) obscures it. Or it could reflect that LIVEBENCH questions (sourced from English-language news websites) favor models trained primarily on English web data (which commercial models are) over models trained on more diverse or multilingual corpora. The paper doesn't disentangle these explanations.

Cross-cutting methodological limitations

Single training data source for contamination analysis: The contamination audit (Section 4.1, Table 4) examines only LLaVA-NeXT training data. While LLaVA is a widely-used open-source LMM family, the contamination landscape for other models (Qwen-VL trained on 1.4B samples, CogVLM on 1.5B) could be substantially different. A benchmark that is contaminated for LLaVA might be clean for Qwen-VL, or vice versa. The paper's tools are open-sourced and could be applied to other training datasets, but the reported numbers are specific to one training corpus.

Test set sizes vary dramatically: The benchmarks in LMMS-EVAL have test/validation sets ranging from 60 instances (LLaVA-W) to 214,354 (VQAv2). Evaluation on very small test sets (LLaVA-W: 60 questions, MM-Vet: 218 questions, Ferret: 120 questions) has high variance — a model's score could change by several percentage points based on a few lucky or unlucky guesses. The paper doesn't report confidence intervals or statistical significance for any benchmark, making it impossible to assess whether observed differences (e.g., LLaVA-OV-72B at 93.7 vs. LLaVA-OV-7B at 86.9 on LLaVA-W) are reliable given the 60-instance test size.

Missing evaluation dimensions: LMMS-EVAL covers many benchmarks but doesn't include some important capability areas: video understanding, multi-turn visual dialogue, visual instruction following in interactive settings, robustness to adversarial inputs, or safety evaluations. The paper's claim to "comprehensive" coverage is relative to existing static benchmarks, not to the full space of LMM capabilities.

The evaluation trilemma itself is not empirically validated: The paper asserts that wide coverage, low cost, and zero contamination form an "impossible triangle" (Figure 1), but this is presented as a conceptual claim without formal proof or systematic empirical demonstration. The claim rests on intuitive arguments: comprehensive static benchmarks are expensive and contaminatable, human evaluation is expensive and unscalable, and dynamic benchmarks may have limited coverage. These are plausible constraints, but the "impossibility" framing is stronger than the evidence supports — it's possible that future methods (e.g., adversarially-generated questions that resist memorization, efficient human-in-the-loop evaluation, or learned difficulty estimators that reduce required instance counts) could relax these tradeoffs. The trilemma is best understood as a useful organizing principle rather than a proven impossibility result.

Closed-source model evaluation limitations: For API-based models (GPT-4V, Gemini, Claude), the paper cannot control inference parameters (temperature, sampling strategy) and must accept the provider's default settings. This introduces a confound: differences between open-source and API-based models could reflect inference configuration rather than underlying capability. Additionally, API models are updated continuously — a GPT-4o evaluation from September 2024 might use a different model version than a December 2024 evaluation, making longitudinal comparisons unreliable.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. LMMS-EVAL LITE depends on a coreset selection pipeline that requires embedding every instance in a dataset using CLIP and BGE-M3, running a greedy k-center algorithm, and validating the pruned benchmark by evaluating multiple model variants on both the full and lite sets to confirm score alignment. The paper acknowledges none of these overheads in its efficiency claims. The 10× instance reduction (90,223 to 9,134 instances) is reported as the efficiency gain, but the cost of constructing the lite benchmark — which includes full-set evaluation of at least "six versions of LLaVA" (Section 3) to validate correlation — is externalized from the accounting. The paper states only that the validation was done on LLaVA variants and provides no guidance on whether this validation must be repeated for each new model family or benchmark addition.

The consequence. A practitioner adopting LMMS-EVAL LITE for a new model family not covered by the paper's validation (e.g., Qwen-VL, InternVL, or a new architecture) cannot trust the published lite set to preserve rankings without re-running the correlation validation themselves. Doing so requires evaluating their new model on the full 90,223-instance benchmark suite — exactly the cost LMMS-EVAL LITE is designed to avoid. The headline efficiency gains are therefore only guaranteed for the LLaVA model family on the specific benchmarks and embedding choices reported in the paper. For any other model family or new benchmark, the validation cost must be paid before the lite set can be trusted, meaning the amortized efficiency depends on how many model variants will eventually be evaluated. If a team is developing a single new model, the lite set offers no net savings — the cost of full evaluation to validate the lite set exceeds the cost of just using the full set.

The MME failure case makes this concern concrete. The paper found that "for MME, due to low correlation between the original and lite set scores, we retain the full version" (Section 3). This was discovered during validation — if the paper had not checked correlation on MME, the lite set would have silently produced misleading scores. A practitioner adding a new benchmark to LMMS-EVAL LITE cannot assume coreset selection will work without performing the same validation. The paper provides no diagnostic for predicting which benchmarks will fail (size? question format? visual diversity?), so every new benchmark addition requires empirical validation.

What evidence exists in the paper. The paper reports correlation results for 15 benchmarks in Table 2 and Table 5, all validated against LLaVA variants. The MME failure is noted explicitly but not analyzed. The paper never quantifies the computational cost of embedding extraction, k-center clustering, or full-set validation in GPU-hours, nor does it discuss amortization scenarios. Figure 2 compares full-set vs. lite-set evaluation time but omits the construction cost entirely. The CLIP+BGE embedding approach requires no model training (unlike the LLaVA embedding alternative), which reduces construction cost relative to alternatives, but the full-set evaluation for correlation validation dominates the total cost regardless of embedding choice.

Mitigation status. The paper partially mitigates this by providing pre-constructed lite sets for the 15 benchmarks in Table 6 (and the extended 22-benchmark set in Table 7), meaning researchers using these exact benchmarks and willing to trust the LLaVA-based validation can use the lite set immediately without re-validation. The CLIP+BGE embeddings are model-agnostic, which the paper argues should transfer across architectures, though this claim is untested beyond LLaVA variants. The paper does not discuss or propose methods for predicting benchmark prunability without full evaluation, and Section 6 (Limitations) does not mention the construction cost. This limitation is not addressed in the paper beyond the implicit strategy of providing pre-built lite sets and hoping the model-agnostic embeddings generalize.

Limitation 2: LIVEBENCH's "Zero-Contamination" Claim Applies Only to Exact Duplication, Not to Similar-Task Generalization

The assumption or constraint. LIVEBENCH achieves contamination resistance through temporal exclusion: questions are generated from news and forum content published after model training cutoffs, so the specific question-answer pairs "could not have been in training corpora regardless of how aggressive the data collection was." The paper's contamination analysis in Section 4.1 identifies three categories of contamination — duplicate images, similar images, and similar questions — and the temporal strategy addresses only the first two. The third category, "similar questions" where "recurring question structures in the training data mirror those in the benchmark dataset," is structurally unaffected by temporal freshness because question templates and reasoning patterns repeat across news cycles. The paper acknowledges this category in its own contamination analysis: "Though not necessarily contamination or overlapping cases, the two images are both testing similar domain knowledge and may help the model to answer questions in the benchmarks" (Appendix C.1, regarding similar questions in MathVista).

The consequence. A model trained on news articles from 2023 containing questions structured as "Analyze the ongoing tennis matches displayed on the webpage, detailing the players involved, their current scores, and the tournaments they are part of" (Table 21's Concrete Recognition example) would have learned the task format — extracting structured sports data from webpage screenshots — even though the specific player names and scores are novel. The paper's LIVEBENCH-2024-09 results would reflect both zero-shot generalization to genuinely new content and transfer from structurally similar training examples, and these cannot be disentangled. This matters because it undermines the strongest version of the paper's claim: that LIVEBENCH measures "models' zero-shot generalization ability on the most recent events" (Section 1). In fact, it may measure a mix of zero-shot generalization to novel facts and few-shot-style transfer from similar task formats seen during training.

The consequence is most severe for models trained on large-scale web data that includes news websites. Commercial models like GPT-4o and Claude-3.5-Sonnet are almost certainly trained on news website screenshots (given their strong OCR and webpage understanding capabilities), meaning they have extensive exposure to the format of LIVEBENCH questions even if the specific events postdate their training cutoffs. Open-source models trained on more curated datasets may have less exposure to webpage-screenshot understanding tasks, producing a format-familiarity gap that confounds the capability comparison. The paper's interpretation that the LIVEBENCH gap reflects "strong zero-shot generalization abilities" in commercial models is one hypothesis; an alternative is that commercial models have more training data resembling LIVEBENCH's format, and the gap reflects domain adaptation rather than general intelligence.

What evidence exists in the paper. The paper's own contamination analysis demonstrates the similar-question phenomenon in existing benchmarks (MathVista examples in Appendix C.1), establishing that this is a real contamination pathway. The LIVEBENCH generation pipeline uses a fixed set of prompt templates (Table 9, Table 13) that produce questions with predictable structural patterns across months — the four Bloom's Taxonomy categories ensure that every LIVEBENCH release contains questions of the form "Identify the key details of [financial instrument] provided in the image for [date]" (Table 8), "Analyze the ongoing [sport] matches displayed on the webpage" (Table 21), and "Evaluate the potential impact of [technology] on the [industry] job market" (Table 24). Models trained on news content — even outdated news content — would have seen these structural templates before.

Mitigation status. The paper does not address this limitation explicitly. The temporal argument is presented as sufficient for contamination resistance, and the third contamination category (similar questions) identified in Section 4.1.1 is not discussed in the context of LIVEBENCH. The paper could partially mitigate this by varying question formats across months (e.g., changing the task structure for the same underlying content) or by measuring format-transfer effects (e.g., by testing models on LIVEBENCH questions with deliberately altered formats to assess template sensitivity). As presented, the "zero-contamination" claim is an overstatement — "zero exact-duplication contamination" would be more precise, but even that undersells the similar-question pathway that the paper itself has documented as significant.

Limitation 3: LIVEBENCH Pipeline Dependency on Commercial Models Introduces Systematic Bias of Unknown Magnitude

The assumption or constraint. The LIVEBENCH curation pipeline (Section 4.2.1) is entirely dependent on commercial models: Claude-3.5-Sonnet performs information extraction and QA generation (Stages 2–3), GPT-4o serves as the primary evaluation judge (Stage 5), and the pipeline's quality control models (Checker, Finalizer, Scorer) are also closed-source commercial systems (Stages 4). The paper is transparent about these dependencies but does not analyze their consequences for benchmark validity. The pipeline design means that LIVEBENCH questions are generated by a model from one commercial family (Anthropic's Claude) and evaluated by a model from a different commercial family (OpenAI's GPT-4o), with both families also appearing as evaluated models in the benchmark (Table 3 includes both Claude-3.5-Sonnet and GPT-4o).

The consequence. This creates two potential bias pathways that cannot be disentangled from genuine capability differences:

Generator bias: Claude-3.5-Sonnet's information extraction and question generation may produce questions that are systematically easier for some model architectures than others. For example, if Claude-3.5-Sonnet tends to extract information in a structured, well-organized format that aligns with how commercial models represent webpage content internally, questions derived from this extraction will test a capability that commercial models have been optimized for. The pipeline's use of Claude-3.5-Sonnet for both the "Information Extraction" and "QA Generation" prompts (Tables 13 and 9) means the entire question formulation reflects Claude's understanding of what is salient and interesting in a webpage, which may not match how open-source models process the same input. A model that "sees" the webpage differently from Claude might answer correctly according to its own understanding but receive low scores because its answer doesn't match the Claude-generated ground truth.

Evaluator bias: GPT-4o's judgments may favor responses that match its own output style, even when the paper's scoring criteria attempt to enforce objectivity. The evaluation prompt (Table 14) asks GPT-4o to "rate whether the assistant response correctly matches the ground truth" using pre-specified criteria, but LLM judges are known to exhibit systematic biases — preferring longer responses, responses in certain styles, or responses that align with the judge model's own world model. The paper's finding (Table 3) that the GPT-4o family achieves the highest scores (GPT-4o: 92.0, GPT-4o-mini: 91.9) while Claude-3.5-Sonnet follows at 90.3 is consistent with evaluator bias (GPT-4o may subtly favor GPT-4o-family responses), generator bias (Claude-generated questions may inherently advantage Claude), or genuine capability differences — and the paper provides no method for distinguishing these explanations.

The circularity is structural, not incidental: if the pipeline used open-source models for generation and evaluation, it could be run independently by third parties, and the benchmark's validity wouldn't depend on access to specific commercial APIs. As designed, LIVEBENCH can only be produced using these specific commercial models, and the results will always embed their particular biases.

What evidence exists in the paper. The paper's case studies (Tables 15–20) provide partial evidence. In Table 20, GPT-4o correctly computes the Bitcoin average closing price while LLaVA-NeXT-OV-72B-Chat makes arithmetic errors — this appears to be a genuine capability gap rather than a bias artifact, since arithmetic correctness should be objective. In Table 18, GPT-4o correctly identifies tennis player names and scores while LLaVA-NeXT-OV-72B hallucinates — again, factual accuracy should be objective. However, these cases were selected by the authors and may represent the clearest examples of genuine differences rather than systematic analysis. The paper mentions using "Claude-3.5-Sonnet and Gemini 1.5 Pro serve as alternative judge models" (Appendix E.3) but does not report inter-judge agreement statistics or show whether GPT-4o-family models maintain their advantage under alternative judges. A systematic comparison of judge models would allow readers to assess evaluator bias, but this is absent.

Mitigation status. The paper partially mitigates evaluator bias by providing question-specific scoring criteria (Table 8 shows an example) that constrain GPT-4o's judgment to specific factual and formatting requirements, reducing the scope for stylistic preference. The alternative judge models (Claude, Gemini) provide a basic robustness check, but without reporting their judgments or inter-rater reliability, the mitigation is incomplete. The generator bias is not addressed at all. The paper acknowledges the quality tradeoff — "the quality of our QA may still fall below that of human-curated answers" (Section 4.2.1) — but frames this as a quality issue rather than a systematic bias issue. The limitation is partially acknowledged but not systematically analyzed or mitigated.

Limitation 4: All Evaluations Are on a Single Model Family (LLaVA) for LITE Validation, with Unknown Transfer to Architecturally Distinct Models

The assumption or constraint. The correlation validation for LMMS-EVAL LITE (Tables 2 and 5) was performed exclusively on LLaVA model variants — the paper states it used "six versions of LLaVA" (Liu et al., 2023a) to compute score correlations between full and lite sets. The embedding space used for coreset selection (CLIP + BGE-M3) is model-agnostic (neither requires model-specific features), which the paper implicitly argues should make the lite set transferable across architectures. However, this claim is untested. The lite set is constructed to preserve the evaluation signal for whatever models were used during validation, and if those models share architectural biases (e.g., all use CLIP-based vision encoders, all use similar instruction-tuning data, all share common failure modes), the selected instances may disproportionately represent capability dimensions where CLIP-based LLaVA-family models show variance while underrepresenting dimensions where other architectures differ.

The consequence. A practitioner evaluating InternVL-1.5 (26B), Qwen-VL-Chat (7B), or Idefics2-8B using LMMS-EVAL LITE cannot be confident that the relative rankings and absolute scores from the lite set will match the full set. The paper's LLaVA-only validation means that the lite set is optimized to distinguish between different LLaVA checkpoints — which differ primarily in LLM backbone, training data scale, and resolution — not between fundamentally different vision encoder architectures, training paradigms, or instruction-tuning strategies. If, for example, Qwen-VL-Chat uses a different vision encoder that excels at certain visual tasks where LLaVA models struggle, but the lite set underrepresents those tasks (because LLaVA models showed little variance on them), the lite set would miss Qwen-VL's advantage.

Some benchmarks may be more sensitive to this than others. The CLIP+BGE embeddings produce an instance representation that captures visual and textual similarity, which should be reasonably architecture-independent, but the validation of which instances produce score-preserving subsets is model-family-dependent. The paper's claim that "we can consider it to be safe to prune the datasets" (Section 3) is conditional on the assumption that score preservation transfers across architectures — an assumption the paper states but does not test.

What evidence exists in the paper. The correlation results in Table 5 show some variation across benchmarks (SeedBench: 0.87, InfoVQA: 0.94), and the MME failure case demonstrates that pruning doesn't always work even within the LLaVA family. The paper evaluates a diverse range of models on the full LMMS-EVAL suite (Table 1, Table 27) but does not evaluate these same models on LMMS-EVAL LITE to verify that non-LLaVA architectures achieve similar lite-set-to-full-set correlation as LLaVA variants do. The LLaVA embedding approach (using a trained LLaVA-Qwen 1.8B model) could theoretically transfer better because the embedding model shares the vision-language architecture, but the paper found CLIP+BGE embeddings to be comparably effective on LLaVA variants (Table 5) — leaving open the question of which embedding strategy transfers better to other architectures.

Mitigation status. The paper does not address this limitation. Section 6 (Limitations) does not discuss model-family generalization, and the paper provides no guidance on when re-validation would be necessary. A pragmatic mitigation would be for practitioners to validate the lite set on at least one model from their target architecture family before trusting it for that family's development cycle. The open-source release of the codebase makes such re-validation possible but doesn't reduce its cost. This limitation is not addressed in the paper beyond the implicit claim that model-agnostic embeddings suffice for transfer.

Limitation 5: LMMS-EVAL LITE Preserves Aggregate Scores but Diagnostic Capability Breakdowns May Be Lost

The assumption or constraint. The coreset selection objective (Section 3) minimizes the absolute difference between a model's average score on the full set and its average score on the lite set:

$\min_{V: |V| \ll |D|} \left| \frac{1}{|D|} \sum_{i=1}^{|D|} S(y_i, \hat{y}_i) - \frac{1}{|V|} \sum_{i=1}^{|V|} S(y_i, \hat{y}_i) \right|$

This objective, and the k-center clustering algorithm that approximates it, optimize for aggregate score preservation — ensuring the overall accuracy number on the lite set matches the full set. The paper's validation (Tables 2 and 5, Figure 3) assesses only this dimension: correlation between full-set and lite-set aggregate scores across model variants. However, the primary value of wide-coverage benchmarking is not aggregate scores — it is diagnostic breakdowns that identify specific capability weaknesses. A researcher running ablations wants to know not just "did overall performance improve?" but "did chart reading improve while captioning declined?" or "did the model get better at spatial reasoning while maintaining OCR capability?"

The consequence. The lite set may preserve the overall ranking of models (model A > model B on aggregate) while distorting the per-capability diagnosis. Two specific failure modes are possible. First, the lite set might compress variance on some capability dimensions — if a benchmark's selected instances happen to be the easier or more homogeneous subset, models may show artificially compressed score differences on that dimension. Second, the lite set might reverse capability-specific rankings — model A might genuinely be better than model B at chart understanding, but the lite set's 400 selected ChartQA instances (from 2,500) might by chance include mostly questions where model B performs well and exclude questions where model A's advantage is most pronounced. The correlation validation (0.96 for ChartQA in Table 5) makes this unlikely on average but provides no guarantee for specific model pairs.

The paper's stated use case for LMMS-EVAL LITE — "provide useful and low-cost signals during model training and ablations" (Section 3) — is precisely the scenario where diagnostic breakdowns matter most. During model development, a researcher comparing two training recipes needs to know not just which recipe produces better average performance, but whether the improvement comes at the cost of regressions on specific capabilities. If LMMS-EVAL LITE masks such regressions (because the lite subset doesn't include instances sensitive to the regression), the researcher might select a recipe that looks better on aggregate but is worse on important capability dimensions.

What evidence exists in the paper. None. The paper reports only aggregate score correlations and Figure 3's aggregated weighted average, never analyzing whether per-benchmark score differences between specific model pairs are preserved — e.g., "LLaVA-1.5-13B outperforms LLaVA-1.5-7B by 10 points on ChartQA in the full set; does the lite set reproduce this 10-point gap?" This is a different and more stringent test than average correlation. Two models could have perfectly correlated scores with r = 0.99 on the full set vs. lite set across 15 benchmarks while still showing a 5-point reversal on one specific benchmark — and it is precisely that reversal that would mislead a researcher running targeted ablations. The paper's correlation analysis would not detect this failure mode.

Mitigation status. The paper does not address this limitation. The recommendation to retain MME in full due to low correlation suggests awareness that per-benchmark validity matters, but the paper doesn't analyze whether even high-correlation benchmarks preserve per-model-pair score differences or capability-specific rankings. A partial mitigation would be to report per-benchmark scatter plots (full-set score vs. lite-set score for each model variant) rather than only correlation coefficients — this would reveal whether the lite set systematically compresses or expands score differences at different performance levels. The paper does not provide these. This limitation is not addressed.

Limitation 6: The Trilemma Framework Asserts Impossibility Without Formal Proof or Systematic Empirical Validation

The assumption or constraint. The paper's central organizing principle is that "it is an impossible triangle to evaluate models with wide coverage and low cost without making the benchmarks susceptible to contamination" (Section 1), and that "one can not achieve the three goals simultaneously but only find a trade-off" (Section 2.2). This is presented as a structural constraint — "we cannot break this impossible triangle" — and the paper's three contributions are explicitly framed as navigating within this constraint rather than challenging it. The trilemma is asserted based on conceptual arguments: comprehensive static benchmarks are expensive to run and can be memorized; human evaluation is expensive and doesn't scale; dynamic benchmarks have limited coverage. No formal proof of impossibility is provided, and the empirical evidence demonstrates tradeoffs (e.g., contamination exists in static benchmarks) but does not demonstrate that these tradeoffs are inescapable rather than merely current.

The consequence. The trilemma framing may prematurely foreclose research directions that could relax or circumvent the alleged constraints. For example, adversarial benchmark generation — where questions are algorithmically perturbed to resist memorization while preserving evaluation validity — could potentially achieve wide coverage and zero contamination at moderate cost. Learned difficulty models could reduce the number of instances needed for reliable evaluation, improving the cost-coverage tradeoff without sacificing contamination resistance. Active learning approaches could dynamically select the most diagnostic instances during evaluation rather than pre-selecting a static subset. None of these approaches obviously violate any fundamental constraint; they exploit structure that the trilemma framework assumes away.

More practically, by framing the trilemma as unbreakable, the paper treats LMMS-EVAL LITE and LIVEBENCH as occupying fixed, opposite corners of the tradeoff space — LITE sacrifices contamination resistance for cost, LIVEBENCH sacrifices coverage for contamination resistance. But future work combining these approaches (e.g., dynamic, continuously-updated lightweight benchmarks that use coreset-style selection on temporally-fresh content) could potentially achieve better simultaneous performance on all three axes than the paper's framework predicts. The paper's architecture treats these as independent tools for separate use cases, which is a reasonable engineering decision given current capabilities but may undersell the potential for integrated solutions.

The trilemma also makes an implicit claim about the relationship between the three desiderata: that they are mutually constraining in all evaluation contexts. This claim is not empirically validated — the paper demonstrates that some benchmarks are contaminated and some comprehensive evaluations are expensive, but it does not demonstrate that the tradeoff is universal. For example, a benchmark testing mathematical reasoning on synthetically-generated diagrams might achieve wide coverage (many concepts tested), low cost (automated generation), and zero contamination (synthetic data, not in any training corpus) simultaneously. The trilemma might hold for natural-image-based evaluation but not for synthetic or procedural evaluation — a distinction the paper doesn't draw.

What evidence exists in the paper. The contamination analysis (Figure 4, Table 4) provides strong evidence that existing static benchmarks are contaminated, supporting the claim that "wide coverage + low cost" (as practiced currently) is incompatible with contamination resistance. Figure 2 provides evidence that comprehensive evaluation is computationally expensive, supporting the claim that "wide coverage + zero contamination" is costly. LIVEBENCH demonstrates that a dynamic benchmark can be cheaper than human evaluation while avoiding contamination, but at limited coverage. These are all existence proofs of tradeoffs, not proofs that the tradeoffs are fundamental. The paper provides no counterfactual analysis (e.g., "what would it take to achieve all three simultaneously?") and no formal model of the constraints.

Mitigation status. The paper acknowledges the trilemma's limitations implicitly through its own supplementary contributions — the extended LMMS-EVAL LITE (Appendix D.3) and the multi-judge LIVEBENCH setup suggest awareness that the initial formulations can be improved. However, the paper never questions whether the trilemma is truly insurmountable, and Section 6 (Limitations) states only that "we assume that the evaluation trilemma cannot be resolved" and suggests "future work that goes deeper into finding a better trade-off among the sides of the trilemma or potentially overcoming it." This acknowledgment is honest but underspecified — the paper treats the impossibility as an assumption rather than a conclusion, and the evidence provided is consistent with both "currently difficult tradeoff" and "fundamentally impossible." The limitation is partially acknowledged but the strength of the impossibility claim exceeds the evidence provided.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model, a new training technique, or a new capability breakthrough. It introduces something arguably more important at this stage of the field: a systematic audit of how we know what we think we know about LMMs. The contribution is a reality check — a demonstration that the evaluation infrastructure the multimodal community has relied on to measure progress, compare models, and guide development decisions is compromised in specific, quantifiable ways, and a practical toolkit for doing better. This is not a paradigm shift in the Kuhnian sense — it doesn't overthrow a dominant theory or introduce a new one. It is a diagnostic intervention: it identifies measurement failures that have been invisibly distorting the field's feedback signals and provides calibrated instruments that correct for them.

The most lasting conceptual contribution is likely the evaluation trilemma as a design constraint. The recognition that wide coverage, low cost, and zero contamination cannot be simultaneously maximized in any single benchmark — and that efforts to do so produce instruments that are mediocre on all three axes — reframes how the field should approach benchmark design. Before this paper, the implicit assumption in benchmark construction was additive: to make a better benchmark, add more tasks (increase coverage), add more instances (increase reliability), add more diverse sources (increase generality). This paper demonstrates that addition alone creates instruments that are simultaneously expensive, contaminated, and only partially diagnostic. The trilemma framing implies that benchmark design is fundamentally about choosing which failure modes to accept, not about eliminating all failure modes. This is a more mature and productive stance than the implicit "our benchmark is comprehensive and unbiased" claims that accompany most benchmark paper introductions.

The paper's reframing of the relationship between static benchmarks and real-world performance is a second landscape-shifting contribution, though one that the paper understates. The LIVEBENCH results (Table 3) combined with the contamination analysis (Figure 4) provide converging evidence for a specific, directional bias in the LMM evaluation ecosystem: static benchmarks systematically overstate open-source model capability relative to commercial models, and this overstatement is caused by a combination of data contamination (Figure 4, showing 46–68% image overlap in widely-used benchmarks) and evaluation format simplicity (LIVEBENCH's unstructured webpage screenshots vs. MME's yes/no questions with isolated images). This is not a "commercial models are better" finding — it is a "our measurement instruments have been miscalibrated in a specific direction" finding. The implication for the research community is that optimizing open-source LMMs against static benchmark scores — the dominant development paradigm — is optimizing against a partially contaminated target that doesn't fully represent the capabilities needed for deployment. This doesn't mean static benchmarks are useless, but it means their scores must be interpreted through a contamination-aware, format-aware lens, and that development decisions based solely on static benchmark improvements may be investing effort in capabilities that don't transfer to real-world use.

The paper also resolves a puzzling contradiction in the LMM literature that the paper itself doesn't explicitly call out as a contradiction, but which its framework explains. Several prior works have noted that open-source LMMs can match or exceed GPT-4V on specific benchmarks (MME, MMBench) while providing substantially worse user experience in practice. This pattern — benchmark-victory but deployment-failure — has been attributed vaguely to "benchmark overfitting" or "real-world complexity." The paper provides a mechanistic decomposition of this gap: (1) direct data contamination (ChartQA at 68.64% image overlap, Figure 4), (2) format-specific optimization (open-source models trained extensively on multiple-choice QA formats that dominate static benchmarks), and (3) weak generalization to unstructured, information-dense inputs (LIVEBENCH's webpage screenshots requiring simultaneous OCR, layout understanding, and content reasoning). Each mechanism is independently demonstrated in the paper, and together they explain why a model that performs well on a clean, isolated benchmark image with a formatted multiple-choice question might fail catastrophically on a cluttered news homepage requiring extraction of specific facts from dense visual-textual layouts. This decomposition is actionable: it tells researchers not just that their benchmarks are misleading but why and in what specific ways, enabling targeted interventions.

The methodological contribution of the SEED-tokenizer-based contamination detection is a capability-building rather than landscape-shifting contribution, but it may prove to be one of the paper's most broadly adopted techniques. The threshold-free, dataset-independent nature of the approach (Section 4.1) makes it immediately applicable to any training-dataset / benchmark pair where both image corpora are available, without requiring per-dataset calibration. As training datasets continue to grow and benchmark contamination becomes an increasingly urgent concern, having a standardized, reproducible contamination audit methodology is valuable infrastructure. The paper's release of the detection tools as open-source means that future benchmark papers can — and should — include contamination analyses against major training corpora as a standard component of benchmark documentation, analogous to how dataset papers now routinely report demographic statistics and annotation quality metrics.

Follow-Up Research This Work Enables

Systematic contamination audits across all major LMM training corpora and benchmarks. The paper's contamination analysis (Figure 4, Table 4) examines only LLaVA-NeXT training data against a set of benchmarks. A natural and high-impact extension would apply the same SEED-tokenizer-based image overlap methodology and 8-gram text overlap analysis to additional training corpora: Qwen-VL's 1.4 billion samples, CogVLM's 1.5 billion samples, InternVL's training data, and the (partially known) training data of commercial models accessible through public datasets like LAION, DataComp, and Common Crawl dumps. The research question is: does the contamination landscape differ across training corpora, and do models trained on specific corpora show benchmark advantages proportional to their contamination levels on those benchmarks? A strong study would produce a contamination matrix — benchmarks as rows, training corpora as columns, cell values as overlap percentages — and then correlate per-model benchmark scores with their training corpus's contamination level on that benchmark after controlling for model size and architecture. If the correlation is strong, it would provide direct evidence that contamination inflates scores. If it's weak, it would suggest that contamination exists but models don't effectively memorize contaminated instances, which would refine our understanding of how contamination actually affects evaluation.

Cross-architectural validation of LMMS-EVAL LITE score preservation. The paper validates LMMS-EVAL LITE's score correlation exclusively on LLaVA model variants (Tables 2 and 5, Figure 3). The critical follow-up is: does the lite set preserve rankings and absolute scores for architecturally distinct models not in the LLaVA family? A strong study would evaluate the full LMMS-EVAL suite (90,223 instances) and the LMMS-EVAL LITE subset (9,134 instances) on InternVL-1.5, Qwen-VL-Chat, Idefics2-8B, XComposer-4KHD, and at least one API-based model (GPT-4o or Gemini-1.5-Pro), then compute per-benchmark correlations between full-set and lite-set scores for each model family. If correlations remain above 0.90 for all families, the lite set is architecture-agnostic and safe for general use. If correlations drop for specific model families on specific benchmarks (analogous to the MME failure case), the paper's approach would need to identify which benchmark-architecture combinations require full-set evaluation. A particularly informative negative result would be finding that Qwen-VL-Chat's rankings are poorly preserved on DocVQA or ChartQA — tasks where Qwen-VL's architecture makes specific design choices about OCR and document understanding that differ from LLaVA's approach — as this would reveal that coreset selection on model-agnostic embeddings doesn't capture architecture-specific capability variance.

Inter-judge reliability analysis for LIVEBENCH with quantitative bias measurement. The paper uses GPT-4o as the primary LIVEBENCH judge, notes that Claude-3.5-Sonnet and Gemini 1.5 Pro serve as alternatives, but reports no inter-judge agreement statistics. A critical follow-up study would: (1) evaluate all LIVEBENCH-2024-09 responses using all three judge models independently, (2) compute pairwise agreement rates and Cohen's kappa or Krippendorff's alpha, (3) test whether judge models systematically favor responses from their own model family (GPT-4o judging GPT-4o-mini and GPT-4o responses vs. Claude judging Claude-3.5-Sonnet responses), and (4) report whether model rankings change under different judges. The research question is: how much of the observed GPT-4o advantage on LIVEBENCH is evaluator bias vs. genuine capability? If GPT-4o's advantage persists under Claude and Gemini judging, the capability gap is genuine. If GPT-4o's advantage shrinks or vanishes under alternative judges, the LIVEBENCH methodology needs judge diversification or debiasing. A strong version of this study would include human evaluation on a stratified sample of 50–100 questions as a ground-truth reference, enabling measurement of both human-judge agreement and judge-model bias relative to human judgments. This would be expensive (human evaluation costs) but would provide the calibration data needed to interpret all LIVEBENCH results.

Format-variation stress testing of LIVEBENCH to measure template transfer effects. The paper's contamination analysis identifies "similar questions" — recurring question structures that may advantage models trained on structurally similar data even when specific content is novel. LIVEBENCH's fixed four-category Bloom's Taxonomy structure and consistent prompt templates (Tables 9, 11, 12, 13) mean that every monthly release contains structurally predictable question types. A stress-test study would: (1) generate a second version of LIVEBENCH-2024-09 where each question is reformulated using a different task format (e.g., Concrete Recognition questions converted from "Identify the key details..." to a structured extraction format with explicit field labels; Analytical Understanding questions converted to multiple-choice; Real-world Application questions converted to fill-in-the-blank), (2) evaluate the same set of models on both the original and reformatted versions, and (3) measure whether the performance gap between commercial and open-source models changes systematically with format. The research question is: how much of model performance on LIVEBENCH is format-specific vs. content-general? If open-source models close the gap substantially on reformatted questions, it suggests that their weakness is partly in format flexibility rather than content understanding per se, which would inform training data curation (include more diverse task formats during instruction tuning). If the gap persists across formats, it suggests a deeper content-understanding deficit.

Difficulty-predictive benchmark design using LIVEBENCH-style dynamic generation. The paper's coreset selection (LMMS-EVAL LITE) and dynamic benchmark generation (LIVEBENCH) are presented as independent tools, but they could be combined into a more powerful instrument. A follow-up study would: (1) use the k-center clustering methodology not on existing benchmarks but on the space of possible LIVEBENCH questions, using historical LIVEBENCH releases to characterize the distribution of question types, difficulty levels, and required capabilities, (2) develop a generation procedure that actively fills capability coverage gaps — when a particular Bloom's Taxonomy level or domain (sports, finance, politics, science) is underrepresented in recent questions, the generator is prompted to produce questions targeting that gap, and (3) evaluate whether this "coverage-aware dynamic benchmark" provides more stable and interpretable capability profiles than the current LIVEBENCH, which relies on whatever news happens to be available in a given month. The research question is: can dynamic benchmarks achieve both contamination resistance and controlled coverage simultaneously, rather than trading one for the other? This would directly challenge the paper's trilemma framework by testing whether the third tradeoff (dynamic benchmarks sacrifice coverage) can be engineered around through active generation strategies.

Temporal contamination decay measurement. LIVEBENCH's contamination resistance rests on the assumption that questions about recent events couldn't have been in training data. But an interesting edge case exists: what about models that are continuously updated or that were trained very recently? A study tracking the same set of models on successive monthly LIVEBENCH releases would measure whether open-source model performance on LIVEBENCH improves over time as models incorporate more recent training data — specifically, whether models released after September 2024 show improved September 2024 LIVEBENCH scores compared to models released before September 2024. If post-September models perform better on September questions, it suggests that temporal freshness alone isn't sufficient for long-term contamination resistance across model versions. If performance is stable, it validates the temporal exclusion strategy. This study requires longitudinal data collection and careful tracking of model release dates and training data cutoffs, but it would provide the first empirical measurement of contamination decay rates in LMM benchmarks.

Practical Applications and Downstream Use Cases

Model development cycle acceleration through LMMS-EVAL LITE. The paper's most immediately deployable contribution is the lite benchmark set for rapid iteration. During model training, developers typically run evaluation on validation sets periodically (e.g., every N training steps) to monitor progress, detect regressions, and decide when to stop training. The full LMMS-EVAL suite at 90,223 instances is too expensive for this use case — evaluating every 1,000 training steps on 50+ tasks across 8×A100 GPUs would consume more compute than the training itself for many model scales. LMMS-EVAL LITE at 9,134 instances (10× reduction) makes periodic multi-task evaluation practical: if a full evaluation previously took 12 GPU-hours, the lite version takes roughly 1.2 GPU-hours, enabling 10× more frequent evaluation checkpoints for the same compute budget. The key practical benefit is earlier detection of capability-specific regressions. If a training recipe improves chart understanding but degrades document QA, this tradeoff might be invisible in a single-benchmark validation metric but would appear in LMMS-EVAL LITE's multi-task aggregate. The paper's validation (Table 5, Figure 3) shows that this signal is reliable for LLaVA-family models — the primary use case the authors target. For non-LLaVA architectures, practitioners should validate correlation on at least one model variant before trusting the lite set for development decisions.

Pre-deployment contamination auditing for benchmark-based model claims. Organizations releasing LMMs with benchmark scores as part of their technical reports or marketing can use the paper's contamination detection methodology (Section 4.1, Figure 4) to audit their training data against the benchmarks they report scores on. The practical workflow: (1) identify all benchmarks being reported, (2) run the SEED-tokenizer-based image overlap analysis and 8-gram text overlap analysis against the model's full training corpus (or as much of it as is accessible), (3) report contamination percentages alongside benchmark scores, and (4) where contamination exceeds a threshold (the paper demonstrates that 20%+ overlap is common in contaminated benchmarks), either exclude those benchmarks from claims or explicitly note that scores may partially reflect training data exposure rather than generalization. This practice, if adopted as a community norm, would transform benchmark reporting from a simple score table into a more honest and diagnostic format that distinguishes memorization-driven performance from genuine capability. The paper's tools are open-source and the methodology requires only access to training data — a constraint that limits third-party audits of commercial models but is feasible for any model developer auditing their own releases.

Dynamic benchmark as a complement to static evaluation in model selection. Organizations selecting between LMMs for deployment (whether choosing between API providers or between open-source checkpoints) currently rely almost exclusively on static benchmark scores. The paper demonstrates that these scores are partially contaminated and may systematically overstate open-source capability relative to real-world performance (Table 3 vs. published static benchmark results). A practical deployment selection workflow would: (1) evaluate candidate models on relevant static benchmarks through LMMS-EVAL to get standardized, comparable scores, (2) evaluate the same models on the most recent LIVEBENCH release to assess zero-shot generalization to unstructured, temporally novel content, and (3) weight the results based on the deployment context. For a customer-facing chatbot that will encounter diverse, unpredictable user queries with real-time information needs, LIVEBENCH performance should be weighted more heavily because it's a better proxy for the actual deployment distribution. For a narrow-domain document understanding system operating on a fixed document type, static benchmarks for that specific domain remain appropriate. The paper provides the infrastructure to make this multi-signal evaluation practical — LMMS-EVAL for static benchmarks, LIVEBENCH for dynamic generalization testing — and the trilemma framework provides the conceptual justification for using multiple evaluation instruments rather than trusting any single one.

When to Prefer This Method

The paper does not position LMMS-EVAL, LMMS-EVAL LITE, and LIVEBENCH as competing alternatives among which practitioners should choose. Rather, it presents them as complementary instruments for different evaluation scenarios, each addressing a different tradeoff within the evaluation trilemma. The decision framework is therefore not "which tool should I use?" but "which combination of tools is appropriate for my specific evaluation need?" The paper's architecture implies the following practical guidance, grounded in the specific evidence provided:

For comprehensive model capability auditing and cross-model comparison: Use the full LMMS-EVAL suite with its standardized pipeline. This is appropriate when publishing model results, comparing across model families, or establishing baseline capability profiles. The key value is comparability — all models evaluated through the same preprocessing, inference, and metric calculation pipeline produce scores that can be directly compared, which is not true of scores from different papers using different evaluation scripts (Section 2.1). The cost is high (Figure 2), but this is the "wide coverage + zero contamination" corner of the trilemma where comprehensiveness justifies the expense.
For rapid model development iteration and ablation studies: Use LMMS-EVAL LITE with the understanding that it trades some contamination resistance and diagnostic granularity for 10× faster evaluation cycles. This is appropriate when comparing checkpoints within a model family during training, hyperparameter tuning, or architecture search. The paper's validation (Tables 2, 5) shows that lite-set scores track full-set scores for LLaVA-family models at correlations above 0.87 on most benchmarks, making it reliable for relative comparisons within that architecture family. For non-LLaVA architectures, the paper recommends validating correlation on at least one model variant before trusting the lite set. The paper explicitly states that LMMS-EVAL LITE "is not designed to fully compare the performance of different model families" but rather to "provide useful and low-cost signals during model training and ablations" (Section 3).
For assessing zero-shot generalization and contamination-free evaluation: Use LIVEBENCH, with awareness that it measures a specific capability — interpreting unstructured, temporally novel web content — rather than general multimodal capability. The paper's results (Table 3) show that LIVEBENCH reveals capability gaps (the 6.1-point GPT-4o vs. best-open-source gap) that static benchmarks may obscure. This is appropriate when evaluating models intended for deployment in open-domain settings with real-time information needs, or when suspicious that static benchmark scores are inflated by contamination. The paper's contamination analysis (Figure 4) provides the justification: if widely-used benchmarks like ChartQA have 68.64% image overlap with training data, then high scores on those benchmarks may not reflect generalizable chart-reading ability, and LIVEBENCH provides an orthogonal signal that is structurally resistant to this specific failure mode. The limitation is coverage — LIVEBENCH questions are necessarily about current events and may underrepresent capabilities that don't appear in news content (mathematical reasoning, scientific diagrams, abstract visual reasoning).

The paper does not provide a specific contamination threshold above which static benchmarks should be distrusted in favor of LIVEBENCH, nor does it provide guidance on how to weight static vs. dynamic evaluation signals when they conflict. These are natural next steps for the research community to establish through accumulated evidence across multiple model releases and LIVEBENCH months.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: LMM Evaluation Is Fragmented, Unstandardized, and Potentially Misleading

Why This Problem Matters

Where Existing Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 LMMS-EVAL: Standardized Evaluation Infrastructure

3.4.2 LMMS-EVAL LITE: Coreset Selection for Efficient Evaluation

3.4.3 Contamination Detection Methodology

3.4.4 LIVEBENCH: Dynamic Benchmark Generation Pipeline

3.4.5 Score Aggregation for LMMS-EVAL LITE

3.4.6 Evaluation Metrics and Answer Extraction Methods

3.4.7 Model Integration and Inference Details

3.4.8 LIVEBENCH Case Analysis Protocol

3.4.9 Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: The Evaluation Trilemma as an Explicit Organizing Principle

Innovation 2: SEED-Tokenizer-Based Image Contamination Detection as a Tunable-Free Method

Innovation 3: Coreset Selection for Multimodal Benchmark Pruning via Joint Visual-Textual Embedding

Innovation 4: The LIVEBENCH Pipeline as a Practical Countermeasure Against Contamination Through Temporal Freshness

Innovation 5: Empirical Evidence That Static Benchmarks Systematically Overstate Open-Source LMM Capabilities Relative to Real-World Performance

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

LMMS-EVAL Standardized Benchmark Results

Contamination Analysis Results

LMMS-EVAL LITE Validation Results

LIVEBENCH Evaluation Results

Score Aggregation Results

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: "LMMS-EVAL provides a unified and standardized multimodal benchmark framework that ensures transparent and reproducible evaluations"

Claim 2: "LMMS-EVAL LITE provides an efficient benchmark set with reliable and aligned results with the time-consuming full-set evaluation"

Claim 3: "LIVEBENCH provides a low-cost and zero-contamination evaluation through continuously updating content from news and forum websites"

Claim 4 (implicit): "Static benchmarks systematically overstate open-source model capability relative to real-world performance"

Cross-cutting methodological limitations

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

Limitation 2: LIVEBENCH's "Zero-Contamination" Claim Applies Only to Exact Duplication, Not to Similar-Task Generalization

Limitation 3: LIVEBENCH Pipeline Dependency on Commercial Models Introduces Systematic Bias of Unknown Magnitude

Limitation 4: All Evaluations Are on a Single Model Family (LLaVA) for LITE Validation, with Unknown Transfer to Architecturally Distinct Models

Limitation 5: LMMS-EVAL LITE Preserves Aggregate Scores but Diagnostic Capability Breakdowns May Be Lost

Limitation 6: The Trilemma Framework Asserts Impossibility Without Formal Proof or Systematic Empirical Validation

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

EXAONE 3.0 7.8B Instruction Tuned Language Model

KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models