DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain

🎯 Pitch

State-of-the-art VLMs, including GPT-4.1 and Gemini, achieve a mediocre 24.45% nutrition estimation error rate at best and fundamentally fail at fine-grained visual discrimination when menu items look alike, exposing a brittle understanding that simple classification benchmarks hide. DiningBench’s 15,000+ multi-view restaurant images systematically break these models by testing not just what they see, but whether they can reason about portions, ingredients, and dietary implications across multiple angles.

1. Executive Summary

This paper introduces DiningBench, a hierarchical multi-view benchmark that evaluates Vision-Language Models (VLMs) across three levels of cognitive complexity — Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering — using 3,021 distinct dishes with an average of 5.27 images per entry sourced from real restaurant user-generated content. The benchmark operationalizes fine-grained discrimination through hard distractors (visually similar dishes from the same menu, such as distinguishing Smoked Salmon Salad from Fresh Salmon Avocado Salad) and multi-view synthesis (aggregating complementary angles to resolve occlusion and ambiguity), while evaluating Chain-of-Thought reasoning (structured step-by-step visual observation and ingredient breakdown prompting) as a mechanism for complex dietary queries. Evaluating 29 state-of-the-art proprietary and open-source models reveals that even the strongest model — Gemini-3-Pro-Preview — achieves only 24.45% average MAPE on Nutrition Estimation and that fine-grained classification remains challenging despite 81.83% top accuracy, establishing that current VLMs struggle with precise volumetric reasoning and subtle visual discrimination even when general reasoning capabilities appear strong.

2. Context and Motivation

The Core Problem: Food Benchmarks Have Not Kept Pace with Vision-Language Models

The central gap this paper addresses is a mismatch between the capabilities of modern VLMs and the benchmarks used to evaluate them in the food domain. VLMs like GPT-4o, Gemini 2.5, and Qwen-2.5-VL have demonstrated remarkable general visual reasoning — producing detailed image descriptions, answering complex questions, and performing multi-step inference across diverse domains. Yet the food benchmarks against which these models are measured remain conceptually frozen in the era of single-label image classification.

The consequences of this mismatch are not merely academic. When a VLM achieves high accuracy on existing food benchmarks like Food-101, a developer might reasonably conclude the model "understands food" and deploy it for real-world dietary applications — automated nutrition logging, allergen detection, meal planning for chronic disease management, or intelligent kitchen assistants. But as DiningBench demonstrates, strong performance on coarse-grained benchmarks masks fundamental failures in the precise, multi-view, hierarchical reasoning that actual dietary assistance demands. A model that correctly distinguishes a pizza from a salad at 90% accuracy on Food-101 tells you nothing about whether it can distinguish a Smoked Salmon Salad from a Fresh Salmon Avocado Salad — the exact kind of fine-grained discrimination needed when a user with a shellfish allergy photographs their meal for automated safety checking. The paper is thus motivated by a safety-critical gap between benchmark signaling and real-world competence in the dietary domain.

Beyond safety, the gap has practical implications for the food industry's AI ambitions. Platforms like Meituan (the paper's data source) handle millions of food images daily, with applications ranging from menu digitization and nutritional transparency to personalized dietary recommendations. Deploying VLMs for these tasks requires models that can perform not just "what dish is this?" but "how many calories does this portion contain?" and "is this suitable for a low-sodium diet?" — questions that move from identification to quantification and reasoning. Existing benchmarks provide no signal on these dimensions.

The Problem Decomposed: Four Limitations of Existing Food Benchmarks

The paper identifies four specific, interlocking limitations that render existing benchmarks inadequate for evaluating modern VLMs. Each is worth understanding in detail because they collectively explain why strong performance on prior datasets does not transfer to DiningBench.

Limitation 1: Tasks Are Overly Simplistic — Evaluation Stops at Coarse-Grained Classification

The vast majority of food benchmarks — Food-101 (Bossard et al., 2014), UEC-Food (Matsuda et al., 2012), VIREO Food-172 (Chen and Ngo, 2016), ISIA Food-500 (Min et al., 2020), Food2K (Min et al., 2023) — are structured as single-label classification problems: given an image of a dish, select its category from a predefined list. These benchmarks have driven genuine progress in visual recognition, but they test only one dimension of what constitutes "understanding food." They ask what, never how much, why, or what if.

This narrowness matters because it creates a capability illusion. A model that correctly classifies a bowl of noodles as "Dan Dan Noodles" has demonstrated coarse-grained pattern matching — it has recognized the combination of minced pork, chili oil, noodles, and peanuts as statistically co-occurring with that label. But it has not demonstrated that it can estimate the calorie content (which depends on oil volume and portion size), identify whether the dish contains allergens (sesame paste is often hidden in the sauce), or reason about whether the dish is nutritionally appropriate for a specific dietary need. These are the questions real users and real applications ask, and existing benchmarks provide no signal about them.

The paper explicitly contrasts this with its hierarchical task design (Section 3.1): Fine-Grained Classification tests identification under adversarial difficulty, Nutrition Estimation tests continuous-value quantification from visual cues, and VQA tests higher-order reasoning about culinary techniques, dietary compliance, and counterfactual scenarios. This progression from identification → quantification → reasoning is not an arbitrary taxonomy — it maps to the cognitive demands of real dietary applications.

Limitation 2: Single-View Assumption — Real-World Multi-Angle Photography Is Ignored

Traditional food datasets overwhelmingly treat food recognition as a single-image problem: one dish equals one photograph, typically a canonical front-facing or overhead shot in controlled lighting (Food-101, UEC-Food). This makes the problem artificially clean and fails to reflect actual user behavior. In practice, when people photograph their meals — whether for dietary logging, social media, or restaurant review platforms — they typically capture multiple angles: a top-down shot to show the full composition, a side angle to reveal height and layering, a close-up to capture texture details, and sometimes a cross-section to reveal internal structure (e.g., the doneness of a steak, the filling distribution of a dumpling).

These multiple views carry complementary information that a single canonical image cannot provide:

A top-down view alone cannot convey portion height or volume, which are critical for nutrition estimation.
A single angle may occlude key ingredients (e.g., a protein buried under salad greens).
Different lighting between user photos reveals surface properties (glossiness indicating oil content, char indicating grilling) that vary with angle.

The paper's design choice to provide an average of 5.27 images per dish (Table 1) is thus not merely about scale — it is about operationalizing a fundamental shift from single-image classification to multi-view synthesis as a first-class evaluation criterion. This enables testing whether models can fuse complementary views (Section 4.3) or whether they treat additional images as noise — a question that existing single-view benchmarks cannot even ask.

Limitation 3: Lack of Fine-Grained Discrimination — Weak Distractors Mask Perceptual Failures

In existing multiple-choice food recognition evaluations, distractor options (the wrong answers presented alongside the correct one) are typically randomly sampled from unrelated categories. If the target dish is "Pad Thai," the distractors might be "Caesar Salad," "Ice Cream," and "Pizza." This makes the task trivially easy for any model with even rudimentary visual-semantic alignment — the model doesn't need to distinguish Pad Thai from a visually similar noodle dish like "Drunken Noodles" or "Pad See Ew"; it simply needs to recognize that the image contains noodles and match against the most noodle-associated label.

This is the "bag-of-features" problem that Section 4.5 explicitly diagnoses: models function as high-level feature detectors (this image has noodles + peanuts + lime → Pad Thai) without performing the fine-grained visual analysis needed to distinguish genuinely similar dishes. The consequence for benchmark validity is severe: high accuracy on weakly-distracted benchmarks can reflect semantic shortcut learning rather than genuine visual understanding. A model might "recognize" a dish by detecting a single dominant ingredient and mapping it to the most probable label in its training distribution, entirely ignoring contradictory visual evidence from secondary ingredients, plating style, or cooking technique.

DiningBench's solution, derived in Section 3.2.2, is to construct hard negatives — distractors sourced from the same merchant's menu within the same category. The example in the paper's Abstract is illustrative: distinguishing a Smoked Salmon Salad from a Fresh Salmon Avocado Salad requires the model to notice the presence or absence of avocado slices, detect smoking marks versus fresh texture on the salmon, or identify garnish differences — subtle visual cues that coarse-grained classifiers can safely ignore. The dataset construction pipeline (Figure 2 and Table 7 in Appendix C) uses Gemini-3-Pro-Preview to select candidates that are "visually or semantically similar" and then applies a two-pass filtration to remove both overly ambiguous samples (where the correct answer cannot be reliably determined from images) and trivially distinguishable samples (where the distractor is too easy).

Limitation 4: Inaccurate Nutritional Metadata — Unreliable Ground Truth for Quantification Tasks

The paper is particularly critical of existing nutrition estimation datasets, identifying two distinct failure modes that compromise their utility as benchmarks for VLMs.

First, low image quality in large-scale cross-modal corpora. Recipe1M+ (Marin, 2019), one of the most widely used food datasets, pairs recipes with images but suffers from noisy web-scraped photos — low resolution, poor lighting, non-food content, and weak correspondence between the image and the listed ingredients (the image may show a plated version while the recipe's nutritional calculations assume raw ingredients in bulk quantities). Using such data as ground truth for a regression task is fundamentally problematic because the model is being evaluated against targets that may not even correspond to what is visible.

Second, narrow domain coverage in high-quality quantification datasets. Nutrition5K (Thames et al., 2021) provides carefully measured nutritional data but focuses on standardized cafeteria settings — the same dish photographed under controlled conditions, with ingredients and portions measured in a commercial kitchen. This provides high-quality ground truth but fails to capture the diversity of real-world dining: restaurant dishes with variable portion sizes, home-cooked meals with improvised recipes, and dishes from diverse culinary traditions that don't appear in institutional food service. Similarly, FastFood (Qi et al., 2025) focuses on fast-food chain restaurants where portions are standardized and menus are limited. These datasets evaluate models on narrow, controlled distributions that don't reflect the messy, diverse reality of how people actually eat.

DiningBench's approach to nutritional ground truth (Section 3.2.3) addresses both gaps simultaneously. First, it sources images from authentic, diverse restaurants via Meituan's platform, ensuring coverage across Chinese, Western, Japanese, Asian, Latin American, and Indian cuisines (Table 3). Second, it obtains nutritional data through a two-tier verification system: direct extraction for dishes where merchants provide explicit nutritional labeling, and LLM-assisted estimation (using Gemini-3-Pro-Preview with image, ingredient list, and portion size inputs) cross-referenced against the USDA FoodData Central database, followed by systematic manual verification. An Atwater system consistency check (Appendix C, Table 11: verifying that $\text{Calories} \approx 4 \times \text{Protein} + 4 \times \text{Carbohydrates} + 9 \times \text{Fat}$ ) is applied during quality assurance. This pipeline aims for comprehensive coverage without sacrificing accuracy — a point the paper is transparent about flagging as a limitation (the LLM-assisted generation "may inevitably inherit latent biases," Section 8).

Why These Limitations Matter Now: The VLM Evaluation Gap

The four limitations above would matter for any food benchmark, but they are acutely relevant now because of the recent trajectory of VLM development. Prior to 2023, food recognition was primarily the domain of specialized vision models (CNNs, ViTs) trained end-to-end on food-specific datasets. These models were designed for classification; evaluating them on classification made sense. The advent of general-purpose VLMs — models pre-trained on internet-scale image-text pairs and capable of open-ended visual reasoning — fundamentally changes what it means to "evaluate food understanding."

A modern VLM can, when prompted, attempt all three of DiningBench's tasks: it can choose among dish options (classification), it can estimate numerical nutritional values (quantification), and it can answer open-ended dietary questions (reasoning). But existing benchmarks provide no signal on the latter two capabilities and weak signal on the first (due to easy distractors). This creates an evaluation blind spot: a developer choosing between VLMs for a food application has no principled way to compare their nutrition estimation accuracy or their fine-grained discrimination ability. The paper's Figure 1 framework illustrates this gap visually — moving from identification to quantification to reasoning represents a progression that existing benchmarks simply do not cover.

Furthermore, the recent emergence of "thinking" models (e.g., Qwen-3-VL-Thinking, o4-mini) that employ Chain-of-Thought reasoning at inference time adds another dimension to the evaluation gap. When a model can now explicitly verbalize its reasoning — "I observe the char marks indicating grilling, the oil separation indicating a high-fat marinade, therefore..." — we need benchmarks that can test whether that reasoning is grounded in visual evidence or merely plausible-sounding rationalization. DiningBench's systematic CoT experiments (Section 4.4) directly address this by comparing performance with and without CoT prompting across all three tasks, revealing that CoT often degrades performance on visual discrimination tasks (Figure 5, Figure 9) — a finding that has direct implications for how thinking models should (or should not) be deployed in dietary applications.

How This Paper Positions Itself Relative to Existing Work

DiningBench is explicit about its relationship to prior benchmarks: it does not claim to replace Food-101 or Nutrition5K but rather to fill the evaluation dimensions they leave empty. The paper's positioning can be understood along three axes.

Axis 1: From Single-Task to Hierarchical Multi-Task Evaluation. Unlike prior benchmarks that specialize in one task (Food-101 for classification, Nutrition5K for estimation, FoodieQA for culturally-aware VQA), DiningBench integrates all three into a single framework. This is not merely about convenience — it enables cross-task analysis that would be impossible with separate datasets. The paper can ask questions like: Does a model that excels at classification also excel at nutrition estimation? (Table 2 suggests not necessarily — Gemini-3-Flash-Preview leads classification at 81.83% but trails Gemini-3-Pro-Preview on nutrition estimation at 25.21% vs. 24.45% Average MAPE.) Are CoT benefits consistent across tasks or do they vary? (Section 4.4 shows they vary dramatically — CoT helps some models on VQA but hurts on classification and estimation.) This cross-task perspective is a genuine contribution of the benchmark design.

Axis 2: From Random Distractors to Adversarial Hard Negatives. The paper positions its fine-grained classification task against the implicit assumption in existing benchmarks that classification difficulty comes from the number of categories, not the similarity of distractors. Food-101 has 101 categories; Food2K has 2,000. DiningBench has only 8 options per question — fewer than most classification benchmarks — but makes each option maximally confusable by sourcing from the same menu category. This is a quality-over-quantity argument about what makes classification genuinely difficult: it's not the size of the label space but the visual similarity of the alternatives. A model that achieves 90% on Food-101's 101-way classification might achieve far lower accuracy on DiningBench's 8-way classification because the latter eliminates the semantic shortcuts that make large-category classification artificially easy for VLMs with strong language priors.

Axis 3: From Controlled Settings to In-the-Wild Realism. Prior nutrition estimation datasets (Nutrition5K, FastFood) trade ecological validity for measurement precision — they work in cafeterias and chain restaurants because those settings allow for controlled portion sizing and ingredient tracking. DiningBench trades some of that measurement precision (by relying partly on LLM-assisted estimation verified against USDA data) for dramatically broader coverage of real-world dining scenarios — user-uploaded photos from diverse restaurants with variable lighting, angles, plating styles, and portion sizes. The paper explicitly acknowledges this tradeoff in Section 3.2.3 (noting the "two complementary methods" for obtaining nutritional data and the subsequent cross-referencing and manual verification), positioning it as a necessary compromise to achieve the diversity that prior benchmarks lack.

The Intellectual Lineage: Building on Prior Work While Addressing Its Gaps

DiningBench's design draws on several prior innovations while addressing their limitations:

From FoodieQA (Li et al., 2024b) and IndiFoodVQ (Agarwal et al., 2024), it inherits the idea that food understanding involves cultural and domain-specific reasoning beyond visual recognition. But the paper notes these benchmarks "typically treat logic in culture" (Section 2.2) — they focus on culturally-aware question answering without the hierarchical progression through classification and quantification that DiningBench provides.
From Nutrition5K and FastFood, it inherits the framing of nutrition estimation as a visual regression task. But it extends this from cafeteria/fast-food settings to diverse restaurant dining and adds the multi-view dimension (an average of 5.27 images vs. single images in prior work).
From the MATH and MMLU evaluation paradigms in the LLM literature, it implicitly inherits the idea that benchmarks should have hierarchical difficulty — easy, medium, and hard questions that probe different levels of model capability. DiningBench operationalizes this through its task hierarchy (classification → estimation → reasoning) and through the explicit difficulty filtering in its VQA construction pipeline (Appendix C, Table 12, which rejects "trivial tasks" answerable "in <0.5s without thinking").
From the broader multimodal benchmark literature (MMBench, SEED-Bench, MME), it inherits the comprehensive evaluation ethos — testing many models across many dimensions — but applies it to a domain-specific, ecologically valid setting rather than general visual reasoning.

The Stakes: Why Getting Food VLM Evaluation Right Matters

The paper's motivation extends beyond academic benchmarking into practical consequences. Errors in food-domain VLMs are not merely embarrassing — they can be medically consequential. A model that misclassifies a dish containing peanuts as peanut-free could endanger an allergic user. A model that underestimates the calorie content of a meal by 40% could undermine a diabetic user's insulin dosing calculation. A model that fails to detect that a "vegetarian" dish contains hidden animal products (fish sauce, chicken stock) could violate religious dietary restrictions.

The paper does not overstate these risks — it addresses them soberly in Section 8 (Limitations & Potential Risks), noting that "deployment of VLMs evaluated on this benchmark for real-world dietary guidance necessitates extreme caution" and that "human-in-the-loop oversight" is essential for health-related applications. But the implicit argument is that without benchmarks like DiningBench that actually measure the capabilities relevant to these risks, we cannot even assess how close or far current models are from safe deployment. A VLM achieving 81.83% fine-grained classification accuracy and 24.45% Average MAPE on nutrition estimation is clearly not ready for autonomous dietary guidance — but without DiningBench, we wouldn't know these numbers, and a developer might mistakenly deploy based on Food-101 accuracy alone.

This connects to a broader theme in the VLM evaluation literature: the distinction between headline performance (high accuracy on general benchmarks) and tail performance (reliability on specific, safety-critical sub-tasks). DiningBench is designed to expose tail performance failures — the "hard" classification distractors that exploit model over-reliance on high-level semantics, the nutrition estimation errors that reveal lack of volumetric reasoning, the VQA failures that show reasoning chains ungrounded from visual evidence. The benchmark's value proposition is not that it replaces existing food datasets but that it reveals failure modes they systematically hide.

3. Technical Approach

3.1 Reader orientation

DiningBench is not a model or an algorithm — it is a curated evaluation dataset and benchmarking framework that tests Vision-Language Models on food understanding through three increasingly difficult tasks. The problem it solves is the absence of realistic, multi-view, hierarchically difficult evaluation for dietary-domain VLMs; the "shape" of the solution is a pipeline that transforms raw user-generated restaurant data into structured test sets that probe fine-grained visual discrimination, continuous nutritional quantification, and dietary reasoning.

3.2 Big-picture architecture (diagram in words)

The DiningBench system has four major components:

Base Data Acquisition Pipeline — ingests ~20M raw user-generated food images and merchant metadata from the Meituan platform, applies quality filtering and deduplication, and produces a curated base of 6,057 dish entries with associated multi-view photos.
Task-Specific Dataset Construction Pipelines — three separate sub-pipelines that transform base data into evaluation sets for Fine-Grained Classification, Nutrition Estimation, and Visual Question Answering using AI-assisted generation (Gemini-3-Pro-Preview, Gemini-2.5-Pro) followed by rigorous human verification.
Evaluation Harness — a standardized inference and scoring framework that prompts 29 VLMs (10 proprietary, 19 open-source) on each task, collecting predictions and computing task-specific metrics (Accuracy, MAE, RMSE, MAPE, LLM-as-a-Judge accuracy).
Analysis Layer — systematic experiments that vary input conditions (image count from 1 to 4, Chain-of-Thought prompting enabled/disabled) and compute per-task metrics, difficulty-bin breakdowns, and qualitative failure mode taxonomies.

Information flows sequentially: raw platform data → filtering and quality checks → curated base data → task-specific AI-assisted generation → human verification → evaluation prompts → model inference → metric computation → multi-view and CoT ablation analysis.

3.3 Roadmap for the deep dive

First, the base data acquisition pipeline (Section 3.2.1), because all downstream tasks depend on the quality and filtering decisions made at this stage.
Second, the Fine-Grained Classification dataset construction (Section 3.2.2), which introduces the "hard negatives" concept that distinguishes DiningBench from prior benchmarks.
Third, the Nutrition Estimation dataset construction (Section 3.2.3), which tackles the dual challenge of obtaining reliable ground truth across diverse cuisines without controlled-laboratory measurement.
Fourth, the VQA dataset construction (Section 3.2.4), which operationalizes higher-order dietary reasoning in both single-image and multi-image formats.
Fifth, the evaluation metrics (Section 3.3), because the choice of metrics determines what the benchmark actually measures and what properties model rankings inherit.
Finally, the experimental setup details (Section 4.1) and the CoT prompting templates, since these govern how models interact with the benchmark and must be understood before interpreting results.

3.4 Detailed, sentence-based technical breakdown

This is primarily a benchmark construction paper whose core idea is that evaluating VLMs for food understanding requires multiple orthogonal difficulty axes — fine-grained visual discrimination against adversarial distractors, continuous nutritional regression from visual cues, and multi-step dietary reasoning — all operating on multi-view, ecologically valid restaurant data rather than single-view, controlled-laboratory imagery.

Base Data Acquisition Pipeline

The base data pipeline transforms raw, noisy platform data into a curated, high-quality foundation suitable for benchmark construction. This is the most critical quality-gating stage because errors introduced here propagate to all downstream tasks.

Input source. All raw data originates from Meituan, described as "China's preeminent local life service platform," which aggregates merchant-provided reference images, user-generated content (UGC) photos, and structured metadata (dish names, portion sizes, ingredient lists, and textual descriptions). The starting volume is approximately 20M user-generated images accompanied by disjoint metadata fields across different merchants.

Multi-stage filtering sequence. The pipeline applies five sequential filtering operations, each reducing the dataset while enforcing a specific quality criterion:

Quality and consistency filtering via distilled discriminators. The authors train two specialized models by knowledge distillation from GPT-4. Specifically, GPT-4 is used as a teacher to generate quality judgments, and these judgments are used to fine-tune Qwen-2.5-VL-7B as two separate discriminators:
- An Image Quality Assessment model that evaluates visual quality (clarity, exposure, composition, absence of artifacts).
- A Reference-Matching model that verifies whether a user-uploaded photo actually depicts the same dish as the merchant's reference image — critical because users sometimes upload photos of different dishes or non-food content.
The paper does not disclose the exact training data volume or agreement rate between GPT-4 and the distilled models, but states that application of these models reduces the dataset from ~20M to 685k images — a roughly 29× reduction, indicating aggressive filtering. This step is essential because raw UGC on any platform contains substantial noise: blurry photos, non-food images, mismatched dish-photo pairs, and promotional graphics rather than actual food photos.
Frequency thresholding by dish. The remaining 685k images are grouped by dish identity (linking user photos to their claimed dish). Dishes with fewer than three user photos are excluded. This threshold ensures that every retained dish has sufficient multi-view coverage for the benchmark's evaluation paradigm; a dish with only one or two angles cannot test whether models synthesize complementary views. After this step, 90k distinct dishes remain — roughly 7.6 images per dish on average based on the 685k total images divided by 90k dishes.
Reference image quality validation. For the 90k dishes, the pipeline checks whether the merchant's reference image meets quality standards. Dishes where the reference image is low-quality are discarded even if user photos are adequate, because the reference image serves as the canonical representation for downstream task construction (e.g., as one of the multiple views, and as a baseline for the reference-matching quality check). This retains 41k dishes — about 45% of dishes survive this check.
Metadata enrichment filtering. The pipeline selects only dishes whose merchant descriptions contain detailed ingredient lists — explicit enumerations of components rather than vague marketing language ("delicious sauce" vs. "soy sauce, sesame oil, garlic, ginger"). This criterion is crucial for the Nutrition Estimation task, where ingredient lists (combined with portion sizes) provide the LLM-assisted nutritional ground truth (Section 3.2.3), and for the VQA task, where questions about dietary restrictions or cooking techniques require ingredient-level knowledge. After this filter, 15k candidate dishes remain.
Category-based deduplication, cuisine balancing, and manual quality check. The final stage performs three operations:
- Deduplication: removing near-duplicate dishes that appear under slightly different names at different merchants (e.g., "Spicy Beef Noodle Soup" vs. "Hot Beef Noodles").
- Balancing across cuisine origins: the paper's Table 3 shows the resulting distribution includes Chinese (2,086 dishes), Western (286), Worldwide (252), Asian excluding Chinese/Japanese (187), Japanese (118), Latin-American (48), and Indian (44). The balancing prevents any single cuisine from dominating evaluation results.
- Manual quality check: human reviewers inspect the retained dishes, leading to a final Base Data of 6,057 high-quality, well-balanced dish entries.

Output of the base data pipeline. Each of the 6,057 entries contains: at least three user-uploaded photos from different angles, one high-quality merchant reference image, a canonical dish name, a category label (e.g., "salads," "noodle soups"), a merchant identifier, an ingredient list, and available nutritional metadata (if the merchant provided it). This structured output feeds into the three task-specific construction pipelines.

Design rationale for the filtering pipeline. The sequence is deliberately ordered to minimize human review burden. Early automated filters (Steps 1–4) remove the vast majority of data without requiring manual inspection, leaving only 0.25% of the original 20M images (15k dishes) for the final manual quality check. The reliance on GPT-4-distilled Qwen-2.5-VL-7B discriminators rather than rule-based filtering (e.g., resolution thresholds, file size checks) is motivated by the need for semantic quality assessment — a well-lit, high-resolution photo of a restaurant table with no identifiable food should be filtered out, which rule-based checks cannot accomplish.

Fine-Grained Classification Dataset Construction

This pipeline transforms base data dishes into multiple-choice classification samples where the distractors are systematically chosen to be maximally confusable with the correct answer.

Core challenge. Standard classification benchmarks evaluate whether a model can match an image to its category label. But VLMs have strong language priors — they can often guess the correct dish by detecting a single salient ingredient and mapping it to the most probable label without performing fine-grained visual analysis. DiningBench's classification task instead asks: given an image of a specific dish, can the model distinguish it from seven other dishes that share ingredients, visual appearance, or cooking methods? Success requires genuine visual discrimination of subtle cues (cutting style, ingredient proportions, plating arrangement, sauce distribution) rather than coarse category matching.

Candidate generation: sourcing from the same menu category. For each target dish, the pipeline operates within a constrained search space: the same merchant's menu within the same category. This constraint is the key insight. If the target is a salad from Restaurant X, the distractors are other salads from Restaurant X — dishes that share not only ingredient vocabulary but also the restaurant's plating style, lighting conditions, and photographic conventions. This eliminates the "different restaurant, different photo style" shortcut that models could exploit if distractors came from random sources.

The paper describes the selection mechanism: Gemini-3-Pro-Preview is prompted with the target dish and a list of candidate items from the same menu category, and it selects seven items that are "visually or semantically similar." The prompt (Table 7 in Appendix C) instructs the model to:

Standardize merchant marketing names to universal dish names (removing modifiers like "Secret," "Grandma's," "Signature").
Filter out items with the same standardized name as the target.
Exclude items with "obviously huge visual differences."

Two-pass automated filtration. After distractor generation, each sample undergoes two rounds of AI-based quality filtering using Gemini-3-Pro-Preview and Gemini-2.5-Pro sequentially. The filtration serves two complementary purposes:

Remove overly ambiguous samples. Some dish pairs are genuinely impossible to distinguish from images alone without tasting — for example, two soups that differ only in a spice that doesn't change appearance. The filter identifies cases where "image resolution or visual features were insufficient to uniquely identify the ground truth" and excludes them. This prevents the benchmark from penalizing models for failure on impossible tasks.
Remove trivially easy samples. Conversely, the filter identifies cases where distractors "were too easily distinguishable from the target" — for example, a salad dressed with a bright red beet purée among otherwise green salads, where color alone would make the task trivial. These are excluded to maintain benchmark difficulty.

The QA prompt used for this filtration (Table 10 in Appendix C) specifies four pass criteria: image must be clear with distinct subject, ground truth must correctly describe the food, distractors must be incorrect descriptions (no multiple-correct-answer scenarios), and the difficulty must be sufficiently high that "the gap between the standard answer and other categories is not too large." Failure cases include "Label Error" (image is "Burger" but GT is "Sandwich") and "Multi-Solution" ("Stir-fried Potato Strips" vs. "Fried Potato Strips" both valid). The exact criteria for these thresholds are not disclosed — the model makes a judgment call based on the prompt's natural language description.

Human verification. Each filtered sample receives a final round of human verification. The paper does not specify the number of human verifiers or the inter-annotator agreement statistics, but the external audit (Section B.4, Appendix B) reports that three independent PhD students reviewed a stratified random sample of 210 entries (70 per task) and achieved a 100% approval rate using the same inclusion criteria.

Output format and statistics. The resulting Fine-Grained Classification dataset contains 2,884 samples (dishes), each with K = 8 options: one ground-truth label and seven hard distractors. The number of associated images per dish averages 5.32 (from Table 1: 15,330 images ÷ 2,884 dishes). The sample format is a prompt listing the eight labeled options (A–H) and asking the model to output a single letter, as shown in Figure 6. This is a forced-choice single-answer format — models cannot abstain or express uncertainty.

Why this construction method is necessary. Alternative approaches to creating hard negatives — human sourcing of similar dishes, using embedding similarity in a pre-trained vision model, random sampling from the same broad category — each have flaws. Human sourcing is expensive and slow at scale (2,884 dishes requires 20,188 distractors, an infeasible annotation burden). Embedding similarity using a pre-trained vision model would inherit that model's biases — if the embedding model itself cannot distinguish fine-grained food differences, it would select distractors that are genuinely different but appear similar in a flawed feature space, creating misleadingly easy or meaninglessly hard questions. The AI-assisted approach using Gemini-3-Pro-Preview leverages its broad food knowledge (accumulated from training on internet-scale recipe and food description data) to identify semantically and visually similar dishes — a task where language model capabilities (understanding ingredient lists, cooking methods, plating conventions) complement rather than replace vision.

Nutrition Estimation Dataset Construction

This pipeline creates samples for the regression task where models must predict continuous nutritional values — Calories, Carbohydrates, Protein, and Fat — from visual input alone.

Core challenge. Nutritional content is a function of ingredients, quantities, and cooking methods — none of which can be directly measured from an image. A VLM must infer: (1) what ingredients are present (visual ingredient recognition), (2) how much of each ingredient is present (volumetric estimation from 2D imagery, requiring inference of depth and container size), and (3) how cooking methods affect nutritional content (oil absorption during frying, water loss during roasting). This is fundamentally harder than classification because it requires continuous prediction with physical world understanding rather than discrete label selection.

Ground truth acquisition: two complementary methods. The paper acknowledges that obtaining reliable nutritional ground truth for diverse, real-world restaurant dishes is challenging — you cannot bring laboratory-grade food composition analysis equipment to thousands of different restaurants. The solution is a hybrid approach:

Method 1: Direct extraction from merchant metadata. For dishes where merchants explicitly provide nutritional information (calories, macronutrients), the pipeline directly uses these values. The paper does not report what fraction of the 15k base data dishes had merchant-provided nutrition data, but given that this is relatively uncommon outside chain restaurants with standardized menus, this method likely covers a minority of dishes.

Method 2: LLM-assisted estimation. This is the primary method for the majority of dishes. The pipeline prompts Gemini-3-Pro-Preview with three information sources:

The food image (both merchant reference and user photos)
The ingredient list (from the merchant's detailed description)
The portion sizes (also from merchant metadata)

The model is instructed to synthesize these sources and generate estimates for Calories, Protein, Carbohydrates, and Fat. The prompt (Table 6, Appendix C) is remarkably detailed, specifying:

Unit handling: if nutritional values are "per 100g," calculate totals based on the specified food_unit.
Marketing skepticism: "Merchants often under-report calories/fat or over-report protein for marketing purposes. You must detect such false advertising or underestimation. If detected, completely discard the food_description numerical data and generate values based on your world knowledge and the food_info ingredients."
Decimal precision: "Do not overlook decimal points (.) in numerical values."
Source selection: "If calorie info exists in both picture and description, choose the source that is more comprehensive."

This prompt design reflects a deliberate awareness that merchant-provided data is not always trustworthy — the instruction to detect and override suspected marketing bias is a form of automated anomaly detection built into the generation process.

Cross-referencing and verification. All LLM-generated estimates are cross-referenced against the USDA FoodData Central database — the authoritative U.S. government database of food composition data. The paper does not specify the mechanism for this cross-referencing (e.g., whether it involves ingredient-level database lookups with portion scaling vs. dish-level nearest-neighbor matching). All estimates then undergo systematic manual verification — human reviewers check that the values are reasonable given the dish's visual appearance and ingredient list.

Additionally, during the quality assurance phase (Table 11, Appendix C), an Atwater system consistency check is applied to every nutrition sample. The Atwater system is the standard method for estimating metabolizable energy from macronutrient content:

$\text{Energy (kcal)} \approx 4 \times \text{Protein (g)} + 4 \times \text{Carbohydrates (g)} + 9 \times \text{Fat (g)}$

The QA prompt requires that "if the error between labeled Total Calories and theoretical calculation exceeds 10%, it is a logical error" — meaning the sample is rejected. This catches cases where merchant data or LLM estimation produces nutrition vectors that violate basic biochemistry (e.g., claiming 500 calories from 10g protein + 10g carbs + 5g fat, which would yield only 125 kcal from the Atwater formula). The QA prompt also rejects cases where "the Sum of Carbs + Protein + Fat in 100g food exceeds 100g" — another basic physical constraint.

Visual-data matching QA. Beyond mathematical consistency, the QA prompt (Table 11) enforces that nutritional values match the visual appearance. A "Nutritional Density Mismatch" failure occurs when "the labeled values fail to reflect the physical form of the food in the image" — for example, an image showing fatty pork belly with extremely low labeled fat, or a pure vegetable salad labeled at 800kcal. This gate ensures that the benchmark's ground truth is not only mathematically consistent but visually plausible.

Output format. Each Nutrition Estimation sample presents the model with one or more images and asks for a JSON output with four numerical fields: calories, protein, carbohydrates, and fat. The prompt (Figure 7) specifies "all units are grams or kilocalories per serving." The model's predictions are compared against the ground truth vector to compute error metrics.

Resulting dataset statistics. The Nutrition Estimation dataset contains 1,650 samples (dishes) with 8,856 images — an average of 5.37 images per dish (Table 1), slightly higher than the classification subset. The nutritional value distributions (Figure 3) show substantial diversity: Calories range from approximately 100 to nearly 2,000 kcal (with a mean of 670.5 kcal), while macronutrient distributions similarly span wide ranges, indicating the dataset includes both light meals (salads, small appetizers) and calorie-dense dishes (multi-component entrees).

Alternative approaches not chosen. The paper does not use:

Laboratory analysis (combustion calorimetry, chemical extraction) — this would be the gold standard for accuracy but is infeasible at scale for thousands of dishes from different restaurants.
Crowd-sourced human estimation — humans are poor at estimating calories from images, with errors typically exceeding 30–50%, which would make benchmark ground truth too noisy for meaningful model comparison.
Pure database lookup (matching dishes to USDA entries by name) — restaurant dishes rarely have exact USDA equivalents due to recipe variation, sauce differences, and portion size heterogeneity.

The LLM-assisted approach is thus positioned as a pragmatic compromise — it leverages a model's capacity for ingredient- and portion-aware estimation while using cross-referencing, consistency checks, and human verification to catch errors. The paper is transparent about the limitations of this approach (Section 8), noting that "the reliance on LLM-assisted generation... may inevitably inherit latent biases or subtle inaccuracies."

Visual Question Answering Dataset Construction

This pipeline creates open-ended reasoning questions that require models to synthesize visual information with culinary knowledge, dietary principles, and counterfactual logic.

Core challenge. VQA tasks must be difficult enough to differentiate models — simple object recognition questions ("What dish is this?") would be trivially answered by any competent VLM. The questions must require multi-step reasoning grounded in visual evidence rather than general knowledge or surface-level captioning. This means the question design must enforce that the answer depends on what is actually visible in the images, not what could be inferred from the dish name alone.

Two-batch construction strategy. The VQA dataset is built in two batches, each targeting different reasoning types and different input modalities:

Batch 1: Multi-Image Reasoning. These samples require information synthesis across at least two distinct images of the same dish. The prompt (Table 8, Appendix C) defines a specific task configuration — the "Standard vs. Reality Discrepancy Audit" — where the model compares a merchant's promotional image (a professionally styled photo) against a user-uploaded real photo of the same dish, and must identify discrepancies in portion size, ingredient integrity, or presentation, then infer consequences (e.g., "calorie gap or experience degradation"). This task type forces models to:

Attend to multiple images (testing multi-view fusion)
Detect fine-grained differences between idealized and actual food presentation
Reason about the real-world implications of visual differences

Batch 2: Single-Image Reasoning. These samples use individual images and focus on three sub-categories:

Cuisine Technique: inferring cooking methods (pan-fry, deep-fry, roast, steam, sous-vide) from visual surface properties (Maillard reaction browning, dehydration shrinkage, emulsification patterns)
Dietary Suggestion: determining whether a dish complies with specific dietary restrictions (vegan, keto, allergy-sensitive) based on visible ingredients and preparation methods
Counterfactual Reasoning: hypothesizing how a dish would change if a visible element were altered ("How would the taste change if the char marks were absent?")

Generation protocol. All VQA samples are generated by Gemini-3-Pro-Preview using detailed prompts that enforce specific quality criteria. The Multi-Image Analysis prompt (Table 8) specifies constraints including:

Atomic Query Principle: "Strictly Forbidden: Including multiple sub-questions. The question must be single-focused, targeting only one core difficulty. Ensure it ends with a single question mark." This prevents compound questions that test multiple skills simultaneously and make failure attribution ambiguous.
Visual-Agnostic Phrasing: "Strictly Forbidden: Describing image content in the question. Do not say 'Based on the golden crispy crust...'; instead ask 'Evaluate the texture characteristics of the food surface...'." This ensures the model must extract visual information itself rather than being prompted with it.
Inference Depth: "Reject simple 'captioning' style recognition. The question must involve implicit reasoning logic." This eliminates surface-level questions that a captioning model could answer.
Fact-Grounded Answer: "While the question cannot contain Metadata, the Answer and CoT must align with the God-view Metadata to ensure numerical precision and prevent hallucinations." The ground-truth answer uses metadata (ingredient lists, cooking methods, nutritional values) that the question text does not reveal.

The Culinary VQA prompt (Table 9) adds domain-specific difficulty criteria:

Strong Visual Dependency: "The answer cannot be obtained solely by reading the metadata; it must describe visual features mentioned (e.g., Maillard reaction, glossiness, stacking order, oil state)."
Multi-step Logic Chain: "Question → Observe specific features → Combine with culinary principles → Rule out distractors → Conclusion."
Detail Sensitivity: "Must distinguish between extremely similar states (e.g., Are the spring onions 'raw and crisp' or 'wilted from hot oil'? Is the sauce 'drizzled' on top or 'stewed' in?)."

Enforced Chain-of-Thought reasoning. All VQA samples include a cot_gt field — a chain-of-thought reasoning trace that serves as both the ground-truth reasoning path and a model for what constitutes valid multi-step inference. The paper states that "CoT reasoning was enforced for all samples" — meaning the ground truth itself follows a reasoning chain, and models are evaluated on their final answer's alignment with this chain's conclusion (not on the intermediate steps, which models may express differently).

Quality assurance: multi-stage filtering. After generation, VQA samples undergo:

Question deduplication — removing semantically identical questions.
Two-round LLM filtering — using the strict audit prompt (Table 12, Appendix C) which checks: structural integrity (no multi-part questions, no false premises), fact and vision zero-tolerance (objects must be clearly visible, questions must be answerable from visual evidence), difficulty and IQ filtering (reject if "a non-expert human can answer in <0.5s without thinking"), and answer quality (no verbosity, no subjectivity). Questions failing any criterion are rejected with a specific issue_category label.
Manual verification — human review of surviving questions.

Resulting dataset statistics and breakdown. The final VQA dataset contains 804 samples (dishes) with 839 images — only 1.04 images per dish on average (Table 1), far lower than the other two tasks. This is because many VQA questions require only a single image (technique identification, dietary suggestion, counterfactual reasoning), and the multi-image subset (35 samples for Multi-Image Analysis, Table 3) is small relative to the full dataset. The category distribution (Table 3) shows:

Cuisine Technique: 532 samples (66.2% of VQA)
Dietary Suggestion: 219 samples (27.2%)
Multi-Image Analysis: 35 samples (4.4%)
Counterfactual Reasoning: 18 samples (2.2%)

The dominance of Cuisine Technique and Dietary Suggestion reflects the paper's focus on practically relevant dietary reasoning tasks, while the smaller Multi-Image and Counterfactual subsets serve as probes for specific advanced capabilities (cross-view synthesis and hypothetical reasoning, respectively).

Why this VQA construction differs from general VQA benchmarks. General VQA datasets (VQA v2, GQA) evaluate broad visual reasoning across diverse scenes. DiningBench's VQA is domain-specialized: every question concerns food, requires culinary domain knowledge, and is answerable only through visual inspection of the provided image(s). The "Visual-Agnostic Phrasing" rule is particularly distinctive — it prevents questions from becoming reading comprehension tests where the model extracts the answer from the question text rather than the image. This design choice is motivated by the observation that VLMs, especially those with strong language priors, can often perform well on VQA benchmarks by relying on linguistic shortcuts (e.g., the word "tennis" in a question strongly correlates with "racket" as an answer, regardless of image content). By stripping visual descriptions from the question text, DiningBench forces visual grounding.

Evaluation Metrics and Protocol

DiningBench uses task-specific metrics that reflect the nature of each evaluation dimension. The choice of metrics is not arbitrary — each metric operationalizes a specific aspect of model capability that the task is designed to measure.

Fine-Grained Classification: Standard Accuracy. This is computed as:

$Acc = \frac{N_{correct}}{N_{total}}$

where $N_{correct}$ is the number of questions where the model selects the correct option letter and $N_{total} = 2,884$ is the total number of classification samples.

What it computes: the fraction of fine-grained 8-way forced-choice questions answered correctly. This is a single scalar between 0 and 1.

Why this form: accuracy is straightforward for forced-choice tasks and enables direct comparison across models. The paper does not use top-k accuracy or other relaxed metrics because the task has a single unambiguous ground-truth dish — partial credit is not meaningful when you either identify the correct salad or you don't.

Nutrition Estimation: Mean Absolute Error, Root Mean Square Error, and Mean Absolute Percentage Error. For each nutritional component $k \in \{\text{Calories, Carbohydrates, Protein, Fat}\}$ and each sample $i \in \{1, \ldots, N\}$ where $N = 1,650$ :

$MAE_k = \frac{1}{N} \sum_{i=1}^{N} |v_{i,k} - \hat{v}_{i,k}|$

$RMSE_k = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (v_{i,k} - \hat{v}_{i,k})^2}$

$MAPE_k = \frac{1}{N} \sum_{i=1}^{N} \left| \frac{v_{i,k} - \hat{v}_{i,k}}{v_{i,k}} \right|$

where $v_{i,k}$ is the ground-truth value of nutritional component $k$ for sample $i$ , and $\hat{v}_{i,k}$ is the model's predicted value for the same component and sample.

What each metric computes:

MAE is the average absolute prediction error in the original units (kcal for Calories, grams for macronutrients). It is linear in error magnitude — a 100-calorie error contributes 10× as much as a 10-calorie error.
RMSE is the root mean squared error, which penalizes large errors more heavily than small ones (due to squaring before averaging). If a model is usually accurate but occasionally produces wildly wrong predictions, RMSE will be high even if MAE is low.
MAPE is the absolute error expressed as a percentage of the ground-truth value. This normalizes for scale — a 50-calorie error on a 100-calorie dish is a 50% error, while a 50-calorie error on a 1,000-calorie dish is only a 5% error.

The final reported metric is the Average MAPE, averaging the MAPE values across all four nutritional components:

$\text{Avg MAPE} = \frac{1}{4} \sum_{k \in \{\text{Cal, Prot, Carbs, Fat}\}} MAPE_k$

Why this combination of metrics: the paper reports all three for completeness, but Average MAPE serves as the primary ranking metric (the column highlighted in Table 2). MAPE is preferred over MAE and RMSE for nutritional estimation because:

It is scale-invariant: dishes in DiningBench range from ~100 to ~2,000 calories (Figure 3), and raw-error metrics (MAE, RMSE) would be dominated by errors on high-calorie dishes, obscuring whether models struggle more on small or large portions.
It is interpretable: a MAPE of 24.45% (the best result from Gemini-3-Pro-Preview) means that, on average, the model's predictions deviate by roughly a quarter of the true value — a clinically meaningful error magnitude for dietary applications.
It handles the heterogeneous units problem: Calories are in the hundreds, while macronutrients are in the tens. Averaging raw MAE across these different scales would be meaningless without normalization.

Why not use RMSE as the primary metric? RMSE would penalize occasional large errors more heavily. While large errors are clinically dangerous (a 500% overestimate of calorie content could lead to dangerous under-eating), RMSE would obscure whether the model is consistently inaccurate or occasionally catastrophic — MAPE provides a more interpretable picture of the typical relative error.

Visual Question Answering: LLM-as-a-Judge Accuracy. For VQA, the paper "adopts an LLM-as-a-Judge paradigm, where an evaluator LLM assesses the semantic consistency and factual correctness of the predicted answer $\hat{a}$ against the ground truth $a$ , reporting the final performance as Accuracy based on the judge's binary verdict." This is distinct from exact string matching — the model's answer need not be character-for-character identical to the ground truth; it must be semantically equivalent and factually consistent.

What it computes: a binary judgment (correct/incorrect) for each VQA sample, aggregated to a single accuracy score. The paper does not specify which LLM serves as the judge, the prompt template used for evaluation, or whether inter-judge agreement was measured.

Why this form: standard string matching would unfairly penalize models that produce correct but differently-worded answers. For a question like "Is this dish suitable for a vegan diet?", a model answering "No, it contains cheese" should be scored as correct even if the ground truth reads "Does not comply, contains cheese products" (as in Figure 8). The LLM-as-a-Judge approach is a standard technique in the modern VLM evaluation literature (used in MMBench, SEED-Bench, and others) for open-ended generation tasks where surface form varies but semantic content is evaluable.

Cross-task comparison caveat. The three tasks use incommensurable metrics — classification accuracy (0 to 1, higher is better), Average MAPE (0 to infinity as percentage, lower is better), and VQA accuracy (0 to 1, higher is better). There is no single aggregate score across tasks, and the paper does not attempt to create one. This is appropriate given the tasks measure fundamentally different capabilities: a model that excels at discrete identification may fail at continuous quantification, and a single-number summary would mask these divergent profiles.

Experimental Setup: Model Configurations and Inference Protocol

Evaluated models. The paper evaluates 29 models — 10 proprietary and 19 open-source — spanning multiple model families and parameter scales. The selection is deliberately broad to assess whether benchmark difficulty patterns generalize across architectures and training regimens.

Proprietary models (10):

Gemini series: Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash-Preview, Gemini-3-Pro-Preview (Comanici et al., 2025). These span Google's latest generations, with the "Pro" variants being larger and more capable than "Flash" variants.
GPT series: GPT-4o, GPT-4o-mini, GPT-4.1, GPT-5, o4-mini (Achiam et al., 2023). This includes both standard models (GPT-4o, GPT-5) and reasoning-optimized variants (o4-mini).
Anthropic: Claude-Sonnet-4.5.

Open-source models (19):

Qwen-VL series: Qwen-2.5-VL variants at 3B, 7B, 32B, and 72B parameter scales (Bai et al., 2025); Qwen-3-VL variants at 4B and 8B in both "Instruct" and "Thinking" modes; Qwen-3-VL-30B-A3B in both "Instruct" and "Thinking" modes. The "Thinking" variants employ explicit Chain-of-Thought reasoning at inference time.
InternVL series: InternVL-3.5 variants at 4B, 8B, 14B, 30B-A3B, and 38B (Wang et al., 2025).
Other models: Gemma-3-12B-it (Team et al., 2025), Keye-VL-1.5-8B (Yang et al., 2025), Mimo-VL-7B-RL (Xiaomi et al., 2025), MiniCPM-V-4.5 (Yu et al., 2025).

Inference configuration for proprietary models. All proprietary models are accessed via official APIs with:

Temperature = 0: deterministic greedy decoding to ensure reproducibility. At temperature 0, the model always selects the most probable token at each step, eliminating sampling variance.
Maximum context length = 16,384 tokens: sufficient to accommodate multiple high-resolution images plus prompt text without truncation. The paper does not report whether any samples exceeded this limit, but given that the VQA samples average only 1.04 images, the context budget appears adequate.

Inference configuration for open-source models. Open-source models are deployed using vLLM, a high-throughput inference engine. The hardware allocation scales with model size:

Models under 8B parameters: single NVIDIA A100 GPU (80GB)
Models between 30B and 38B parameters: two NVIDIA A100 GPUs (tensor parallelism across GPUs)
72B model (Qwen-2.5-VL-72B-Instruct): four NVIDIA A100 GPUs

Why vLLM? The choice of vLLM over other inference frameworks (HuggingFace Transformers, TGI) is likely motivated by throughput — evaluating 19 models across thousands of samples requires substantial compute, and vLLM's PagedAttention and continuous batching provide higher GPU utilization than standard frameworks. However, the paper does not discuss whether vLLM's optimizations introduce any numerical differences compared to reference implementations.

Shared inference parameters. Across all models, both proprietary and open-source, the paper maintains:

Temperature = 0 (deterministic)
Maximum tokens = 16,384

These settings prioritize reproducibility over diversity — appropriate for benchmark evaluation where the goal is to measure a model's best-effort deterministic performance rather than its sampling distribution.

Chain-of-Thought prompting experiments (Sections 4.4 and B.2). For CoT experiments across all tasks, the paper applies the structured prompting template from Table 5 (Appendix C), which specifies a three-step reasoning process:

Visual Observation: "Describe the food's color, shape, texture, plating characteristics, and inferred cooking state."
Ingredient Breakdown: "Identify and list the main ingredients, auxiliary ingredients, seasonings, and sauce components in detail."
Comprehensive Answer: "Answer the question based on the original image content and the observations and breakdown above."

This template is applied to all three tasks when CoT is enabled. When CoT is disabled, models receive direct task prompts without the reasoning scaffold (e.g., the classification prompt in Figure 6, which asks for "only a single uppercase English letter... without any description, explanation, or content other than the option letter"). The comparison between CoT-enabled and CoT-disabled performance is therefore a test of whether explicit verbalized reasoning helps or hinders each task type.

Multi-view experiments (Section 4.3). To test the effect of image count, the paper varies the number of input images from 1 to 4 for representative models (GPT-4o and the Qwen-3-VL series) on the Classification and Nutrition Estimation tasks. The paper does not specify how images are selected when using a subset — whether they are randomly sampled, always include the reference image, or follow a fixed order. This is a methodological detail that could affect results: if the single-image condition always uses the highest-quality reference image while the 4-image condition includes lower-quality user photos, the marginal benefit of additional images might be underestimated.

English translation experiments (Section B.3). A subset of models is evaluated on an English-translated version of DiningBench, constructed via "Gemini-3-Pro-Preview translation and manual verification." The paper reports results for 17 models (Table 4) comparing performance between the original Chinese dataset and the English translation. This experiment tests whether model performance is language-dependent, particularly for Chinese cuisine dishes where the original Chinese names may carry specific culinary and cultural associations that English translations lose.

Geographic and Cultural Coverage Design

The paper's Table 3 reveals the intentional diversity of DiningBench's cuisine coverage. The geographic distribution is:

Region	Dish Count	Percentage
Chinese	2,086	69.1%
Western	286	9.5%
Worldwide	252	8.3%
Asian (non-Chinese/Japanese)	187	6.2%
Japanese	118	3.9%
Latin-American	48	1.6%
Indian	44	1.5%

The Chinese skew and its justification. Chinese cuisine represents 69.1% of the total — more than all other regions combined. The paper explicitly acknowledges this as stemming from "the sourcing platform" (Meituan's predominantly Chinese user base) and treats it as a limitation (Section 8): "the dataset exhibits a cultural skew towards Chinese cuisine due to the sourcing platform, potentially affecting generalization across underrepresented global culinary traditions."

However, this skew has a methodological justification that the paper does not fully articulate: within Chinese cuisine, the benchmark's fine-grained classification task is maximally challenging because Chinese restaurant menus typically contain large numbers of visually similar dishes within the same category. A Sichuan restaurant might offer 8+ variations of "Boiled Fish" differentiated only by subtle sauce differences, pepper varieties, or garnish. The high dish count within Chinese cuisine (2,086) enables the "same menu category" distractor strategy to operate at its fullest difficulty — there are simply more confusable options to choose from.

Why the international coverage matters despite the skew. The non-Chinese subsets, while individually smaller, serve a critical evaluation function: they test whether models can transfer fine-grained discrimination abilities across culinary traditions. A model that learns to distinguish Chinese salads by attending to sauce color and vegetable cutting style may fail on Western salads where the distinguishing visual cues differ (e.g., cheese type, crouton presence, dressing visibility). The inclusion of 286 Western dishes, 187 non-Chinese/Japanese Asian dishes, and 118 Japanese dishes provides coverage to detect such transfer failures, even if the sample sizes are insufficient for statistically robust per-cuisine performance breakdowns.

The "Worldwide" category. This category (252 dishes) likely represents dishes that are not tied to a specific national tradition — fusion cuisine, internationally standardized fast food, or dishes whose origin is ambiguous in the platform's metadata. Its presence ensures the benchmark is not exclusively evaluating culturally-specific food knowledge.

Summary of Key Design Choices and Their Justifications

Distractor sourcing from same menu category over random negative sampling: ensures distractors are genuinely confusable, preventing models from succeeding via semantic shortcuts based on broad category differences.
AI-assisted distractor generation over human sourcing: enables scale (20,188 distractors across 2,884 dishes) while leveraging the generator model's broad food knowledge; human verification serves as a quality gate rather than the primary generation mechanism.
Hybrid nutrition ground truth (direct extraction + LLM-assisted estimation) over any single method: balances coverage (direct extraction is reliable but scarce; LLM estimation covers the majority) with accuracy (USDA cross-referencing and Atwater consistency checks catch mathematical and physical impossibilities).
Atwater system QA gate (calories ≈ 4P + 4C + 9F, within 10% tolerance) over blind trust: provides a hard, physics-based consistency check that flags logically impossible nutrition vectors regardless of the estimation method.
Atomic question principle for VQA over compound questions: makes failure attribution unambiguous — if a model fails, it fails on exactly one reasoning dimension, enabling precise diagnosis.
Visual-agnostic question phrasing for VQA over descriptive questions: prevents language-prior shortcuts and forces genuine visual grounding.
Temperature = 0 for evaluation over sampling with temperature > 0: ensures reproducibility and measures a model's best deterministic performance, appropriate for benchmark comparison rather than distributional analysis.
MAPE as primary nutrition metric over MAE or RMSE: normalizes for the wide range of nutritional values (100–2000 kcal) and provides interpretable percentage errors relevant to dietary applications.
LLM-as-a-Judge for VQA over exact string matching: accommodates the open-ended nature of dietary reasoning answers where multiple surface forms can express the same correct judgment.

4. Key Insights and Innovations

Innovation 1: The Difficulty Hierarchy as a Diagnostic Framework — Moving Beyond "What" to "How Much" and "Why"

The most fundamental conceptual contribution of DiningBench is not any specific task or metric — it is the three-level difficulty hierarchy (identification → quantification → reasoning) as a principled diagnostic framework for evaluating VLMs in specialized domains. This reframing matters because it reveals that food understanding is not one capability but three distinct cognitive operations that VLMs currently master to dramatically different degrees, and that prior benchmarks conflated these dimensions into a single coarse signal.

What the field did before. Prior food benchmarks operationalized "understanding" almost exclusively as classification accuracy — can the model match an image to a label? Food-101, UEC-Food, VIREO Food-172, ISIA Food-500, and Food2K are all variations on this theme, differing primarily in the number of categories and the scale of training data. Even benchmarks that incorporated additional modalities (Recipe1M+ for image-to-recipe retrieval, RecipeQA for comprehension) still evaluated discrete matching or selection rather than continuous quantification or open-ended reasoning. Nutrition-specific datasets (Nutrition5K, FastFood) existed but were narrow in scope and evaluated estimation in isolation. The implicit assumption across this literature was that food understanding is a unitary construct — that a model good at classification is therefore good at "understanding food."

DiningBench's hierarchy exposes this assumption as false. The evidence is in Table 2: Gemini-3-Flash-Preview achieves the highest classification accuracy (81.83%) but trails Gemini-3-Pro-Preview on nutrition estimation (25.21% vs. 24.45% Average MAPE). GPT-4o scores 65.26% on classification but a poor 42.43% Average MAPE on nutrition. These are not minor rank inversions — they show that classification prowess and quantification ability are weakly coupled. A model can be the best identifier but mediocre at estimation, or vice versa. This finding has direct practical implications: a developer selecting a VLM for a calorie-tracking application would make a substantial error by choosing based on classification benchmarks alone.

Why this is a fundamental shift, not an incremental addition. The hierarchy is not merely "we added two more tasks." It is a capability decomposition — it identifies the specific sub-capabilities that real-world food applications demand and designs tasks that isolate them. This is analogous to how cognitive psychology decomposes "intelligence" into distinct factors (verbal comprehension, perceptual reasoning, working memory) rather than treating IQ as a single number. The paper shows that current VLMs exhibit a capability profile across the three levels — strong at high-level semantic matching (classification), weak at precise physical reasoning (nutrition estimation), and mixed at multi-step inference (VQA) — and that this profile varies across models in ways that a single accuracy number would conceal.

The cross-task CoT experiments (Section 4.4, Figures 5 and 9) provide the strongest evidence for the hierarchy's diagnostic value. CoT prompting — widely believed to improve reasoning — has opposite effects on different levels: it degrades classification and nutrition estimation performance (sometimes catastrophically for smaller models, where MAPE "collapses") while showing mixed effects on VQA. This differential response would be invisible in a benchmark that only tested classification, and it reveals something deep about the relationship between verbalized reasoning and visual grounding: when the task requires direct perceptual judgment (fine-grained discrimination, volumetric estimation), explicit verbalization appears to decouple the model from visual evidence, introducing rationalization noise. This finding is not merely a benchmark result — it is a diagnostic insight about VLM failure modes that generalizes beyond the food domain.

Tying to evidence. The hierarchy's diagnostic power is visible throughout Table 2 (the rank reversals between tasks), Figure 5 (the CoT-induced MAPE explosion for smaller models), and Figure 9 (the CoT-induced classification accuracy decline). These are not speculative interpretations — they are direct empirical demonstrations that the three levels probe distinct and partially independent capabilities.

Innovation 2: Hard Negatives as an Adversarial Evaluation Strategy — Exposing the "Bag-of-Features" Shortcut

DiningBench's most distinctive methodological contribution is the hard negative distractor strategy for fine-grained classification. The key insight is that classification difficulty in food recognition comes not from the size of the label space but from the visual similarity of the alternatives — and that sourcing distractors from the same merchant's menu within the same category creates a maximally confusable option set that exposes models' reliance on coarse semantic shortcuts rather than genuine visual discrimination.

What the field did before. The standard approach in food classification benchmarks — from Food-101 (101 categories) to Food2K (2,000 categories) — is to treat classification difficulty as a function of categorical breadth. The implicit assumption is that distinguishing among 2,000 food categories is harder than distinguishing among 101, so scaling up category count constitutes progress. When multiple-choice evaluation is used, distractors are typically sampled randomly from the full category set, meaning the wrong options are usually from semantically distant categories (a "Pizza" is not easily confused with a "Smoothie"). This makes the task artificially easy for VLMs with strong language priors — the model need only detect a few dominant visual features and map them to the most probable label in its training distribution.

This is what Section 4.5 identifies as the "bag-of-features" failure mode: "current VLMs often function as 'bag-of-features' detectors. They identify dominant components but fail to perceive subtle distinctions in cutting styles or textures." The paper provides a concrete example — confusing Tomato Beef Pot with Spicy Beef Pot due to similar color tones — that illustrates the gap between coarse-grained matching and fine-grained discrimination. The model sees "reddish stew with beef" and selects the most probable label consistent with those features, ignoring whether the red comes from tomatoes or chili oil.

Why this innovation matters beyond the food domain. The hard negative strategy is not domain-specific — it is a general evaluation principle applicable to any domain where fine-grained visual discrimination is safety-critical. Medical image analysis (distinguishing benign from malignant lesions with similar visual appearance), industrial inspection (identifying specific defect types on manufactured parts), and species identification (distinguishing visually similar bird or plant species) all face the same problem: coarse-grained benchmarks overestimate model capability by allowing semantic shortcuts. DiningBench demonstrates a scalable methodology — AI-assisted distractor generation from constrained semantic spaces (same merchant/category) followed by automated and human quality filtering — that could be adapted to these domains. The principle is: to test whether a model truly sees fine details, present it with alternatives that differ only in those details.

The paper's diagnostic finding that "Parametric Knowledge Bias and Hallucination" (Section 4.5) causes models to default to "statistically probable dish names rather than the specific variant present in the image" — for example, misclassifying Scallion Oil Chicken as Roasted Chicken — further demonstrates the value of hard negatives. In a random-distractor benchmark, both Scallion Oil Chicken and Roasted Chicken are "chicken dishes" that would likely be answered correctly if the distractors were non-chicken categories. Only by placing them as alternatives to each other does the benchmark expose that the model is relying on the word "chicken" rather than visually discriminating scallion oil from roast preparation.

Tying to evidence. The classification accuracy hierarchy in Table 2 — where even the best model (Gemini-3-Flash-Preview, 81.83%) leaves nearly 1 in 5 hard-negative questions wrong, and strong general models like GPT-4o (65.26%) and GPT-5 (70.18%) struggle — validates that hard negatives create a difficulty floor that random-distractor benchmarks cannot achieve. The qualitative examples in Figure 6 (distinguishing Roasted Pumpkin Chicken Salad from seven salad variants from the same menu) and the error analysis in Section 4.5 provide specific, interpretable failure cases that illustrate the gap between coarse and fine-grained recognition.

Innovation 3: Multi-View Synthesis as a First-Class Evaluation Dimension — Revealing the Scaling Gap in Information Fusion

DiningBench makes multi-view visual fusion not an optional augmentation but a core evaluation axis by providing an average of 5.27 images per dish and systematically measuring how performance changes with image count. This reveals a previously undocumented phenomenon: large models benefit consistently from additional views while small models degrade, suggesting that effective multi-view fusion requires a minimum representational capacity that current small VLMs lack.

What the field did before. Prior food benchmarks universally treated food recognition as a single-image problem. Even when datasets contained multiple images of the same dish (e.g., from different users or angles), these were treated as independent training/evaluation samples rather than as correlated views of the same underlying entity. The implicit assumption was that a single well-chosen image is sufficient to capture all task-relevant visual information, and that additional views are redundant. This assumption has persisted despite the fact that real-world food photography — on platforms like Meituan, Yelp, or Instagram — routinely involves multiple angles, and that portion estimation (critical for nutrition) inherently requires depth information that a single 2D image cannot provide.

DiningBench's multi-view design operationalizes an insight that challenges this assumption: different viewing angles carry complementary information, and the ability to synthesize this information is itself a measurable model capability. A top-down view reveals ingredient composition and plating arrangement but obscures height and layering. A side angle reveals portion height and internal structure (e.g., the thickness of a steak, the layering of a burger) but may occlude ingredients. A close-up reveals surface texture (char marks, glossiness indicating oil) but loses global context. Fusing these views is not trivial — it requires the model to establish cross-view correspondences (this region of the top-down view corresponds to that region of the side view), resolve conflicts (different lighting conditions between user photos), and extract information that is only available from specific viewpoints (depth from parallax, surface properties from specular highlights at specific angles).

The scaling-dependent finding and its significance. Figure 4 shows that while both GPT-4o and Qwen-3-VL-30B-A3B improve with additional images, smaller models (Qwen-3-VL-4B, Qwen-3-VL-8B) exhibit "instability, with MAPE fluctuating or degrading as more images are added." The paper interprets this as evidence that "for models with limited capacity, excessive visual tokens may act as noise or cause information overload rather than providing helpful context." This is not merely a benchmark observation — it is a diagnosis of a fundamental architectural limitation in small VLMs' multi-view processing. The visual token budget of small models is limited, and when it is exceeded by multiple images, the model cannot attend effectively to the relevant features. This finding has direct implications for VLM architecture design: it suggests that efficient multi-view fusion (through cross-view attention, feature pooling, or learned view selection) is as important as raw parameter count for real-world applications involving multiple images.

The "capability jump" the paper observes — the most significant gains occurring when moving from 1 to 2 images (Figure 4) — suggests that the marginal value of additional views follows a diminishing returns curve that varies by model scale. This is analogous to the observation in ensemble methods that the first few diverse models provide the largest error reduction, but the scaling behavior here is model-size-dependent in a way that ensemble theory does not predict. It raises the question of whether future VLMs should include explicit multi-view fusion modules rather than treating additional images as simply more visual tokens concatenated into the context.

Tying to evidence. The multi-view results in Figure 4 (with specific curves for Classification Accuracy, MAE, RMSE, and MAPE across image counts and model scales) and the qualitative diagnosis in Section 4.5 ("Ineffective Multi-View Aggregation" as a primary failure mode) provide the empirical basis for this innovation. The finding that "performance plateaus or declines when N ≥ 3" for many models is a concrete, falsifiable prediction about current VLM limitations that future architectures can be benchmarked against.

Innovation 4: Chain-of-Thought as a Double-Edged Sword — Evidence That Verbalized Reasoning Can Undermine Visual Grounding

Perhaps the most counterintuitive and practically significant finding in the paper is that Chain-of-Thought prompting harms performance on tasks requiring direct visual perception, particularly for smaller models and particularly on Nutrition Estimation. This is not a minor negative result — it challenges a dominant assumption in the VLM literature that explicit verbalized reasoning is broadly beneficial for complex tasks.

What the field assumed before. Since the success of Chain-of-Thought prompting in LLMs (Wei et al., 2022) and its adaptation to VLMs, a common assumption has been that encouraging models to "think step by step" improves performance on tasks requiring multi-step inference. This assumption has driven the development of "thinking" model variants (o1, o4-mini, Qwen-3-VL-Thinking) and the inclusion of CoT in benchmark evaluation protocols. The implicit model is that verbalized reasoning acts as a form of structured computation — breaking complex problems into simpler sub-problems, surfacing implicit knowledge, and enabling error correction through self-consistency.

DiningBench's CoT experiments reveal a boundary condition on this assumption: CoT helps when the task requires reasoning about visual information (some VQA scenarios), but harms when the task requires direct perceptual judgment from visual information (fine-grained discrimination, volumetric estimation). The diagnosis in Section 4.4 is precise: "instead of refining predictions, the generated reasoning steps appear to introduce noise, drastically increasing MAPE. This suggests that explicit verbalization may decouple the final prediction from direct visual evidence, leading to hallucination or the over-rationalization of incorrect features."

Why this is a significant conceptual contribution. This finding reframes CoT from a universal reasoning enhancer to a task-dependent intervention whose benefits depend on whether the primary cognitive demand is perceptual or inferential. It suggests that for tasks requiring precise visual discrimination — distinguishing subtle texture differences, estimating volume from 2D imagery, identifying whether a specific ingredient is present — the act of generating verbal reasoning may interfere with the visual processing pipeline. The model, forced to produce a plausible-sounding reasoning chain ("I observe a reddish-brown sauce, which suggests..."), may become committed to an interpretation that overrides or distorts the actual visual features. This is a form of confirmation bias at the architectural level — the reasoning process generates expectations that bias the perceptual process.

The finding that smaller models suffer more severely (Figure 5, "performance collapse") suggests that this decoupling effect is capacity-dependent: larger models may have sufficient representational resources to maintain visual grounding while generating reasoning chains, while smaller models experience interference between the verbal and visual pathways. This has direct practical implications: for applications involving nutritional estimation or fine-grained visual inspection, developers should be cautious about enabling CoT in smaller VLMs, and should empirically test whether their specific model benefits or suffers from verbalized reasoning rather than assuming it will help.

Tying to evidence. Figure 5 (the radar chart showing MAPE with and without CoT) and Figure 9 (classification accuracy with and without CoT) provide the quantitative evidence. The qualitative diagnosis in Section 4.5 — "when initial perception is flawed, CoT tends to amplify errors through a chain of incorrect reasoning rather than correcting them" — explains the mechanism. The paper's characterization in Section 4.4 is unambiguous: "CoT is not a 'silver bullet' for the DiningBench."

5. Experimental Analysis

Evaluation Methodology

Dataset. DiningBench comprises 3,021 distinct dishes spanning 2,060 categories, supported by 15,928 high-quality images. It is partitioned into three task-specific subsets: Fine-Grained Classification (2,884 samples, 15,330 images, average 5.32 images/dish), Nutrition Estimation (1,650 samples, 8,856 images, average 5.37 images/dish), and Visual Question Answering (804 samples, 839 images, average 1.04 images/dish). All data is sourced from the Meituan platform's user-generated content and merchant metadata, as detailed in Section 3.2.1. There is no train/validation/test split since DiningBench is an evaluation-only benchmark — all samples are test examples.
Base models. The paper evaluates 29 Vision-Language Models: 10 proprietary (Claude-Sonnet-4.5, Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash-Preview, Gemini-3-Pro-Preview, GPT-4.1, GPT-4o, GPT-4o-mini, GPT-5, o4-mini) and 19 open-source (Gemma-3-12B-it, InternVL-3.5 variants at 4B/8B/14B/30B-A3B/38B, Keye-VL-1.5-8B, Mimo-VL-7B-RL, MiniCPM-V-4.5, Qwen-2.5-VL variants at 3B/7B/32B/72B, Qwen-3-VL variants at 4B/8B/30B-A3B in Instruct and Thinking modes). The selection spans multiple model families, parameter scales (4B to 72B+), and training paradigms (standard instruction-tuned vs. reasoning-optimized "Thinking" variants) to assess whether benchmark difficulty patterns generalize across architectures. The paper argues these models are "representative of the capabilities of many contemporary" VLMs.
Metrics. Fine-Grained Classification uses standard Accuracy (fraction of 2,884 samples where the model selects the correct option letter from 8 alternatives). Nutrition Estimation uses Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) computed per nutritional component (Calories, Carbohydrates, Protein, Fat), with the primary ranking metric being Average MAPE — the mean of MAPE values across all four components, formally $\text{Avg MAPE} = \frac{1}{4} \sum_{k} \frac{1}{N} \sum_{i=1}^N |(v_{i,k} - \hat{v}_{i,k}) / v_{i,k}|$ where $N = 1,650$ . Visual Question Answering uses LLM-as-a-Judge Accuracy, where an evaluator LLM (unspecified) assesses semantic consistency and factual correctness of the predicted answer against the ground truth, producing a binary verdict aggregated to accuracy across 804 samples.
Baselines. The paper does not employ traditional baseline methods (e.g., a ResNet classifier or a heuristic nutrition estimator). Instead, the 29 evaluated VLMs serve as mutual baselines — the comparison is across models rather than against a non-neural reference. The VQA evaluation uses an unspecified LLM judge as the scoring mechanism, which itself constitutes a baseline for answer evaluation. For Chain-of-Thought experiments, the baseline condition is direct prompting without the CoT scaffold (models output answers directly rather than following the three-step observation-breakdown-answer template).
Generation budget / compute accounting. There is no formal generation budget accounting since this is a benchmark evaluation, not a method comparison with variable inference-time compute. Models are evaluated with deterministic decoding (temperature = 0) and a maximum context length of 16,384 tokens. Proprietary models are accessed via official APIs; open-source models are deployed using vLLM on NVIDIA A100 GPUs (1 GPU for ≤8B models, 2 GPUs for 30–38B models, 4 GPUs for the 72B model). The multi-view experiments (Section 4.3) vary image count as a compute proxy (more images = more visual tokens processed). The CoT experiments vary reasoning depth (with/without explicit reasoning scaffold). No FLOPs or wall-clock time measurements are reported.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported in the main experiments. The paper does not report confidence intervals, standard deviations, or p-values for any metric. The only validation-like procedure is the external audit described in Appendix B.4, where three independent PhD students reviewed a stratified random sample of 210 entries (70 per task) using the construction inclusion criteria, achieving a 100% approval rate — but this validates dataset quality, not model performance reliability.

Main Quantitative Results

Overall Model Performance Across the Three Tasks

The headline finding from Table 2 is that no single model comprehensively dominates DiningBench, and that performance varies dramatically across tasks. The best classification model (Gemini-3-Flash-Preview, 81.83% accuracy) fails to lead on Nutrition Estimation (25.21% Average MAPE, second-best behind Gemini-3-Pro-Preview's 24.45%). The best nutrition model (Gemini-3-Pro-Preview at 24.45%) trails slightly on classification (81.55% vs. 81.83%). The best VQA model (Gemini-3-Pro-Preview at 90.42%) is not the classification leader. This cross-task rank instability is the empirical foundation for the paper's claim that the three tasks probe distinct capabilities.

Fine-Grained Classification results (Table 2, "Class." column). The top performers are Gemini-3-Flash-Preview (81.83%) and Gemini-3-Pro-Preview (81.55%), with a substantial gap to the next tier: Gemini-2.5-Pro (73.51%), Gemini-2.5-Flash (71.01%), GPT-5 (70.18%), and GPT-4.1 (68.59%). Among open-source models, Qwen-3-VL-30B-A3B-Instruct leads at 65.43%, followed by Qwen-2.5-VL-72B-Instruct (65.29%) and Qwen-3-VL-8B-Instruct (64.15%). The worst performers include InternVL-3.5-4B (43.76%), InternVL-3.5-8B (45.32%), and Gemma-3-12B-it (48.61%). The gap between the best proprietary model (81.83%) and the best open-source model (65.43%) is 16.4 percentage points — a substantial margin that indicates fine-grained visual discrimination remains a capability concentrated in the largest proprietary models. Even the strongest model leaves 18.17% of questions incorrect despite having only 8 options (chance = 12.5%), meaning the task is far from saturated.

Nutrition Estimation results (Table 2, "Nutrition Estimation" columns). The error rates are universally high. The best model, Gemini-3-Pro-Preview, achieves Average MAPE of 24.45% — meaning predictions deviate from ground truth by roughly one-quarter on average. The second-best, Gemini-3-Flash-Preview, achieves 25.21%. These errors are clinically meaningful: a 24.45% error on a 670.5 kcal average dish (Figure 3 mean) represents approximately 164 kcal, enough to substantially affect dietary tracking. Among proprietary models, GPT-4o (42.43%), GPT-4o-mini (44.08%), and Claude-Sonnet-4.5 (42.62%) perform notably worse, with errors roughly double those of the leading Gemini models. Among open-source models, Qwen-3-VL-30B-A3B-Instruct leads at 37.35% Average MAPE — far behind the Gemini leaders — while InternVL-3.5-14B struggles severely at 50.62%. The worst performers include Qwen-3-VL-4B-Thinking (56.88%) and InternVL-3.5-4B (51.16%).

The per-component MAPE breakdown (Table 2, columns for Cal, Prot, Carbs, Fat MAPE) reveals an interesting pattern: Protein estimation is consistently the most accurate component across models (Gemini-3-Pro-Preview achieves 23.88% MAPE on Protein vs. 27.40% on Carbohydrates, 26.53% on Fat, and 19.99% on Calories), while Carbohydrate estimation is systematically the worst (averaging ~40-60% MAPE even for strong models, and up to 72.59% for InternVL-3.5-8B). This suggests that VLMs can roughly estimate protein quantity (perhaps by identifying visible protein sources like meat, tofu, or eggs) but struggle to estimate carbohydrate content, which may be "hidden" in sauces, batters, or less visually salient components like rice and noodles whose volume is difficult to judge from 2D images.

VQA results (Table 2, "VQA" column). Performance is substantially higher on VQA than on the other tasks. Gemini-3-Pro-Preview leads at 90.42%, followed by Gemini-2.5-Pro (89.93%) and Gemini-3-Flash-Preview (88.56%). Among open-source models, Qwen-3-VL-30B-A3B-Instruct achieves 80.60%, Qwen-2.5-VL-72B-Instruct achieves 76.62%, and Mimo-VL-7B-RL achieves 75.00%. The worst performers include Qwen-2.5-VL-3B-Instruct (47.64%) and MiniCPM-V-4.5 (58.21%). The relatively high top-end VQA scores (near 90%) compared to classification (near 82%) suggest that dietary reasoning is easier for current VLMs than fine-grained visual discrimination — models can reason about food concepts even when they cannot perfectly distinguish visually similar dishes. However, the gap between the best proprietary model (90.42%) and best open-source model (80.60%) persists, and the very low scores for small models (47.64% for 3B Qwen) indicate that dietary reasoning ability scales strongly with model capacity.

Impact of Multi-View Imagery

The multi-view experiments (Figure 4) test a specific hypothesis: that providing additional images of the same dish from different angles improves model performance, and that this improvement varies by model scale. The experiments vary input image count from 1 to 4 for GPT-4o and Qwen-3-VL series models on Classification and Nutrition Estimation.

Classification accuracy trends (Figure 4, left panel). For GPT-4o, accuracy increases from approximately 60% (1 image) to roughly 65% (4 images) — a 5 percentage point gain. For Qwen-3-VL-30B-A3B, the improvement is sharper, from approximately 58% to roughly 66%. Both large models show monotonic improvement, with the largest gain occurring between 1 and 2 images (the "capability jump" the paper identifies). For smaller models (Qwen-3-VL-4B, Qwen-3-VL-8B), the trends are less consistent: Qwen-3-VL-4B shows approximately 57% at 1 image, rising to roughly 59% at 2 images, then declining to roughly 58% at 4 images — suggesting information overload.

Nutrition estimation trends (Figure 4, remaining panels). The MAE, RMSE, and MAPE plots for Nutrition Estimation show more complex patterns. For GPT-4o, MAE decreases from approximately 110 to roughly 100 as images increase from 1 to 3, but then plateaus or slightly increases at 4 images. For Qwen-3-VL-4B, the MAPE pattern is erratic: MAPE decreases from 1 to 2 images, then increases substantially at 3 images, then decreases again at 4 images — exactly the "instability" the paper diagnoses. The larger Qwen-3-VL-30B-A3B shows more consistent improvement, with MAPE declining steadily from roughly 43% at 1 image to roughly 38% at 4 images.

The key comparative finding. The paper claims that "large-scale models... demonstrate consistent improvements" while "smaller models exhibit instability, with MAPE fluctuating or degrading as more images are added." The Figure 4 curves support this qualitatively, though the paper does not provide statistical tests of the difference in trend between model scales. The finding is presented as an observation about the shape of the curves rather than a hypothesis test.

Effectiveness of Chain-of-Thought Prompting

The CoT experiments (Figures 5, 9, 10 and Section 4.4) evaluate whether the structured three-step reasoning template (visual observation → ingredient breakdown → comprehensive answer) improves or degrades performance compared to direct prompting. The paper's finding is that CoT is not universally beneficial and can be actively harmful, particularly for direct visual perception tasks and for smaller models.

Classification results (Figure 9). The radar chart comparing Classification Accuracy with and without CoT shows that most models experience a decline in accuracy when CoT is enabled. GPT-4o drops from roughly 66% to approximately 62%. Qwen-3-VL-30B-A3B-Instruct drops from roughly 66% to approximately 64%. Even Gemini models show small declines. The direction is consistent — across all models shown in the radar chart, none exhibits a CoT benefit on classification. This is evidence that the explicit verbalization scaffold interferes with the direct visual matching required for fine-grained discrimination.

Nutrition estimation results (Figure 5). The CoT effect on Nutrition Estimation MAPE is more dramatic and scale-dependent. For larger models (GPT-4o, Qwen-3-VL-30B-A3B), CoT produces modest changes — MAPE increases slightly for some, decreases slightly for others. But for smaller models, the impact is catastrophic: Qwen-3-VL-4B-Instruct's MAPE increases from roughly 43% (without CoT) to approximately 57% (with CoT); Qwen-3-VL-8B-Instruct's MAPE increases from roughly 39% to approximately 52%. The paper describes this as a "performance collapse" (Section 4.4). The radar chart visual (Figure 5) makes this visually stark — the "With CoT" polygon extends far outward (higher error) for small models compared to the "Without CoT" polygon.

VQA results (Figure 10). The radar chart for VQA shows mixed effects: some proprietary models (Gemini variants) show gains with CoT, while others (GPT-4o-mini, smaller Qwen variants) show degradation. The paper describes this as "inconsistent" and concludes that CoT efficacy is "heavily constrained by the model's fundamental visual grounding capabilities." This is consistent with the interpretation that CoT helps when the model's visual perception is already adequate and the reasoning step adds value (VQA scenarios requiring multi-step inference), but hurts when perception is flawed and the reasoning chain amplifies errors.

The cross-task pattern. The CoT results across the three tasks reveal a gradient: most harmful for Classification (universal degradation), strongly harmful for Nutrition Estimation especially for small models, and mixed-to-slightly-beneficial for VQA for some models. This gradient maps to the paper's difficulty hierarchy — the more a task demands direct visual discrimination (Classification being pure perception, Nutrition Estimation being perception + quantification, VQA being perception + reasoning), the more CoT's verbalization interferes with visual grounding.

Performance on the English-Translated Dataset

The English translation experiments (Table 4, Section B.3) evaluate 17 models on a version of DiningBench where all Chinese prompts and dish names were translated to English using Gemini-3-Pro-Preview with manual verification. This probes whether model performance is language-dependent, particularly for Chinese cuisine dishes where original Chinese names may carry specific culinary associations.

Classification declines on English prompts (Table 4, "Classification" column). Every tested model shows lower accuracy on the English version compared to the Chinese original (Table 2). Gemini-3-Flash-Preview drops from 81.83% to 77.67%; Gemini-3-Pro-Preview drops from 81.55% to 76.91%; GPT-4o drops from 65.26% to 62.38%. The Qwen series shows the largest relative declines: Qwen-2.5-VL-72B-Instruct drops from 65.29% to 62.21% (a 3.1 percentage point decline), while Qwen-2.5-VL-3B-Instruct drops from 51.49% to 50.94% (a smaller absolute decline given the lower baseline). The paper interprets this as a "semantic gap" — "models possess stronger multi-modal alignment for indigenous dish names encountered during pre-training, whereas translated English names may lack the specific cultural or visual associations required for fine-grained discrimination."

Nutrition estimation improves on English prompts for some models (Table 4, "Avg MAPE" column). This is the opposite pattern from classification. GPT-4o improves from 42.43% to 40.62% Average MAPE; Gemini-2.5-Pro improves from 38.21% to 34.20%; Gemini-2.5-Flash improves from 41.60% to 41.09% (slight). However, the pattern is not universal — Gemini-3-Pro-Preview worsens from 24.45% to 28.13%, and Gemini-3-Flash-Preview worsens from 25.21% to 27.83%. Qwen-3-VL-8B-Instruct improves notably from 39.24% to 38.81%. The paper suggests that English prompts may activate "more robust quantitative reasoning pathways or better grounding... potentially due to the dominance of English in their pre-training corpora for reasoning tasks."

VQA shows mixed results (Table 4, "VQA" column). Most models show declines on the English VQA, though the pattern varies. GPT-4o drops from 80.60% to 77.36%; Gemini-2.5-Pro drops from 89.93% to 87.06%. Qwen-2.5-VL-3B-Instruct shows a particularly severe decline from 47.64% to 40.67%.

Interpretation. The divergent language effects — English hurts classification but sometimes helps nutrition estimation — suggest that VLMs develop different types of capability associations with different languages during pre-training. Chinese dish names are tightly coupled to visual features of those dishes (because the model saw Chinese-labeled food images during training), while English may be more strongly associated with quantitative reasoning tasks (because English dominates math/science training data). This is a finding with practical implications: for food classification applications in Chinese contexts, using Chinese prompts yields better results, but for nutrition estimation, English prompts might be preferable for some model families.

Factors Contributing to Suboptimal Performance

Section 4.5 presents a qualitative error analysis identifying five primary failure modes, derived from manual inspection of model errors rather than quantitative categorization. The five factors are:

Limited Fine-Grained Discriminability. Models function as "bag-of-features" detectors that identify dominant components (color, broad ingredient types) but fail on subtle distinctions (cutting styles, textures, garnish details). The example of confusing Tomato Beef Pot with Spicy Beef Pot due to similar red color tones illustrates that models rely on high-level semantics over low-level visual details. This aligns with the classification results showing even top models at only ~82% accuracy despite only 8 options.

Parametric Knowledge Bias and Hallucination. Models default to statistically probable dish names from pre-training rather than the specific variant in the image. The example of misclassifying Scallion Oil Chicken as Roasted Chicken shows the model "ignoring contradictory visual evidence in favor of familiar text priors." This is a direct consequence of VLM training on internet-scale image-text pairs where generic dish names (Roasted Chicken) appear far more frequently than specific regional variants (Scallion Oil Chicken).

Deficiencies in Spatial and Volumetric Reasoning. High nutrition estimation errors stem from inability to perform 2D-to-3D inference — models cannot estimate mass or volume from 2D images, often treating "appetizers and main courses as nutritionally equivalent if they share visual textures." This manifests in the high MAPE values across all models, with even the best model at 24.45% error.

Ineffective Multi-View Aggregation. Confirming the quantitative findings from Figure 4, the qualitative analysis finds that models "struggle to synthesize complementary information or filter redundant features" when given multiple images, such that "increased visual context often acts as noise."

Inference Instability in Reasoning Models. The paper notes that "Thinking" variants of smaller models can "fall into 'infinite thinking loops'" — repetitive generation cycles triggered by visual uncertainty where the model fails to converge on a conclusion. This is a specific failure mode of the reasoning-optimized architecture when applied to ambiguous visual inputs.

Ablation Studies and Robustness Checks

English language translation: All 17 tested models show classification accuracy decline on the English version compared to Chinese (Table 4 vs. Table 2), with Qwen-series models exhibiting larger relative drops. Nutrition estimation shows mixed effects — some models improve (GPT-4o, Gemini-2.5-Pro) while others worsen (Gemini-3-Pro-Preview). This ablation reveals language-dependent capability associations in VLM pre-training data.

Chain-of-Thought effect per task: CoT prompting causes universal classification accuracy decline across all models tested (Figure 9), severe MAPE increase in small models for nutrition estimation (Figure 5, e.g., Qwen-3-VL-4B-Instruct from ~43% to ~57%), and mixed effects on VQA (Figure 10). This ablation demonstrates that CoT is not a universally beneficial intervention for VLMs in the food domain and that its effects are strongly scale-dependent.

Multi-view image count: Increasing images from 1 to 4 shows consistent improvement for large models (GPT-4o, Qwen-3-VL-30B-A3B) but instability or degradation for small models (Qwen-3-VL-4B, Qwen-3-VL-8B) on both classification and nutrition estimation (Figure 4). The largest gain occurs between 1 and 2 images, with diminishing returns thereafter. This reveals a capacity-dependent multi-view fusion capability in current VLMs.

Dataset quality external audit: Three independent PhD students reviewed 210 stratified random samples (70 per task) using the construction inclusion criteria, achieving a 100% approval rate (Appendix B.4). This validates that the filtering, generation, and verification pipeline produced high-quality samples, though the small audit sample (210 of 5,338 total samples across tasks) and low evaluator count (three, all from the same academic institution) limit the robustness of this quality claim.

Thinking vs. Instruct model variants: Qwen-3-VL "Thinking" models (which employ explicit reasoning at inference time) generally underperform their "Instruct" counterparts on Classification and Nutrition Estimation. For example, Qwen-3-VL-30B-A3B-Instruct achieves 65.43% Classification accuracy vs. 61.34% for the Thinking variant; on Nutrition Estimation, Instruct achieves 37.35% Average MAPE vs. 40.22% for Thinking (Table 2). This is consistent with the CoT harm finding — baked-in reasoning degrades visual discrimination and quantification.

Critical Assessment

Claim from the executive summary: "even the strongest model — Gemini-3-Pro-Preview — achieves only 24.45% average MAPE on Nutrition Estimation"

This claim is directly supported by Table 2, where Gemini-3-Pro-Preview's Average MAPE of 24.45% is indeed the best result across all 29 models. However, what the claim does not convey is that this 24.45% error is the average across four nutritional components, and the per-component errors vary substantially: the model achieves 19.99% MAPE on Calories but 27.40% on Carbohydrates. The averaging masks that the model is substantially better at estimating some nutrients than others. More importantly, the paper does not establish what MAPE would constitute "good enough" for practical dietary applications — is 24.45% clinically acceptable for a calorie-tracking app, or does it need to be below 10%? Without a utility-based threshold, the claim that nutrition estimation "remains an open challenge" is qualitative and unfalsifiable.

Claim: "fine-grained classification remains challenging despite 81.83% top accuracy"

This claim is supported but requires nuance. The 81.83% top accuracy from Gemini-3-Flash-Preview (Table 2) is substantially above the 12.5% random-chance baseline for 8-way classification, indicating that models are extracting genuine visual signal. However, the fact that GPT-4o — a model widely considered state-of-the-art for general vision — achieves only 65.26% validates that the hard-negative distractor strategy creates difficulty beyond general visual capability. The claim would be stronger with a comparison to how these same models perform on existing food classification benchmarks (Food-101, Food2K) — such a comparison would quantify how much of the apparent food recognition capability on standard benchmarks is due to semantic shortcuts that DiningBench's hard negatives eliminate. The paper does not provide this.

Claim: "current VLMs struggle with precise volumetric reasoning and subtle visual discrimination even when general reasoning capabilities appear strong"

This claim has strong evidential support from the cross-task pattern in Table 2. Models that perform well on VQA (a proxy for "general reasoning" in the food domain) do not necessarily perform well on classification or nutrition estimation. GPT-4o achieves 80.60% on VQA but only 65.26% on classification and 42.43% Average MAPE on nutrition. Claude-Sonnet-4.5 achieves 83.58% on VQA but only 54.40% on classification and 42.62% Average MAPE. This dissociation between reasoning performance and perceptual/quantitative performance directly supports the claim. However, the paper does not provide a formal statistical test of this dissociation (e.g., rank correlation between VQA and Classification scores across models, testing whether it differs significantly from zero), leaving the claim as a qualitative observation rather than a statistically established finding.

Claim: "CoT often degrades performance on visual discrimination tasks" (from Section 4.4)

This is the most strongly supported and practically consequential finding in the paper. Figures 5 and 9 show consistent CoT-induced degradation across models and tasks, with the effect being most severe for small models on nutrition estimation. The evidence is clear and the pattern is robust. However, the paper does not explore which specific aspect of the CoT template causes the degradation. The template has three steps (visual observation, ingredient breakdown, comprehensive answer). It is possible that the degradation comes from the length of the prompt (forcing models to generate long reasoning chains that consume context and shift attention), from the content of the template (the specific sub-tasks of visual observation and ingredient breakdown may bias the model toward particular features that are not discriminative for the final task), or from the format (the structured JSON-like output may conflict with the model's native reasoning style). Without disentangling these factors, the claim that "CoT harms visual discrimination" is correct as an empirical observation but underspecified as a diagnosis — we know it happens but not exactly why.

Missing experiments that would have strengthened the paper:

Performance on existing benchmarks for calibration. The paper does not evaluate the same 29 models on Food-101 or Food2K to quantify how DiningBench difficulty differs from standard classification. Without this, the claim that existing benchmarks are "overly simplistic" rests on qualitative argument rather than comparative evidence. A figure showing model accuracy on Food-101 vs. DiningBench classification would make the case quantitatively.
Per-cuisine performance breakdown. Table 3 shows that 69.1% of dishes are Chinese cuisine. The paper does not report whether models perform differently on Chinese vs. non-Chinese dishes within the same task. Given the English-translation experiments showing language-dependent performance, there is likely a cuisine-dependent performance pattern as well — models may be better at classifying Chinese dishes (which dominate their pre-training data if trained on Chinese platforms) than Western or Indian dishes. This would be important for understanding benchmark fairness and model generalizability.
Statistical significance and confidence intervals. The paper reports point estimates without any measure of uncertainty. For the multi-view experiments (Figure 4), where the claim is that small models show "instability" while large models show "consistent improvement," error bars or confidence bands would distinguish genuine trend differences from sampling noise. With only 2,884 classification samples and 1,650 nutrition samples, per-bin sample sizes in the image-count experiments could be small enough that apparent trends are not statistically reliable.
Inter-model agreement analysis. The paper evaluates 29 models independently but does not analyze whether models make the same errors. An error overlap analysis would reveal whether all models struggle on the same hard samples (suggesting benchmark-level difficulty that may require fundamentally better vision) or whether different models fail on different samples (suggesting model-specific weaknesses that ensemble methods could address).
Human performance baseline. The paper never establishes how well humans perform on DiningBench. Without a human baseline, it is impossible to calibrate what "81.83% classification accuracy" or "24.45% Average MAPE" means — is this near human-level, far below it, or somewhere in between? For nutrition estimation, human performance is known to be poor (errors of 30-50% are typical in studies of human calorie estimation from images), which would contextualize the model errors. For fine-grained classification, expert human performance (e.g., chefs or food critics) would provide an upper bound.
Effect of image selection strategy in multi-view experiments. The paper does not specify how images are selected when reducing from 5.27 average images per dish to exactly 1, 2, 3, or 4 views. If the single-image condition always uses the highest-quality merchant reference image, the measured benefit of additional views may be underestimated (since the baseline is already strong). If images are randomly sampled, results could vary substantially across random seeds. This is a methodological detail that affects the reproducibility and interpretation of the multi-view findings.
Robustness to prompt phrasing. All classification prompts use a single English template translated from Chinese. The English-translation experiments (Table 4) show that language matters. The paper does not test whether minor prompt variations (different wording, different instruction format) affect model performance — this is relevant because VLM evaluation is known to be sensitive to prompt engineering, and the reported rankings could shift with different prompt templates.

6. Limitations and Trade-offs

Limitation 1: Cultural Skew Toward Chinese Cuisine — Generalizability Across Global Culinary Traditions Is Unverified

The assumption or constraint. DiningBench sources its data from Meituan, a Chinese local life service platform, resulting in a dataset where Chinese cuisine accounts for 69.1% of dishes (2,086 of 3,021, per Table 3). The paper explicitly acknowledges this in Section 8:

"the dataset exhibits a cultural skew towards Chinese cuisine due to the sourcing platform, potentially affecting generalization across underrepresented global culinary traditions despite our efforts to include international dishes"

While the dataset does include Western (286 dishes), Asian non-Chinese/Japanese (187), Japanese (118), Latin-American (48), and Indian (44) cuisines, these subsets are individually small — the next largest cuisine category after Chinese is Western at only 9.5% of the dataset. This is not a balanced international benchmark; it is a predominantly Chinese benchmark with international supplements.

The consequence. A practitioner evaluating VLMs for a food application outside China cannot confidently extrapolate from DiningBench results to their target cuisine distribution. Several specific failure modes are plausible:

Fine-Grained Classification difficulty may not transfer. The "hard negative" distractor strategy relies on same-merchant, same-category sampling. Chinese restaurant menus are structurally different from Western menus — Chinese restaurants often have large menus with many finely differentiated dishes within a single category (e.g., 8+ variations of stir-fried dishes differentiated by sauce, protein cut, and garnish), making the distractor strategy maximally effective. A Western restaurant typically has fewer dishes per category, and the visual differences between dishes within a category (e.g., salads) may be qualitatively different — more dependent on ingredient presence/absence than on subtle sauce and cutting differences. A model that achieves 81.83% accuracy on Chinese-dominated classification might perform substantially better (if Western distinctions are visually coarser) or worse (if the model's training data skews toward Chinese food imagery) on a Western-dominated benchmark.
Nutrition estimation may not transfer across cuisines. Nutritional profiles of Chinese dishes differ systematically from Western dishes — Chinese cuisine uses different cooking oils (peanut, sesame), different staple carbohydrates (rice vs. bread/pasta), and different protein sources (tofu, thinly sliced meat vs. whole cuts). A VLM that learns to estimate nutrition from Chinese food imagery may develop cuisine-specific heuristics (e.g., "shiny surface implies high oil content" may apply differently to a Western salad with vinaigrette vs. a Chinese stir-fry with wok-fried oil absorption) that do not generalize.
VQA reasoning may rely on cuisine-specific culinary knowledge. The Cuisine Technique questions probe cooking methods (wok-frying, steaming, braising) that are heavily represented in Chinese cuisine but may be less recognizable or differently visually cued in other cuisines. The Dietary Suggestion questions test knowledge of dietary restrictions (vegan, keto) that may have different cultural expressions — the visual cues for "contains animal products" differ between Chinese cuisine (oyster sauce, fish sauce are common hidden non-vegetarian ingredients) and Western cuisine (butter, cream, cheese).

What evidence exists in the paper. The English-translation experiments (Table 4) provide indirect evidence of cultural/language dependency. Classification accuracy drops for all 17 tested models when switching from Chinese to English prompts (e.g., Gemini-3-Flash-Preview drops from 81.83% to 77.67%). The paper attributes this to "stronger multi-modal alignment for indigenous dish names encountered during pre-training." If language alone (keeping the same images and same cuisine distribution) causes a 4-percentage-point drop, the effect of changing the cuisine distribution entirely would likely be larger. However, the paper does not report per-cuisine performance breakdowns — it does not tell us whether models perform differently on Chinese vs. non-Chinese dishes within the benchmark. This is a missing measurement that prevents quantifying the generalizability risk.

The geographic diversity breakdown (Table 3) confirms the skew numerically: Western, Worldwide, Asian, Japanese, Latin-American, and Indian cuisines collectively account for 935 dishes (30.9%), but individually none exceeds 286 dishes (Western). The Fine-Grained Classification subset, with 2,884 samples spanning 1,977 categories (Table 1), likely inherits this skew proportionally, meaning the number of non-Chinese classification samples may be insufficient for statistically reliable per-cuisine evaluation.

Mitigation status. The paper acknowledges the limitation in Section 8 but makes no attempt to mitigate it through experimental design. There are no per-cuisine performance analyses, no weighting schemes to ensure balanced evaluation across cuisines, and no discussion of how future work might construct cuisine-balanced benchmarks. The limitation is flagged as a transparency note rather than actively addressed.

Limitation 2: LLM-Assisted Nutritional Ground Truth — The Labels Themselves May Inherit Model Biases

The assumption or constraint. For dishes lacking merchant-provided nutritional labels (which the paper implies is the majority, though no fraction is reported), the ground truth values for Calories, Protein, Carbohydrates, and Fat are estimated by Gemini-3-Pro-Preview using the food image, ingredient list, and portion sizes as inputs (Section 3.2.3). The paper acknowledges this explicitly in Section 8:

"the reliance on LLM-assisted generation for distinct parts of the nutritional ground truth and distractor selection, although systematically verified by humans, may inevitably inherit latent biases or subtle inaccuracies"

The verification steps — cross-referencing against the USDA FoodData Central database, Atwater system consistency checks (Table 11), manual review, and the external audit with 100% approval — are quality gates, not independent measurements. They can catch mathematical impossibilities (calories not matching macronutrients within 10%) and gross visual-nutritional mismatches (800 kcal for a pure vegetable salad), but they cannot detect systematic biases in the estimation model that produce plausible-seeming but incorrect values.

The consequence. This limitation creates a circularity risk for evaluating models from the same family as the ground-truth generator. The best-performing models on Nutrition Estimation are Gemini-3-Pro-Preview (24.45% Average MAPE) and Gemini-3-Flash-Preview (25.21%) — both from the same model family that generated the ground truth labels. This is expected if the ground truth labels reflect Gemini's estimation biases: a Gemini model evaluated against Gemini-generated labels will achieve lower error than a non-Gemini model evaluated against the same labels, even if the non-Gemini model's estimates are more objectively accurate against true laboratory-measured nutritional values. The evaluation conflates "agreement with Gemini's estimation biases" with "nutritional estimation accuracy."

The paper does not disentangle this circularity. The Atwater checks can only flag internally inconsistent labels, not systematically biased but internally consistent labels. The USDA cross-referencing uses a database of generic food items (e.g., "stir-fried beef with broccoli") that may not match the specific restaurant preparation in terms of oil quantity, sauce composition, or portion size. The human verification can catch visually obvious mismatches but cannot determine whether a dish labeled at 670 kcal should actually be 580 or 760 kcal — human calorie estimation from images is notoriously unreliable, with errors of 30-50% being typical in the nutrition science literature.

What evidence exists in the paper. The paper provides no measurement of ground-truth accuracy. There is no comparison of LLM-estimated nutritional values against laboratory-analyzed values for a subset of dishes. There is no inter-annotator agreement study for the manual verification step. There is no analysis of whether Gemini models' apparent advantage on nutrition estimation is attributable to label-generation bias — for example, by comparing how Gemini and non-Gemini models rank on the subset of dishes where nutritional data comes from direct merchant extraction (no LLM estimation involved) vs. the LLM-estimated subset. The external audit (Appendix B.4) validated image quality, label correctness (correct dish name), and uniqueness (no multiple correct answers) — it did not validate nutritional value accuracy against an independent measurement.

Mitigation status. The paper is transparent about the limitation in Section 8 but does not mitigate it experimentally. The authors describe the LLM-assisted approach as a pragmatic necessity — "the combination of a powerful language model and comprehensive source materials helped ensure reasonable data reliability" (Section 3.2.3) — and suggest that the cross-referencing and manual checks provide sufficient quality. A partial mitigation would be to report per-method performance: accuracy on the subset of dishes with merchant-provided nutritional data (direct extraction) vs. LLM-estimated data. This would reveal whether model rankings are consistent across the two ground-truth sources. The paper does not provide this breakdown.

The external audit of 70 nutrition samples with 100% approval rate (Appendix B.4) is insufficient to address this limitation. The audit verified that samples pass the construction inclusion criteria, not that the nutritional values are accurate against an independent gold standard. An auditor checking that a dish labeled at 670 kcal "looks reasonable" is evaluating face validity, not measurement accuracy.

Limitation 3: The Difficulty Estimation Cost for Adaptive Inference Is Effectively Infinite — No Practical Path to Deployment

The assumption or constraint. DiningBench is a static evaluation benchmark, not a system with adaptive test-time compute allocation. However, the paper's architecture — hierarchical tasks at different cognitive complexity levels — implicitly suggests a vision where VLMs should adapt their reasoning strategy based on task difficulty. The multi-view experiments (Section 4.3) and CoT experiments (Section 4.4) test whether varying compute (more images, reasoning chains) improves performance, but they do so uniformly across all samples rather than adaptively per sample. The paper provides no mechanism for a deployed system to determine which task a given input requires, which level of multi-view fusion is needed, or whether CoT should be enabled.

This is not a flaw in the benchmark design — DiningBench is an evaluation instrument, and static evaluation is appropriate. But it is a critical limitation for any practitioner who reads the paper's results and asks: "Given these findings, how should I configure my VLM for a real dietary application?" The paper identifies that different models perform differently across tasks, that CoT helps in some regimes and hurts in others, and that multi-view fusion benefits depend on model scale — but it provides no decision framework for selecting configuration based on input characteristics in a deployed system.

The consequence. A practitioner deploying a VLM for a food application based on DiningBench insights faces three unresolved tradeoffs:

Task classification overhead. In a real application, the system does not know whether the user's query is a fine-grained classification problem ("Which salad is this?"), a nutrition estimation query ("How many calories?"), or a VQA reasoning question ("Is this keto-friendly?"). DiningBench evaluates these as separate tasks with separate prompts, but a deployed system must either (a) use a single unified prompt that works reasonably across all task types, (b) route queries to task-specific prompts based on some form of intent classification, or (c) always apply the most comprehensive prompt (including CoT, multi-view aggregation). DiningBench provides no guidance on which approach is best, nor does it evaluate any unified-prompt baseline across all three tasks simultaneously.
CoT configuration decision. The paper shows CoT harms classification (Figure 9), harms nutrition estimation especially for small models (Figure 5), and has mixed effects on VQA (Figure 10). A deployed system cannot know ex ante whether CoT will help or hurt for a given query — DiningBench evaluates CoT effects per-task, aggregated across all samples, but does not identify which specific sample characteristics predict whether CoT will help. A practitioner must either disable CoT globally (sacrificing potential VQA gains), enable it globally (risking classification and nutrition degradation), or develop a routing mechanism that DiningBench does not inform.
Multi-view image acquisition cost. The paper shows that additional images improve performance for large models (Figure 4), with the largest gain from 1 to 2 images. But in a deployed application, acquiring multiple images requires the user to photograph their dish from multiple angles — a significant UX friction. The paper does not evaluate whether the performance gain from a second image is worth the additional user burden, nor does it identify which sample types benefit most from multiple views (e.g., perhaps nutrition estimation benefits more from multiple angles than classification, since volume estimation requires depth cues).

What evidence exists in the paper. The task-separated results (Table 2) demonstrate exactly the configuration sensitivity that creates the deployment challenge: rankings shift across tasks, CoT effects are task- and scale-dependent, and multi-view benefits are model-scale-dependent. But the paper never evaluates cross-task generalization (e.g., does a model that performs well on classification with a specific prompt also perform well on nutrition estimation with a different prompt?) or configuration robustness (e.g., how much does classification accuracy degrade if the model is prompted with a CoT-style template that includes an unnecessary reasoning step?). The ablation studies (Sections 4.3, 4.4) isolate specific factors but do not address the combined configuration optimization problem.

Mitigation status. The paper does not address this limitation — it is a benchmark paper, and deployment engineering is outside its scope. However, the absence of even a discussion of this practical gap means the results are more useful for "understanding model capabilities" than for "building better food VLM systems." The paper provides diagnostic insights (what models are bad at) but no prescriptive guidance (what to do about it in production).

Limitation 4: Single Benchmark, Single Domain — The Difficulty Hierarchy May Not Transfer to Other Specialized Domains

The assumption or constraint. All experiments are conducted on DiningBench, which evaluates a single domain (food) on a single type of data (restaurant dishes from a single platform). The paper's core conceptual contribution — the three-level hierarchy of identification, quantification, and reasoning — is presented as a general evaluation framework, but it is validated only on food. The paper does not claim that the specific difficulty patterns (hard negatives degrading classification, CoT harming visual discrimination, multi-view benefits scaling with model capacity) generalize to other specialized domains. The limitation acknowledged in Section 2.1 is that "general-purpose benchmarks rarely capture the unique challenges of the food domain," but the reverse question — whether the food domain's challenges are representative of other specialized domains — is not addressed.

The consequence. A practitioner in a different specialized domain (medical imaging, industrial inspection, satellite imagery analysis, retail product recognition) cannot assume that DiningBench's findings transfer. Each specialized domain may have different relationships between the hierarchical task levels:

Medical imaging: The "hard negative" problem exists (distinguishing benign from malignant lesions of similar appearance), and the consequences of failure are far more severe. But the relationship between classification and quantification may differ — in radiology, classification (lesion present/absent) and quantification (lesion size, growth rate) are more tightly coupled than in food (ingredient recognition does not tightly constrain calorie estimation). The CoT degradation finding may not apply if medical VLMs are fine-tuned on domain-specific reasoning data that teaches them to ground verbal reasoning in visual evidence.
Industrial inspection: Defect classification (scratch vs. crack vs. corrosion) is fine-grained and safety-critical, but the "multi-view" problem is solved differently — industrial cameras are fixed-position with controlled lighting, making multi-view fusion a calibration problem rather than a user-behavior problem. The nutrition estimation analog (quantifying defect severity) would require different metrics.
Retail product recognition: Distinguishing visually similar products (different brands of the same product type) maps well to DiningBench's hard negative strategy. But the "VQA" analog might involve questions about product attributes or usage instructions that are more text-dependent than visually-dependent, potentially changing how CoT interacts with performance.

Without replication across domains, DiningBench's findings are food-specific empirical observations rather than general principles of specialized-domain VLM evaluation. The paper's claim to "drive the next generation of food-centric VLM research" (Abstract) is appropriately scoped, but the broader implication — that hierarchical task design with hard negatives, quantification tasks, and visual-agnostic reasoning questions is a useful paradigmatic template — remains untested.

What evidence exists in the paper. The paper provides no cross-domain evidence. All 29 models are evaluated only on food tasks. The geographic diversity analysis (Table 3) shows cuisine diversity within the food domain, not domain diversity. The related work section (Section 2.1) discusses general multimodal benchmarks (MMBench, SEED-Bench, MME) but does not compare DiningBench's difficulty patterns to those benchmarks' difficulty patterns — for example, it does not report whether models that struggle with DiningBench's hard-negative classification also struggle with fine-grained classification in other domains.

Mitigation status. Not addressed. The paper's scope is explicitly food-domain. The limitation is therefore about the strength of claims that can be drawn rather than about a flaw in the benchmark. The paper appropriately scopes its claims to food-centric VLM research. However, the conceptual framework (hierarchical task design, hard negatives, multi-view evaluation) is presented as a contribution that could generalize, and the absence of cross-domain validation means this remains an untested hypothesis. A future multi-domain benchmark suite using the same hierarchical structure (identification, quantification, reasoning) across food, medical, industrial, and retail domains would test this.

Limitation 5: Missing Human Performance Baseline — Model Results Are Uncalibrated Against Human Capability

The assumption or constraint. DiningBench evaluates 29 VLMs but provides no human performance measurements on any of the three tasks. The paper never establishes what performance level constitutes "good enough," "near human-level," or "superhuman." The abstract and conclusion describe the benchmark as "challenging" and note that models "struggle significantly," but these characterizations are relative — the models struggle compared to what? Without human baselines, the claim that 81.83% classification accuracy is "challenging" is uncalibrated. A human might achieve 75%, in which case Gemini-3-Flash-Preview is superhuman; or a human expert (chef, food critic) might achieve 95%, in which case the model is substantially below expert performance.

The consequence. This calibration gap matters for practical deployment decisions in two ways:

Safety threshold determination. For a dietary guidance application, the acceptable error rate on fine-grained classification depends on the consequences of misclassification. If the human error rate on distinguishing visually similar salads is 10% (one in ten salads misidentified), a model at 81.83% accuracy (one in five misidentified) is worse than human and likely unsafe for autonomous use. If human error is 30%, the model is substantially better. DiningBench provides no basis for making this determination.
Nutrition estimation calibration. Human calorie estimation from images is known to be poor — studies in nutrition science consistently find errors of 30-50% when humans estimate calories from food photos. Gemini-3-Pro-Preview's 24.45% Average MAPE might therefore represent a substantial improvement over human capability, which would reframe the finding from "models struggle" to "models outperform humans but are still not accurate enough for clinical use." Alternatively, if expert dietitians using structured estimation protocols achieve 15% MAPE, the model is below expert performance. The paper provides no context for interpreting what 24.45% MAPE means in human terms.

What evidence exists in the paper. The paper provides no human evaluation data. The external audit (Appendix B.4) evaluated dataset quality (whether samples meet construction criteria), not human task performance. The qualitative error analysis in Section 4.5 diagnoses model failure modes but does not compare them to human error patterns — for example, it does not report whether humans also confuse Tomato Beef Pot with Spicy Beef Pot, or whether human nutrition estimates show the same Carbohydrate estimation deficit that models exhibit. The paper cites no external human performance data from the nutrition science or food recognition literature to contextualize its model results.

Mitigation status. Not addressed. The paper does not suggest human baselines as future work, nor does it discuss the absence as a limitation. This is a significant gap for a benchmark paper — the standard practice in the VLM evaluation literature (following MMLU, MMBench, and others) is to include human performance as a calibration point. The omission limits the paper's ability to make claims about whether current VLM performance on food tasks is "close to usable" or "far from adequate."

Limitation 6: English-Only Prompt Engineering — Results Are Conditioned on a Single Prompt Template per Task

The assumption or constraint. All models are evaluated using fixed, English-language prompt templates (provided in Appendix C and illustrated in Figures 6, 7, and 8). The prompts are well-designed and task-appropriate, but they represent a single point in prompt space. The paper does not evaluate whether model rankings or absolute performance levels are robust to prompt variations — different wording, different instruction format, different output specification syntax (e.g., "Output only a letter" vs. "Respond with: A"), or different levels of detail in the task description. The English-translation experiments (Table 4) show that language alone causes substantial performance shifts (classification drops, nutrition estimation sometimes improves), which strongly implies that prompt phrasing — within the same language — would also affect results.

This is a consequential limitation because VLM evaluation is known to be sensitive to prompt engineering. The same model can produce substantially different outputs depending on the exact instruction phrasing, the presence or absence of formatting examples, the specification of output constraints ("only output a letter" vs. open-ended response), and even the ordering of options in multiple-choice tasks. The paper's fixed-prompt evaluation therefore measures model performance under a single prompting strategy — which may not represent the model's best possible performance (if the prompt is suboptimal for that model family) or its typical performance (if the prompt is unexpectedly well-suited).

The consequence. A practitioner using DiningBench results to select a VLM for deployment faces a confounding risk: model rankings may be prompt-dependent, not capability-dependent. For example, if the classification prompt's forced-choice format (output a single letter, no explanation) systematically disadvantages models that perform better with chain-of-thought (Qwen-3-VL-Thinking variants), then the benchmark understates those models' classification capability. Conversely, if a model family was explicitly trained on structured JSON output formats similar to the Nutrition Estimation prompt (Figure 7), it may have an advantage over models equally capable at nutrition estimation but less familiar with that specific output format.

This prompt sensitivity could explain some of the paper's own findings. The CoT experiments (Section 4.4) show that models perform worse with the CoT template than without it on classification and nutrition estimation. But the "without CoT" prompts (e.g., "Your answer should only be a single uppercase English letter... without any description, explanation, or content other than the option letter") are actively hostile to reasoning — they forbid the model from producing any intermediate output. A model that naturally "thinks" by generating reasoning tokens before outputting a final answer might perform poorly under this constraint not because it lacks classification ability, but because the prompt prevents it from using its native reasoning process. A fairer comparison would include an intermediate condition: allow reasoning but extract only the final answer for scoring.

What evidence exists in the paper. The English-translation experiments (Table 4) provide direct evidence of prompt sensitivity — the same images, same tasks, same models, but different language produce different results. The magnitude varies by model and task, confirming that performance is not purely a function of visual capability. The CoT experiments (Figures 5, 9, 10) show that adding a structured reasoning scaffold changes performance. These two sources of evidence together make a strong case that prompt sensitivity is real and significant in DiningBench's evaluation setting.

Mitigation status. Not addressed. The paper does not discuss prompt sensitivity as a limitation, nor does it perform any prompt robustness experiments (e.g., evaluating with 3-5 different prompt templates per task and reporting mean and variance). The prompts are described in Appendix C, which enables reproduction, but the paper does not acknowledge that results are conditioned on these specific prompts. For a benchmark intended to drive model comparison and selection, this is a significant methodological gap — practitioners need to know whether the reported rankings are stable or whether a different prompt choice could reverse them.

7. Implications and Future Directions

How This Work Changes the Landscape

DiningBench does not introduce a new model, training method, or architectural innovation. Its contribution is diagnostic infrastructure: a measurement instrument that reveals capability gaps that prior benchmarks systematically concealed. This type of contribution — a well-constructed benchmark — can shift a field's attention as effectively as a new algorithm, because it changes what researchers optimize for. Before DiningBench, the food-domain VLM community had no way to measure whether advances in general visual reasoning translated to the fine-grained, multi-view, quantitative demands of real dietary applications. After DiningBench, that measurement exists, and the results are sobering enough to redirect research priorities.

The shift is best characterized as a reframing of food understanding from a classification problem to a hierarchical capability spectrum. Prior benchmarks (Food-101, Food2K, Nutrition5K) implicitly defined "food understanding" as the ability to match images to labels, or in the case of nutrition datasets, to estimate values from controlled cafeteria imagery. DiningBench demonstrates that these operationalizations are insufficient — that a model achieving 90% on coarse-grained food classification may fail to distinguish visually similar dishes from the same menu (fine-grained classification), may misestimate calories by 25-40% (nutrition estimation), and may produce plausible-sounding but visually ungrounded dietary advice (VQA). The paper's three-level hierarchy (identification → quantification → reasoning) provides a conceptual vocabulary for discussing these distinctions that the field previously lacked.

The most consequential reframing concerns the relationship between general visual reasoning and specialized visual perception. The CoT experiments (Section 4.4, Figures 5 and 9) reveal a counterintuitive dissociation: explicit verbalized reasoning — widely believed to enhance complex task performance — actively degrades performance on tasks requiring direct visual discrimination. This is not a food-specific curiosity; it suggests a fundamental tension in VLM architecture between the pathways supporting linguistic reasoning and those supporting precise perceptual judgment. The finding that small models experience "performance collapse" with CoT on nutrition estimation (MAPE exploding from ~43% to ~57% for Qwen-3-VL-4B-Instruct, Figure 5) while large models show only modest degradation implies that this tension is capacity-dependent — larger models have sufficient representational resources to maintain visual grounding while generating reasoning chains, while smaller models experience destructive interference. This reframes CoT from a universal reasoning enhancer to a task- and scale-dependent intervention whose costs must be weighed against its benefits.

The paper also resolves a latent contradiction in prior work that the field had not explicitly recognized. Previous food benchmarks reported high accuracy on classification (Food-101 models exceeding 90%) while nutrition estimation datasets showed that even specialized models struggle with quantification. Were these findings contradictory? DiningBench's cross-task evaluation of the same 29 models on the same underlying data reveals they are not — they reflect the fact that classification and quantification are distinct capabilities with different scaling properties. Gemini-3-Flash-Preview leads classification at 81.83% but achieves only 25.21% Average MAPE on nutrition; Qwen-3-VL-30B-A3B-Instruct achieves strong relative performance on both (65.43% classification, 37.35% MAPE) but is far from the frontier on either. The cross-task rank instability (no model leads all three tasks) provides empirical evidence that these are genuinely separate capabilities rather than manifestations of a single "food understanding" construct.

Perhaps most importantly, DiningBench redirects research attention toward verifier/estimator robustness as the primary bottleneck for real-world dietary VLM deployment — a parallel to the observation in the LLM test-time compute literature that verifier over-optimization, not search algorithm sophistication, is the limiting factor for inference-time scaling. The finding that even the best model (Gemini-3-Pro-Preview, 24.45% Average MAPE) produces nutrition estimates that deviate from ground truth by roughly one-quarter on average means that no current VLM is accurate enough for autonomous dietary guidance. The qualitative error analysis (Section 4.5) identifies specific failure modes — "bag-of-features" detection, 2D-to-3D inference deficits, ineffective multi-view aggregation — that are not fixed by scaling model size alone (GPT-4o and GPT-5 achieve 42.43% and 32.17% Average MAPE respectively, dramatically worse than the Gemini leaders). This implies that better nutrition estimation will require architectural innovations (explicit depth estimation modules, ingredient-aware volumetric reasoning) rather than merely larger models trained on more data. The benchmark thus reshapes the research agenda: rather than pursuing incremental gains on Food-101 classification accuracy, the high-impact target is closing the nutrition estimation gap, starting from the 24.45% MAPE floor.

Follow-Up Research This Work Enables

Per-cuisine performance analysis to quantify the cultural skew limitation. The paper acknowledges that 69.1% of dishes are Chinese cuisine and that classification accuracy drops universally when switching to English prompts (Table 4). But it never reports whether models perform differently on Chinese vs. non-Chinese dishes within the same task. A critical follow-up would compute per-cuisine accuracy and MAPE for all 29 models across the seven cuisine categories in Table 3 (Chinese, Western, Worldwide, Asian, Japanese, Latin-American, Indian), testing whether the performance gap between Chinese and non-Chinese dishes is statistically significant and whether it correlates with model training data composition. The hypothesis — suggested by the English-translation results — is that models perform better on Chinese dishes because their pre-training data skews toward Chinese food imagery and text. Confirming this would quantify the generalizability limitation and motivate construction of cuisine-balanced benchmarks. Disconfirming it (no cuisine-dependent performance difference) would strengthen DiningBench's claim to international applicability despite the skew. The analysis is straightforward given the existing data and requires only per-sample metadata that the paper already possesses.

Disentangling the CoT degradation mechanism through template ablation. The paper demonstrates that CoT harms visual discrimination (Figures 5 and 9) but cannot distinguish whether the harm comes from the length of the reasoning chain (consuming attention budget), the content of the template sub-tasks (visual observation and ingredient breakdown may bias feature attention toward non-discriminative features), or the decoupling of final answers from visual evidence (the model becomes committed to its verbalized interpretation and overrides perception). A targeted follow-up would evaluate variants of the CoT template: (a) a "short CoT" that requests reasoning in 1-2 sentences rather than the full three-step scaffold, (b) a "silent CoT" where the model produces reasoning tokens but they are masked from the final answer generation (testing whether the act of reasoning vs. the consumption of output tokens causes the degradation), and (c) a "verification CoT" that asks the model to first answer directly, then check its answer with reasoning (reversing the order to test whether commitment effects drive the error). These ablations would identify which component of CoT is responsible, enabling principled design of reasoning strategies that preserve visual grounding.

Human performance calibration on all three tasks. The paper's most significant missing measurement is human baselines. A follow-up human study should recruit three groups — lay users (representative of dietary app users), domain experts (chefs, dietitians), and the original dataset constructors — and measure their performance on stratified random samples from each DiningBench task. For Fine-Grained Classification, human accuracy on the 8-way hard-negative task would calibrate whether Gemini-3-Flash-Preview's 81.83% is sub-human, near-human, or super-human. For Nutrition Estimation, lay human MAPE (likely 30-50% based on nutrition science literature) would contextualize Gemini-3-Pro-Preview's 24.45% — is this a substantial improvement over human capability, or still below expert dietitian performance with structured estimation protocols? For VQA, human accuracy would establish whether the 90.42% top model score represents near-ceiling performance or leaves room for improvement. The study should also collect human error-type annotations to compare with the model failure taxonomy in Section 4.5 — do humans exhibit the same "bag-of-features" errors, parametric knowledge bias, and volumetric reasoning deficits, or are these uniquely model failure modes?

Testing whether LLM-assisted ground truth biases Gemini model evaluation. The circularity risk in nutrition estimation ground truth (Gemini-generated labels evaluated against Gemini models) demands empirical investigation. A targeted experiment would identify a subset of dishes for which independent laboratory nutritional analysis is feasible (e.g., 50-100 dishes from chain restaurants with published nutrition information, or dishes that can be prepared and analyzed in a food science lab). Comparing model errors against both the LLM-assisted labels and the laboratory gold standard would reveal whether Gemini models' apparent advantage (24.45% vs. 37-50% for non-Gemini models) persists against objective measurements or is an artifact of label-generation bias. A finding that model rankings are consistent across ground-truth sources would validate the LLM-assisted approach and strengthen DiningBench's credibility. A finding that rankings invert would indicate that the benchmark's nutrition estimation results are unreliable for model comparison and that future versions must use independently verified ground truth.

Extending the hierarchical benchmark design to other safety-critical visual domains. DiningBench's three-level structure (identification, quantification, reasoning) with hard negatives, multi-view imagery, and visual-agnostic question phrasing is a methodological template that could be replicated in domains where fine-grained visual discrimination has safety implications. A natural follow-up would construct analogous benchmarks for medical imaging (dermatology: distinguish visually similar skin lesions, estimate lesion size and growth rate, answer treatment-relevant reasoning questions), industrial inspection (distinguish visually similar defect types, estimate defect severity/dimensions, answer root-cause reasoning questions), and species identification (distinguish visually similar protected vs. non-protected species, estimate population-relevant attributes, answer ecological reasoning questions). Each domain-specific benchmark would test whether the findings from DiningBench — hard negatives reducing accuracy, CoT harming visual discrimination, multi-view benefits scaling with model capacity — replicate across domains or are food-specific. A cross-domain comparison of the same 29 models would reveal whether fine-grained visual discrimination is a domain-general capability (models that excel on DiningBench also excel on dermoscopy) or domain-specific (performance is uncorrelated across domains, requiring domain-specialized training).

Investigation of multi-view fusion architectures informed by DiningBench's scaling results. The finding that small models degrade with additional views while large models improve (Figure 4) suggests that naive concatenation of visual tokens from multiple images hits a capacity bottleneck. A systems-level follow-up would benchmark dedicated multi-view fusion architectures (cross-view attention, learnable view pooling, iterative view selection, or latent 3D reconstruction modules) against the default concatenation approach on DiningBench's multi-view data, measuring whether architectural innovations can shift the "capability jump" curve — enabling small models to benefit from additional views rather than being overwhelmed by them. The experiment would use the same 1-to-4 image count sweep as Figure 4 but with architecture as an additional independent variable, testing whether a 4B-parameter model with explicit fusion can match or exceed a 30B-parameter model with naive concatenation. Positive results would motivate incorporating multi-view fusion into VLM pre-training objectives; null results would suggest that the bottleneck is fundamental representational capacity rather than architecture.

Practical Applications and Downstream Use Cases

Model selection and capability auditing for dietary application developers. The most immediate practical use of DiningBench is as a decision support tool for teams building food-related VLM applications. Table 2 provides a direct capability comparison across 29 models on three task dimensions, enabling developers to select models based on their specific application profile. A team building a calorie-tracking app would prioritize nutrition estimation performance — they would select Gemini-3-Pro-Preview (24.45% Average MAPE) or Gemini-3-Flash-Preview (25.21%) for accuracy, or Qwen-3-VL-30B-A3B-Instruct (37.35%) if open-source deployment is required, understanding that errors averaging roughly one-quarter of true values remain the state of the art. A team building an allergen-detection system would prioritize fine-grained classification (since distinguishing a dish with nuts from a visually similar dish without nuts is the critical capability) — they would note that even the best model (Gemini-3-Flash-Preview, 81.83%) misclassifies nearly one in five dishes under adversarial distractors, indicating that autonomous allergen detection is not yet safe without human verification. A team building a culinary education assistant would prioritize VQA performance — they would select Gemini-3-Pro-Preview (90.42%) or Gemini-2.5-Pro (89.93%), understanding that dietary reasoning accuracy is relatively high but that open-source alternatives (Qwen-3-VL-30B-A3B-Instruct, 80.60%) provide a viable lower-cost option. The English-translation results (Table 4) further guide language-specific deployment: if the target user base is Chinese-speaking, use Chinese prompts for classification; if English-speaking, accept a ~4 percentage point classification accuracy penalty but potentially improved nutrition estimation.

Calibrating safety margins for health-critical dietary AI. DiningBench's quantitative error measurements enable risk-aware deployment decisions that were impossible with prior benchmarks. A healthcare provider considering a VLM-powered dietary logging tool for diabetic patients can use the nutrition estimation metrics to bound expected errors: with the best current model (Gemini-3-Pro-Preview), calorie estimates deviate by approximately 20% (19.99% Cal MAPE, Table 2), protein by 24%, carbohydrates by 27%, and fat by 27%. For a patient logging a 670 kcal average meal (Figure 3 mean), the expected absolute error is roughly 130-180 kcal. An endocrinologist can assess whether insulin dosing based on estimates with this error magnitude is clinically acceptable or requires human verification. Similarly, the classification accuracy bounds (81.83% for the best model, 65.26% for GPT-4o) enable risk assessment for allergen-related misclassification: with ~18% error rate on hard negatives, approximately one in five visually similar dishes will be misidentified. This quantifies the need for human-in-the-loop oversight that the paper's Limitations section (Section 8) advocates — and provides the specific numbers that a risk assessment requires. Without DiningBench, developers had no principled way to estimate these error rates; with it, they can make evidence-based decisions about deployment scope and safety protocols.

Guiding CoT configuration for food-domain VLM deployments. The CoT experiments (Section 4.4) provide actionable guidance for system configuration: disable CoT for classification and nutrition estimation, consider enabling it only for reasoning-heavy VQA queries on large models where it shows modest benefits. A deployed food VLM system can implement task-specific prompting: when the user query is classified as a classification or quantification task (routed via a lightweight intent classifier), use direct prompting without reasoning scaffolds; when the query requires dietary reasoning, enable CoT for proprietary models (Gemini, GPT-4o) but disable it for smaller open-source models where Figure 10 shows degradation. This routing strategy is directly derivable from Figures 5, 9, and 10 and does not require additional research — it is an immediately deployable configuration optimization that the paper's ablation experiments justify. The finding that Qwen-3-VL "Thinking" variants generally underperform their "Instruct" counterparts on classification and nutrition (Table 2 — e.g., Qwen-3-VL-30B-A3B-Thinking at 61.34% classification vs. 65.43% for Instruct, and 40.22% vs. 37.35% Average MAPE) further supports selecting non-thinking variants for food applications unless the application's primary demand is open-ended reasoning.

Prioritizing investment in verifier/estimator robustness over general VLM scaling. The paper's finding that larger models do not consistently outperform smaller ones on nutrition estimation — GPT-4o (42.43% Average MAPE) is dramatically worse than Gemini-3-Flash-Preview (25.21%), and Qwen-3-VL-4B-Instruct (42.55%) nearly matches GPT-4o — implies that parameter count alone is not the solution to food-domain quantification. For organizations with limited R&D budgets, this suggests prioritizing investment in domain-specific verifier training (fine-tuning VLMs on nutrition estimation data with ground-truth labels from laboratory analysis or dietitian annotation) over acquiring access to ever-larger general-purpose models. The benchmark provides the evaluation instrument to measure whether such fine-tuning closes the gap: a fine-tuned Qwen-3-VL-8B could be benchmarked against DiningBench's nutrition estimation task, with Gemini-3-Pro-Preview's 24.45% Average MAPE as the target to beat. If fine-tuning can bring a small open-source model within striking distance of the proprietary frontier, the cost-benefit case for domain-specific adaptation over general scaling becomes quantitative rather than speculative.