DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

ArXiv: 2512.04324

🎯 Pitch

Even top-tier agents like GPT-5 achieve under 20% success on real-world data engineering tasks—not because they can't write code, but because they fail at holistic pipeline orchestration across multi-layered architectures. Meanwhile, open-ended business analysis tasks top out below 60%, revealing that engineering and analysis are distinct, equally stubborn bottlenecks that demand fundamentally different kinds of reasoning.

---

1. Executive Summary

This paper introduces DAComp, a benchmark of 210 tasks that evaluates LLM-based agents across the full data intelligence lifecycle—spanning repository-level data engineering (DAComp-DE) and open-ended data analysis (DAComp-DA)—using enterprise-scale SaaS schemas averaging 412 columns and pipelines requiring over 4,000 lines of code. DAComp-DE tests three distinct capabilities: designing engineering specifications (DE-Architecture), building multi-stage SQL pipelines from scratch (DE-Implementation), and modifying existing systems under new requirements (DE-Evolution), while DAComp-DA assesses open-ended analytical reasoning through hierarchical rubrics that enumerate multiple valid solution paths per problem and an LLM judge validated to match human expert agreement (κw = 0.834). Even state-of-the-art agents falter dramatically—GPT-5 achieves a strict Success Rate of only 20.00% on engineering tasks and a DA Score of 56.14% on analysis tasks—establishing that holistic pipeline orchestration and strategic insight synthesis, rather than isolated code generation, are the core bottlenecks, and that engineering and analysis constitute distinct, independently challenging capabilities.

2. Context and Motivation

The Core Problem: Benchmarks Don't Test What Data Agents Actually Need to Do

The fundamental gap this paper addresses is that existing benchmarks evaluate data agents on skills that don't reflect how data professionals actually work in enterprises. The paper diagnoses a systematic mismatch along two axes described in Section 1:

• The "Hard" axis (engineering realism): Real data engineering requires coordinating multi-stage SQL pipelines across layered architectures (staging, intermediate, marts layers) as a directed acyclic graph (DAG), managing dependencies across dozens of files, and evolving existing codebases when business requirements change. Existing benchmarks reduce this to generating a single SQL query or a single Python script in isolation.

• The "Soft" axis (analytical openness): Real data analysis starts with ambiguous business questions ("Why are high-value customers churning silently?"), requires strategic planning to determine what metrics to compute and what hypotheses to test, involves iterative exploration through SQL and Python, and demands synthesizing numerical results into decision-oriented reports with visualizations and actionable recommendations. Existing benchmarks reduce this to answering questions with deterministic, verifiable answers—treating analysis as a lookup problem rather than a reasoning problem.

This gap is not merely about scale. It's about whether test environments capture the cognitive demands of data work: planning before coding, reasoning about dependencies in complex systems, adapting to changing requirements, and translating quantitative results into qualitative judgments. The paper argues (Section 1) that "progress in data agents demands a shift from mere code accuracy to the nuanced capabilities—planning, open-ended reasoning, and systematic synthesis—required to deliver insights that are both analytically rigorous and strategically actionable."

Why This Problem Matters: The Enterprise Reality

The paper's motivation is grounded in how data intelligence actually functions in modern enterprises, as illustrated in Figure 1. The workflow is not a single step but a chain:

1. Raw data sources (Salesforce, Stripe, Marketo, Zendesk) contain fragmented, inconsistently formatted records.
2. Data engineering transforms these through a multi-layered pipeline:
   • Staging layer: Ingest raw data and apply cleaning operations—validity constraints (email format, phone numbers), consistency constraints (temporal logic like "end date must be after start date"), integrity rules (non-null foreign keys), and anomaly detection (statistical outlier checks, as categorized in Figure 3).
   • Intermediate layer: Apply complex business logic, entity integration, and reusable transformations that feed multiple downstream models.
   • Marts layer: Build final fact and dimension tables optimized for executive dashboards, KPIs, and predictive modeling.
3. Semantic layer: Present analysis-ready tables to data analysts.
4. Data analysis: Convert those tables into insights—not just numbers, but reports that identify problems, explain causes, and recommend actions.

This workflow has several properties that make it a demanding testbed:

• Repository-level scope: A typical DAComp-DE task involves ~2,300 lines of code across ~37 files (Table 2). An agent can't succeed by writing one correct query; it must orchestrate an entire system.
• Dependency management: If an intermediate table is incorrect, every downstream table that depends on it produces wrong results. The paper's Cascading Failure Score (CFS) metric was specifically designed to measure this propagation effect.
• Evolving requirements: In production, specifications change—a metric definition is revised, a new data source is added, a business rule is updated. Agents must modify existing systems without breaking working components.
• Open-endedness: When a business stakeholder asks "Why are we losing high-value customers?", there is no single correct answer. The analyst must decide what metrics to look at, explore multiple hypotheses, and synthesize findings into a coherent narrative.

The practical stakes are high because enterprises are actively trying to deploy LLM-based agents for these workflows. A benchmark that overestimates agent capability by testing only isolated skills will lead to premature deployment and costly failures. Conversely, a benchmark that underestimates capability by testing only narrow, unrealistic tasks will fail to differentiate between truly capable systems and those that merely excel at narrow benchmarks. DAComp aims to provide the missing calibration.

Where Prior Benchmarks Fall Short

The paper provides a detailed comparison with existing work in Table 1, which is worth examining closely because it reveals the specific dimensions of the gap. I'll walk through the major categories:

Agentic Benchmarks Beyond Data

Benchmarks like SWE-Bench (Jimenez et al., 2023), WebArena (Zhou et al., 2024), OSWorld (Xie et al., 2024), and BrowserComp (Wei et al., 2025) have advanced the field of autonomous agents by introducing repository-level tasks (SWE-Bench tests whether an agent can resolve real GitHub issues by editing code patches) and open-ended evaluation (BrowserComp uses answer-based scoring for deep research tasks). These benchmarks demonstrate that agents can operate in complex, realistic environments. However, none of them address the data intelligence lifecycle specifically. SWE-Bench evaluates general software engineering patches on ~32 lines of code, not multi-thousand-line SQL pipelines with layered dependencies. WebArena tests web navigation—following links, filling forms—not analytical reasoning. The data domain has its own distinct challenges: schema-level reasoning, data quality constraints, metric definitions, and the translation of business questions into computational workflows.

Data Agent Benchmarks: The "Single Query" Paradigm

The dominant paradigm in data agent evaluation has been text-to-SQL and single-script data science. Let's trace the progression:

BIRD (Li et al., 2023) and Spider 2.0 (Lei et al., 2024): These are the state-of-the-art text-to-SQL benchmarks. BIRD provides 12,751 queries over schemas averaging 54 columns, and Spider 2.0 pushes toward realism with schemas averaging 320 columns. However, both evaluate exactly one SQL query per task. There is no pipeline, no dependency graph, no layered architecture. An agent that can write one correct query on a 320-column schema is not necessarily capable of orchestrating 40 queries into a coherent pipeline where each query depends on the output of previous ones. The paper notes (Section 2.3) that DAComp-DE schemas average 412 columns across 32 tables, and Implementation tasks require building from scratch, averaging 4,612 lines of code across 43 files. This is not merely a linear increase in difficulty—it introduces qualitatively new challenges around dependency management, data lineage tracking, and pipeline-level debugging.

DS-1000 (Lai et al., 2023) and DA-Code (Huang et al., 2024): These evaluate data science code generation—DS-1000 with ~3.6 lines of code per task and DA-Code with ~85 lines. Both test single-script generation (one Python or SQL script per task). They do not test multi-file repositories, and they do not test the ability to evolve an existing codebase. Critically, they evaluate only whether the code runs and produces the correct output—they do not evaluate whether the agent planned the right analysis, explored the data intelligently, or communicated findings effectively.

DSBench (Jing et al., 2024): This benchmark moves closer to realism by requiring multiple scripts (N scripts) and incorporating data science workflows. However, its schemas average only 27 columns, and its evaluation is objective—there is a right answer, and the agent either gets it or doesn't. This misses the open-endedness that characterizes real data analysis, where the quality of the reasoning process and the clarity of the conclusions matter as much as numerical accuracy.

BIRD-CRITIC (Li et al., 2025) and BIRD-Interact (Huo et al., 2025): These are notable recent efforts that test SQL debugging and multi-turn interaction, respectively. They introduce elements of repair and conversational context, which are important real-world requirements. However, they still operate in the single-query paradigm—debugging one SQL query, not a pipeline of 40 interdependent queries. The debugging and interaction they test are local, not systemic.

Data Analysis Benchmarks: The Early Attempts

BLADE (Gu et al., 2024), DABStep (Egg et al., 2025), and KramaBench (Lai et al., 2025): These are the closest prior work to DAComp-DA, and they represent important steps toward evaluating data analysis as distinct from code generation.

• BLADE evaluates 259 tasks where agents produce analytical reports (~70–80 lines of code). It uses an LLM judge for evaluation, which is a key methodological innovation that DAComp adopts and extends. However, BLADE's schemas are small (10–12 columns on average), and its tasks are focused on isolated analysis rather than the full lifecycle that includes engineering and evolution.

• DABStep evaluates 450 multi-step reasoning tasks over data, but its evaluation is objective—the agent produces a single answer that is checked against a ground truth. This treats analysis as a question-answering problem, not an insight-generation problem.

• KramaBench evaluates 104 data science pipeline tasks (50–100 lines of code) with LLM-judge evaluation. It is the closest in spirit to DAComp-DA in testing multi-step workflows, but it shares the limitation of small schemas (13 columns on average) and does not evaluate the engineering side of the pipeline.

The paper's key observation (Table 1) is that no prior benchmark simultaneously tests both repository-level engineering and open-ended analysis, and no prior benchmark tests the full lifecycle from engineering specification through implementation, evolution, and analysis. This is the gap DAComp fills.

What Makes the Gap Critical

The paper argues that engineering and analysis are complementary but distinct capabilities, and that testing only one creates a dangerous blind spot. An agent that excels at writing individual SQL queries (high BIRD score) may fail catastrophically when asked to build a pipeline where each query's output feeds into the next, because it cannot reason about dependency graphs. An agent that excels at computing accurate statistics (high DS-1000 score) may fail when asked to translate those statistics into a recommendation that a business stakeholder can act on, because it cannot synthesize quantitative results into qualitative insights.

The empirical results bear this out. The paper finds that even GPT-5, which achieves a Component Score of 61.85% on DE-Implementation (Table 4)—meaning it writes mostly correct individual SQL files when given perfect upstream inputs—drops to a Cascading Failure Score of only 30.49%, because it cannot manage the dependencies between those files. The gap between component-level correctness and pipeline-level success is the key diagnostic that DAComp provides, and it is invisible to benchmarks that test only isolated code generation.

How DAComp Positions Itself

The paper positions DAComp not as an incremental improvement over existing benchmarks, but as a category-defining benchmark that introduces a new dimension of evaluation. This is evident in several design choices:

First, DAComp introduces repository-level data engineering with three task types (Architecture, Implementation, Evolution) that correspond to real roles and responsibilities in data teams. The Architecture task tests whether an agent can produce a design document from a high-level business requirement—essentially, can it plan? The Implementation task tests whether it can execute that plan from scratch—can it build? The Evolution task tests whether it can modify an existing system—can it maintain? This decomposition is important because it allows diagnosing where agents fail: is the failure in planning (bad design), execution (bad code), or adaptation (bad understanding of existing systems)?

Second, DAComp introduces open-ended analytical evaluation with validated LLM-judge rubrics. Rather than forcing analysis into a multiple-choice or single-answer format, the paper constructs hierarchical rubrics that decompose each analytical question into requirements, sub-requirements, and multiple valid solution paths (Figure 2, Table 11). This is a significant methodological contribution because it solves the evaluation challenge for tasks where no single correct answer exists. The paper backs this up with rigorous validation: human inter-rater agreement on rubric items achieves κw = 0.906, and the chosen LLM judge (Gemini-2.5-Flash) achieves κw = 0.834 against human experts (Table 7). This establishes that LLM-judge evaluation can be as reliable as human evaluation when guided by well-constructed rubrics.

Third, DAComp provides multi-metric evaluation that separates component quality from pipeline integrity. The three execution-based metrics—Component Score (CS, each node evaluated in isolation with perfect upstream inputs), Cascading Failure Score (CFS, nodes evaluated sequentially with actual upstream outputs), and Success Rate (SR, every node must be perfect)—create a diagnostic ladder (Section 2.2, Appendix A.1). If an agent has high CS but low CFS, the bottleneck is dependency management. If it has low CS, the bottleneck is fundamental code generation. If it has moderate CFS but zero SR, the bottleneck is consistency—the agent can write mostly correct code but cannot guarantee correctness across the entire pipeline. This diagnostic capability is what enables the paper to make the claim that "holistic orchestration is the core bottleneck" (Section 3.3), not just generic difficulty.

Fourth, DAComp is explicitly positioned as a shift in research focus. The paper states (Section 1) that it aims to "steer the community beyond mere technical accuracy, driving the evolution of truly autonomous and capable data agents for the enterprise." This is not a claim about a new model or method—it's a claim about what the field should prioritize: planning over isolated coding, dependency reasoning over single-query accuracy, and insight synthesis over numerical correctness.

The Implicit Argument About Evaluation Methodology

A meta-contribution of the paper is its argument about how to evaluate open-ended tasks. The hierarchical rubric design (Figure 2, Appendix A.3) is worth understanding because it represents a solution to a problem that has limited prior work in data agent evaluation.

The challenge: when a business question has multiple valid analytical approaches, how do you score an agent's output fairly? If you define a single correct answer, you penalize valid alternative approaches. If you use a generic "quality" rubric, you introduce scorer subjectivity and inconsistency.

DAComp's solution has three key properties:

1. Top-down decomposition: Each question is decomposed into requirements, each requirement into sub-requirements (standards), and each sub-requirement admits multiple valid paths—methodologically distinct, self-contained solution strategies. For example, in Table 11, the "Penetration Rate Analysis" standard could be satisfied by a sales-based approach (Path 1.1.A) or an orders-based approach (Path 1.1.B). Each path has its own completeness, accuracy, and conclusion items.

2. Path selection by best match: During scoring, the LLM judge identifies which path best matches the agent's output and applies only that path's criteria. This means an agent is not penalized for choosing Path A over Path B, as long as it executes its chosen path correctly. This is crucial: it rewards methodological soundness rather than conformity to a single expected approach.

3. Anchor values and pseudo-code for objectivity: For accuracy items, the rubric includes pre-calculated anchor values (e.g., "West-Consumer avg ≈ 29.72%") or pseudo-code specifications of the expected computation. This constrains the LLM judge's subjectivity—it's not asking "does this look right?" but "does this match the pre-computed value or follow the prescribed procedure?"

The paper also includes a fallback mechanism for genuinely unenumerated paths (Appendix F.1). When an agent's approach doesn't match any pre-defined path, the judge applies principle-based assessment: is the method appropriate, correctly executed, and soundly interpreted? This prevents the rubric from penalizing valid but unanticipated strategies while still maintaining scoring standards.

This evaluation framework is substantively different from prior LLM-judge approaches that use generic "rate the quality from 1-5" prompts. The hierarchical rubric provides structure that constrains the judge's task to atomic, verifiable decisions, which is why the human-model agreement is so high.

Reconciling the Two Axes

A subtle but important insight in the paper's positioning is that the "Hard" and "Soft" axes are not just different difficulty levels—they test qualitatively different cognitive capabilities. The DE tasks are "hard" in the sense of requiring systematic, large-scale code implementation with precise correctness constraints. The DA tasks are "soft" in the sense of requiring strategic reasoning and communication in addition to computation. An agent could be excellent at one and terrible at the other, and the paper's results confirm this: the code-specialized Qwen3-Coder achieves a competitive DE Score of 32.80% (Table 3) but collapses on DA dimensions like Readability (3.15%) and Visualization (1.93%) (Table 5). The ability to write correct SQL does not transfer to the ability to explain what the SQL revealed. By testing both axes in a single benchmark, DAComp forces the research community to confront the fact that a "data agent" needs both capabilities, and that optimizing for one may not improve the other.

3. Technical Approach

3.1 Reader Orientation

DAComp is a benchmark and evaluation framework—not an agent or model—that defines 210 structured tasks across the full data intelligence lifecycle and provides validated scoring protocols for assessing LLM-based agents on those tasks. The system it builds is a two-part evaluation infrastructure: one part measures deterministic correctness for repository-level data engineering (using execution-based metrics on real database pipelines), and the other measures open-ended analytical quality (using an LLM judge guided by human-authored hierarchical rubrics with pre-computed anchor values). The problem it solves is that existing benchmarks evaluate isolated skills (single queries, single scripts) under closed-form grading, making them unable to distinguish between an agent that writes correct code and an agent that orchestrates correct systems or produces strategically useful insights. The "shape" of the solution is a task suite + evaluation protocol pair, where the tasks escalate from design (Architecture) through construction (Implementation) to maintenance (Evolution) for engineering, and from planning through execution to interpretation for analysis, with evaluation designed to pinpoint exactly where in these chains agents fail.

---

3.2 Big-Picture Architecture (Diagram in Words)

The DAComp framework has five major components:

1. Task Definitions — The 210 structured problems spanning three DE task types (Architecture, Implementation, Evolution) and one DA task type (open-ended analysis), each defined with input artifacts (business requirements, schemas, existing codebases) and expected output specifications.

2. Data and Schema Infrastructure — 73 enterprise-scale SaaS schemas populated with relationally consistent synthetic data (~400 columns average, multi-layered into staging/intermediate/marts layers) that serve as the substrate on which agents operate.

3. Agent Interaction Framework — The harness (OpenHands CodeAct agent or the custom DA-Agent with a ReAct-style four-action space of BASH, CREATE_FILE, EDIT_FILE, TERMINATE) through which LLMs write code, execute queries, inspect files, and produce outputs within sandboxed environments.

4. Evaluation Engine — A dual-mode scorer: (a) execution-based evaluation for deterministic DE-Impl and DE-Evol tasks, using three metrics of increasing strictness (CS, CFS, SR) computed by running agent-generated SQL against gold-standard outputs in DuckDB; (b) LLM-judge evaluation for open-ended DE-Arch and DA tasks, using hierarchical rubrics with enumerated solution paths, anchored accuracy checks, and a Good-Same-Bad (GSB) comparison protocol.

5. Validation Infrastructure — The human-agreement studies (300 model responses, 7,000+ rubric items annotated by expert humans) that establish the LLM judge's reliability (κw = 0.834 for rubric items, ICC(A,1) = 0.890 for case-level scores) and cross-judge consistency (τb = 1.00 across multiple judge models).

Information flows as follows: an agent receives a task specification (a business question for DA, a data_contract.yaml for DE-Impl, a change request for DE-Evol) and a sandbox environment containing the relevant database and file system → the agent generates code, executes it, inspects results, and produces final artifacts (SQL pipelines, analytical reports, visualizations) → the evaluation engine scores these artifacts against the appropriate rubric or execution oracle → the framework produces per-dimension and aggregate scores at three granularity levels (item, case, model).

---

3.3 Roadmap for the Deep Dive

• First, the task formalization: what exactly is an agent being asked to do in each of the four task types, and what are the formal definitions that constrain the problem space. This establishes the "contract" that agents must satisfy.

• Second, the evaluation metrics for deterministic DE tasks: the three-metric ladder (CS, CFS, SR) and their mathematical definitions, because these metrics are what enable the paper's core diagnostic claim about orchestration being the bottleneck.

• Third, the evaluation framework for open-ended tasks: the hierarchical rubric design, path enumeration, anchor value strategy, and GSB protocol, because this is the paper's primary methodological contribution and the mechanism that makes open-ended evaluation reliable.

• Fourth, the LLM-judge validation methodology: the agreement metrics (κw, ICC, τb) and the experimental design that establishes the judge's reliability, because without this validation, the open-ended evaluation claims would be unsupported.

• Fifth, the annotation pipeline: how tasks, rubrics, and ground-truth solutions were constructed by the 8 expert annotators, because the quality of the benchmark depends on the rigor of this construction process.

• Sixth, the agent frameworks used for baselines: OpenHands and DA-Agent, their action spaces, and their sandbox configurations, because these are the experimental apparatus through which the paper's empirical claims are generated.

---

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and evaluation methodology paper whose core idea is that data agent evaluation must simultaneously test repository-level pipeline orchestration (systematic code generation across dependency graphs) and open-ended analytical reasoning (strategic planning, iterative exploration, and insight synthesis), and that scoring open-ended tasks requires structured rubrics with enumerated solution paths and pre-computed anchor values to achieve human-level reliability.

---

Task Formalization: What Agents Are Being Asked to Do

DAComp defines four task types, each with a formal specification that constrains the problem space. Understanding these formalisms is essential because they determine what capabilities are being tested and what failure modes the evaluation can detect.

DE-Arch (Architecture): Specification Generation

The task is modeled as:

$(S, C^{\star}) = \pi_{\text{de}}(Q_{\text{de}}, C_0, B)$

where $\pi_{\text{de}}$ is the agent's policy, $Q_{\text{de}}$ is the high-level business requirement expressed in natural language, $C_0$ is the initial repository (which may contain existing code or data structures), $B$ is the database, $S$ is the engineering specification the agent must produce, and $C^{\star}$ is the final DE repository (relevant only in later task types).

What it computes: The agent receives a business question like "Can we build a 'true performance profile' for each sales representative that measures not just sales volume but customer quality and long-term value?" (Appendix C.1) and must produce a design document—essentially a data engineering blueprint—that specifies the tables, columns, dependencies, and business logic transformations required to answer that question. The output $S$ is a structured document covering business alignment, technical feasibility, and design quality.

Why this form: The Architecture task isolates the planning capability from the execution capability. By scoring the specification independently of implementation, DAComp can determine whether an agent's failure to build a correct pipeline is due to bad planning (it designed the wrong thing) or bad coding (it designed the right thing but wrote buggy SQL). This decomposition is not tested by any prior data agent benchmark.

DE-Impl (Implementation): Pipeline Construction from Scratch

The task is modeled as:

$(S, C^{\star}) = \pi_{\text{de}}(Q_{\text{de}}, \emptyset, B)$

where $C_0 = \emptyset$ indicates an empty repository—the agent must build the entire pipeline from the ground up. The input is a detailed specification $S$ in the form of a data_contract.yaml that formalizes: model inventory and lineage, table and column schemas with constraints, declared grains and time windows, metric definitions with units and currency normalization, and data quality policies (Appendix E.2.2).

What it computes: The agent must generate a multi-layered SQL DAG. The staging layer ingests raw data and applies cleaning operations—validity constraints (email format, phone numbers), consistency constraints (temporal logic), integrity rules (non-null foreign keys), and anomaly detection (statistical outlier checks). The intermediate layer applies complex business logic and entity integration. The marts layer builds final fact and dimension tables optimized for analytics.

Why this form: The from-scratch construction task tests the agent's ability to coordinate large-scale code generation under precise contracts. The data_contract.yaml provides unambiguous requirements, so the evaluation can be fully deterministic—there are correct and incorrect outputs, and the scoring is execution-based rather than judge-based. This eliminates subjectivity from the Implementation evaluation and enables the diagnostic metrics (CS, CFS, SR) that reveal the orchestration bottleneck.

DE-Evol (Evolution): Modification Under Changing Requirements

The task is modeled as:

$(S, C^{\star}) = \pi_{\text{de}}(Q_{\text{de}}, C_0, B)$

where $C_0$ is now a non-empty, functioning repository. The agent receives a change specification $S$—for example, adding cost allocation fields, ROI metrics, and revenue recognition fields to an existing sales pipeline (Appendix C.3)—and must modify the repository to satisfy the new requirements while preserving existing functionality.

What it computes: The agent must identify which files to modify (discriminative scope identification), understand how changes propagate through the dependency graph, implement the modifications correctly, and verify that unmodified components still work. A typical Evolution task involves editing ~1,718 lines of code across ~13 files (Table 2).

Why this form: Evolution tests a fundamentally different capability from Implementation: the ability to understand and safely modify an existing system. The paper finds that even for top models, the discriminative task of identifying which files to change is harder than the constructive task of building from scratch—GPT‑5's SQL Omission rate is higher in Evolution (11.69%) than Implementation (5.18%) (Table 17). This finding would be invisible in a benchmark that tests only construction.

DA (Data Analysis): Open-Ended Insight Generation

The task is modeled as:

$O = \pi_{\text{da}}(Q_{\text{da}}, D)$

where $Q_{\text{da}}$ is the open-ended analytical question, $D$ is the analysis-ready data from the semantic layer, and $O$ is the agent's output artifacts—analytical reports, key insights, visualizations, and actionable recommendations. Critically, there is no single correct $O$; the task is inherently open-ended.

What it computes: The agent must plan an analytical strategy (what metrics to compute, what hypotheses to test), execute that strategy through SQL and Python, interpret intermediate results, and synthesize findings into a human-readable report with visualizations. The paper classifies DA tasks into five analytical objectives (Table 14): Descriptive (22% of tasks, e.g., "Analyze sales trends"), Diagnostic (20%, e.g., "Investigate root causes of volatility"), Strategic (30%, e.g., "Recommend which sales outlets to expand"), Pattern Recognition (18%, e.g., "Explore factors impacting diamond prices"), and Profiling (10%, e.g., "Identify characteristics of top performers").

Why this form: The open-ended design forces evaluation to assess the quality of reasoning, not just correctness of output. This is what distinguishes DAComp-DA from benchmarks like DABStep (which uses objective answer checking) or DSBench (which uses deterministic answers). The hierarchical rubric is the mechanism that makes this feasible—it decomposes open-endedness into discrete, scorable requirements while accommodating multiple valid solution paths.

Task statistics that characterize the benchmark's scale:

These scales are extracted from Table 2 and represent the average per task, meaning that individual tasks can be substantially larger. The DE-Impl tasks, in particular, are an order of magnitude larger in code scale than any prior data agent benchmark (compare to Spider 2.0's 104.6 LOC or DA-Code's 85 LOC in Table 1).

---

Evaluation Metrics for Deterministic DE Tasks: The CS-CFS-SR Ladder

For DE-Impl and DE-Evol tasks, where outputs are deterministic SQL pipelines, DAComp uses a three-tier execution-based evaluation that progressively increases in strictness. This ladder is the key diagnostic tool—it reveals not just whether an agent fails, but why.

Component Score (CS): Individual Node Quality

The CS evaluates each table (node) in the pipeline in isolation, using perfect (gold-standard) upstream inputs:

$S_{d,\ell} = \frac{\sum_{t \in T_{d,\ell}} w_{d,t} m_{d,t}}{\sum_{t \in T_{d,\ell}} w_{d,t}}$

where $S_{d,\ell}$ is the per-layer score for task $d$ and layer $\ell$, $T_{d,\ell}$ is the set of tables in that layer, $w_{d,t} \geq 0$ are per-table weights (which can be configured to emphasize business-critical tables over auxiliary ones), and $m_{d,t} \in \{0, 1\}$ is a binary match indicator determined by exact equivalence of both schema and data between the predicted and gold outputs, checked in DuckDB. The task-level CS is a weighted sum across layers:

$\text{CS}_d = 100 \cdot \sum_{\ell \in L} \alpha_\ell S_{d,\ell}$

where $\alpha_\ell \geq 0$ are layer weights satisfying $\sum_\ell \alpha_\ell = 1$, and the benchmark CS is the average across all tasks.

What it computes: For each table in the pipeline, the evaluator runs the agent's SQL with perfect upstream data (as if all dependencies were correct) and checks whether the output table matches the gold standard exactly in both schema and data. The CS is the weighted fraction of tables that pass this check.

Why this form: By providing perfect upstream inputs, CS isolates the agent's ability to write individual SQL transformations correctly. It answers: "If all dependencies were magically correct, can this agent produce the right output?" This distinguishes code-generation failures from dependency-management failures. The per-layer weighting and table weighting allow the benchmark to emphasize business-critical components over auxiliary ones, though the paper does not specify the exact weight configuration in the main text.

Cascading Failure Score (CFS): Pipeline Integrity Under Real Upstreams

The CFS evaluates nodes sequentially along the DAG, using the agent's actual upstream outputs rather than perfect inputs. For task $d$ with pipeline DAG $G_d = (V_d, E_d)$ and node weights $w_{d,j} \geq 0$, let $\text{Anc}_d(j)$ be the ancestor set of node $j$. The cascading indicator is defined recursively:

$s^{\text{CFS}}_{d,j} = m_{d,j} \prod_{k \in \text{Anc}_d(j)} s^{\text{CFS}}_{d,k}$

where $m_{d,j} \in \{0, 1\}$ is the node-level exact match under predicted upstreams. In plain language: a node scores 1 only if it is correct and all its ancestors (recursively) are also correct. If any upstream node is wrong, the current node is blocked (score 0) regardless of its own SQL quality. The task-level CFS is the weighted fraction of nodes that survive this cascading check:

$\text{CFS}_d = 100 \cdot \frac{\sum_{j \in V_d} w_{d,j} s^{\text{CFS}}_{d,j}}{\sum_{j \in V_d} w_{d,j}}$

The benchmark CFS is the average across all tasks.

What it computes: Starting from the source nodes (staging layer), the evaluator feeds each node's actual (agent-generated) output as input to its downstream dependents. When a node fails, all nodes downstream of it in the DAG are automatically scored 0, even if their individual SQL is bug-free, because they are operating on corrupted input data.

Why this form: The CFS operationalizes "dependency management" as a measurable quantity. The gap between CS and CFS—for GPT-5, CS = 61.85% vs. CFS = 30.49% (Table 4)—directly quantifies the cost of poor orchestration. Figure 10 provides a concrete illustration: in a simple three-node pipeline where node B fails, CS awards 2/3 (nodes A and C pass independently), but CFS awards only 1/3 (A passes, but B blocks C because C depends on B's output). The CFS is the metric that enables the paper's central claim that "holistic orchestration is the core bottleneck in data engineering" (Section 3.3).

Success Rate (SR): Perfect Pipeline Execution

The SR is the strictest metric—every single component must match perfectly:

$\text{SR}_d = \prod_{j \in V_d} m_{d,j} \in \{0, 1\}$

The benchmark SR is the fraction of tasks where every node is correct:

$\text{SR} = \frac{1}{|D|} \sum_{d \in D} \text{SR}_d$

For DE-Evol tasks specifically, the paper uses a slightly relaxed threshold: a task is considered successful if $\text{CFS}_d \geq 80$, acknowledging the difficulty of perfect pipeline integrity under cascading evaluation.

What it computes: The SR is a binary per-task metric—the agent either solves the entire pipeline correctly (all nodes pass the cascading check) or it does not. It is the most demanding metric and the one closest to "can this agent replace a data engineer for this task?"

Why this form: The SR reveals whether agents can achieve consistency across the entire pipeline. Even GPT-5, with its relatively high CS (61.85%), achieves an SR of only 20.00% (Table 4). The 10× gap between CS and SR for weaker models exposes that even when individual components are mostly correct, the probability that all components are simultaneously correct is very low. This is the metric that matters for deployment—a pipeline with even one broken node produces wrong business metrics.

Tolerance mechanisms in execution-based evaluation. To ensure fairness, the evaluator applies several leniency measures (Appendix A.1). It evaluates only key columns (business-relevant and computational columns), excluding time columns that might differ due to timestamps. All numerical columns are rounded to two decimal places before comparison. These tolerances prevent spurious failures from floating-point differences or timestamp granularity issues while still catching substantive errors.

Why this matters: Without these tolerances, any floating-point difference (e.g., 10.0000001 vs. 10.0) or timestamp mismatch would be scored as a failure, making the benchmark measure numerical precision rather than analytical correctness. The tolerances focus evaluation on whether the business logic is correct, not whether the implementation produces bit-identical floating-point representations.

---

Evaluation Framework for Open-Ended Tasks: Hierarchical Rubrics and LLM Judge

For DA and DE-Arch tasks, where no single correct output exists, DAComp uses an LLM judge guided by a hierarchical rubric. This framework has several interlocking components that together address the challenge of evaluating open-ended work reliably.

The Hierarchical Rubric Structure (Top-Down Decomposition)

The rubric decomposes each task into four nested levels (Figure 2):

1. Requirement (Req): The highest-level objective, directly corresponding to a core analytical request from the user. A typical DA task has 3.1 requirements (Table 2). For example, in Table 11, Requirement 1 is "Penetration & Profitability Analysis."

2. Standard (Std): A key analytical step or conclusion needed to fulfill the Requirement. Each requirement decomposes into multiple standards. For example, Requirement 1 decomposes into Standard 1.1 ("Penetration Rate Analysis") and Standard 1.2 ("Profitability Analysis"). DA tasks average 5.7 sub-requirements (standards) per task.

3. Path: A methodologically distinct and self-contained strategy for meeting a Standard. Each standard admits multiple valid Paths—for example, Standard 1.1 could be satisfied by a sales-based approach (Path 1.1.A) or an orders-based approach (Path 1.1.B). The paper emphasizes that Paths must represent distinct strategies, not incremental variations: analyzing all provinces vs. analyzing a subset is not two Paths because the latter is merely incomplete execution of the former (Appendix E.3.1). DA tasks average 12.7 valid Paths per task.

4. Sub-standard / Rubric Item: The smallest scorable unit, nested under a specific Path. Items are categorized into three types by the principle of dimensional separation (Appendix E.3.1):

   • Completeness items (14% of all items): Assess whether all required steps for the chosen Path were executed.
   • Accuracy items (66% of all items): Assess whether computational results are correct, verified against pre-calculated anchor values or pseudo-code specifications.
   • Insightfulness items (20% of all items): Assess whether reasonable conclusions were derived from the results.

What it computes (scoring workflow): The LLM judge receives the agent's output, the rubric, and the task definition. For each standard, it first identifies which Path best matches the agent's approach (path selection). It then scores only the items under that Path—other Paths' items are ignored. Each item is scored as a binary satisfaction check: $s_k \in [0, w_k]$, where $w_k$ is the item's maximum weight. The rubric score for a task is:

$\text{Score}_{\text{rubric}}(O, R) = \frac{\sum_{k=1}^{N} s_k}{\sum_{k=1}^{N} w_k}$

where $O$ is the agent's output, $R$ is the rubric, $N$ is the total number of scored items, $s_k$ is the score awarded for item $k$, and $w_k$ is the maximum possible score for that item. The result is a value in $[0, 1]$, normalized by the total possible points.

Why this form (path-based scoring): The path selection mechanism is the critical innovation. Without it, a rubric would either (a) prescribe a single correct approach (penalizing valid alternatives) or (b) use vague quality criteria (introducing scorer subjectivity). Path selection allows the rubric to be precise about what correct execution looks like (specific anchor values, specific steps) while accommodating methodological diversity. The path-based design also means that an agent is rewarded for executing its chosen strategy correctly, not for guessing which strategy the rubric authors expected.

Anchor Value Strategy for Accuracy Items

For deterministic computations where outcomes can be exhaustively pre-calculated, the rubric embeds numeric anchor values directly (Tier 1 in Appendix F.1). For example, in Table 11, item 1.1.A.2 states: "Calculations must match anchors (e.g., West-Consumer avg ≈ 29.72%)." The LLM judge compares the agent's computed value to this anchor and scores accordingly.

For computations with well-specified procedures but multiple equivalent derivations, the rubric provides pseudo-code anchors (Tier 2). These specify the expected computational steps—inputs, ordering, aggregation logic, null handling—without prescribing a single implementation path. The judge verifies that the agent's code follows these steps.

For genuinely open-ended analysis (Tier 3), the rubric uses principle-based assessment: Is the method appropriate for the objective and data? Is it correctly executed? Are conclusions soundly derived from the evidence? This tier is reserved for a minority of items and serves as a safety net for valid but unenumerated solution paths.

Why this form: The three-tier strategy progressively relaxes constraints as the task becomes less deterministic. Tier 1 provides absolute, reproducible checks where possible. Tier 2 constrains the procedure while allowing implementation flexibility. Tier 3 ensures that novel approaches are not penalized. The paper reports (Section 3.6) that this strategy achieves human-level agreement, confirming that the constraints are sufficient to guide the LLM judge reliably.

Good-Same-Bad (GSB) Protocol for Qualitative Dimensions

For three dimensions—Readability, Analytical Depth, and Visualization—scoring by item-level check is impractical because quality is inherently comparative. The GSB protocol addresses this by comparing the agent's output against five pre-provided baseline reports:

$\text{Score}_{\text{gsb}}(O, O_{\text{base}}) = \frac{\max(0, |G| - |B|)}{|G| + |S| + |B|}$

where $|G|$ is the number of baseline reports the agent's output is better than, $|B|$ is the number it is worse than, and $|S|$ is the number it is the same as. The score ranges from 0 (worse than or equal to all baselines) to 1 (better than all baselines).

What it computes: The LLM judge compares the agent's report against each of the five baseline reports along two sub-dimensions: Readability (conveying complex information concisely, appropriate visualizations, clear writing structure, concise language) and Analytical Depth (multi-dimensional analysis, professional causal reasoning, practical grounded conclusions, estimation of recommendation impact). Each comparison yields a Good, Same, or Bad judgment. The GSB score is the excess of Good over Bad judgments, normalized by the total number of comparisons.

Why this form: The GSB score converts subjective quality assessment into relative ranking against calibrated baselines. By pre-providing the baseline reports (generated by combining outputs from multiple LLMs), the protocol anchors the judge's expectations. The formula $\max(0, |G| - |B|)$ means that being "same as" the baselines earns zero credit—the agent only gains by being strictly better. This prevents score inflation from mediocre outputs that merely match the baselines.

The Final DA Score Aggregation

The overall DA score combines the rubric and GSB components:

$\text{Score}_{\text{da}} = \alpha \cdot \text{Score}_{\text{rubric}} + (1 - \alpha) \cdot \text{Score}_{\text{gsb}}$

where $\alpha = 0.6$ is the standard weight, placing 60% weight on objective technical correctness (rubric) and 40% on qualitative presentation (GSB). The paper validates this choice through sensitivity analysis across values $\alpha \in \{0.5, 0.8, 0.9\}$ (Table 9), finding that relative rankings remain invariant (τb = 1.00) across all settings, confirming that the choice of $\alpha$ does not bias the leaderboard.

Hierarchical Rubric Prompt Design

The LLM judge operates through a structured prompt (Appendix A.3.1) that enforces specific behaviors. The prompt instructs the judge to: (a) analyze each rubric item individually, (b) require explicit evidence from the agent's output before awarding points ("if evidence is missing, assign 0 points directly"), (c) never guess or make subjective assumptions, and (d) for negative-scoring criteria (rubric items that describe undesirable conditions), judge whether the agent meets the undesirable criterion rather than whether the output is generally good or bad.

Why this form: The evidence-first policy constrains the judge to base decisions on observable content rather than holistic impressions, which is a known weakness of unconstrained LLM-as-judge approaches. The negative-criterion handling (Example 2 in the prompt) addresses the subtle logical inversion where a "good" response should be scored false for a negatively framed criterion—without explicit instruction, judges may incorrectly score good responses as true for negative criteria because the response "looks good."

DE-Architecture Evaluation

DE-Arch tasks use a non-hierarchical rubric (no Path decomposition required, since architectural specifications have fewer valid alternatives) and do not incorporate the GSB component. The evaluation covers three dimensions (Appendix A.2): Business Alignment and Semantic Accuracy (does the blueprint correctly reflect the business logic?), Technical Feasibility and Structural Completeness (can it be implemented given available resources and dependencies?), and Design Quality (is the model well-structured with clear naming conventions and modular design?). The scoring uses the same evidence-first, item-by-item protocol as the DA rubric.

---

LLM-Judge Validation: Establishing Reliability

The paper conducts extensive validation to demonstrate that the LLM judge's scores are reliable proxies for human expert judgment. This validation is methodologically essential because the entire open-ended evaluation framework rests on the claim that an LLM can grade analytical work as accurately as a human.

Validation Dataset Construction

The validation dataset consists of 300 model responses generated by 8 distinct LLMs (including GPT-5, o3, Gemini-2.5-Pro, DeepSeek-V3.1, Qwen3-Coder, and others) on a subset of DAComp tasks. Expert human annotators manually scored these responses against over 7,000 specific rubric items and GSB document pairs, creating a ground-truth baseline. Inter-rater agreement among human annotators was computed first to establish an upper bound on achievable agreement.

Agreement Metrics at Three Granularities

The paper measures agreement at three levels, each aligned with the statistical nature of the scoring signals (Appendix B.4):

1. Item-level agreement (Krippendorff's α / Weighted κ): Rubric items are ordinal with heterogeneous weights, so the paper uses Krippendorff's α:

$\alpha = 1 - \frac{D_o}{D_e}$

where $D_o$ is the observed disagreement and $D_e$ is the expected disagreement. For GSB labels (categorical: Good/Same/Bad), the paper uses weighted Cohen's κ with quadratic penalties:

$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} O_{ij}}{\sum_{i,j} w_{ij} E_{ij}}$

where $O_{ij}$ is the observed contingency table, $E_{ij}$ is its chance expectation, and $w_{ij}$ are the quadratic penalty weights.

2. Case-level agreement (ICC(A,1)): Each DA task yields a numerical aggregated score. The two-way single-measure intraclass correlation coefficient for absolute agreement measures how closely the LLM judge's per-task scores match human scores:

$\text{ICC}(A, 1) = \frac{\text{MS}_R - \text{MS}_E}{\text{MS}_R + (k - 1)\text{MS}_E}$

where $\text{MS}_R$ is the between-target mean square, $\text{MS}_E$ is the residual mean square, and $k$ is the number of raters.

3. Model-level agreement (Kendall's τb): To validate that the LLM judge produces the same model ranking as human evaluators, the paper computes Kendall's τb between the human and LLM leaderboards:

$\tau_b = \frac{n_c - n_d}{\sqrt{(n_c + n_d + t_x)(n_c + n_d + t_y)}}$

where $n_c$ and $n_d$ count concordant and discordant pairs and $t_x$, $t_y$ correct for ties.

Why these three metrics together: Each metric answers a different question about the judge. Item-level agreement asks: "Do the human and the judge make the same decisions on individual scoring checks?" Case-level agreement asks: "Do they assign the same overall score to the same response?" Model-level agreement asks: "Do they rank the same models as better or worse?" A judge could achieve high case-level agreement (scores are similar) but low model-level agreement (the ordering of models is wrong), or vice versa. Using all three provides a comprehensive validation.

Validation Results (Table 7)

The human inter-rater agreement establishes the ceiling:

• Rubric Item: κw = 0.906 (near-perfect)
• Rubric Case: ICC = 0.925
• GSB Items: κw = 0.601 (Readability), 0.751 (Professional), 0.753 (Visualization)

The chosen judge (Gemini-2.5-Flash) achieves:

• Rubric Item: κw = 0.834
• Rubric Case: ICC = 0.890
• GSB Items: κw = 0.604 (Readability), 0.759 (Professional), 0.735 (Visualization)

Interpretation: The LLM judge achieves near-human agreement on rubric-based scoring (the main component, 60% weight) and comparable agreement on GSB-based scoring (40% weight). The Readability dimension shows the lowest agreement (κw ≈ 0.53–0.61) even among humans, reflecting its inherent subjectivity. The paper notes this variance but argues it is acceptable given that the overall DA score weighting (α = 0.6) emphasizes the more objective rubric component.

Cross-Judge Consistency (Table 8)

To rule out the possibility that the LLM judge exhibits family-specific bias (e.g., preferring outputs from its own model family), the paper tests multiple alternative judges (GPT-4.1, Gemini-2.5-Pro, Qwen3-235B, Qwen3-30B) and computes rank correlation with the primary judge. The result is τb = 1.00 across all alternatives except Qwen-235B (τb = 0.90), confirming that "the choice of judge model does not statistically alter the leaderboard" (Section 3.6).

Stochastic Stability (Table 10)

To assess reproducibility, the paper runs 8 independent grading runs on fixed agent outputs. The standard deviations of final scores are consistently below 0.35, demonstrating that the evaluation protocol yields stable grades despite the inherent randomness of LLM generation.

---

Annotation Pipeline: How Tasks and Rubrics Were Constructed

DAComp was constructed by 8 experts through a multi-stage pipeline designed to ensure realism, quality, and consistency. The pipeline differs for DE and DA tasks due to their different natures.

Data Collection (Stage 1)

The benchmark is grounded in permissively licensed assets (Apache-2.0, MIT). For DE tasks, the paper collects 73 enterprise-scale SaaS schemas with companion data transformation projects, averaging 400 columns each. These are populated with large-scale, relationally consistent synthetic data (Appendix E.1) generated through a pipeline that respects: (a) schema fidelity (PK/FK, uniqueness, not-null, and domain constraints preserved), (b) inter-column distributions and dependencies (e.g., country ⇒ currency/timezone correlations modeled), (c) temporal coherence (seasonality, trend, and holiday effects injected), and (d) noise and edge cases (controlled missingness, outliers, type coercions, duplicate "current" rows, currency conflicts, timezone mismatches) designed to expose pipeline fragility. For DA tasks, the paper curates 100 complex databases from the Web and supplements them with analytical modeling layers derived from DE-transformed data.

Task Design (Stage 2)

For DA tasks, annotators first draft 8 open-ended analytical questions per analysis-ready table. Five annotators then vote based on realism and difficulty, retaining the top 2, yielding the final 100 tasks. For DE-Evol tasks, practicing data engineers author new business requirements aligned with enterprise scenarios and professional standards—for example, adding cost allocation, ROI metrics, and revenue recognition fields to an existing sales pipeline (Appendix C.3). For DE-Impl tasks, selected SaaS transformation projects are reverse-engineered into a single data_contract.yaml that captures the full DAG and semantics (model inventory, lineage, schemas with constraints, grains, time windows, metric definitions, and data quality policies). For DE-Arch tasks, starting from the analytics layer of DE-Impl and DE-Evol examples, DA annotators propose 5 candidate business requirements per project, from which a data engineer selects 1 feasible yet challenging requirement.

Evaluation Construction (Stage 3)

For DA tasks, annotators build hierarchical rubrics as described in §3.4. At least 3 annotators work on each question, followed by alignment discussions to resolve discrepancies. The Path enumeration follows three principles (Appendix E.3.3 and F.1): (i) Paths must represent distinct, methodologically-sound strategies, not incremental steps or partial executions; (ii) deterministic outputs are validated against programmatically calculated anchor values (annotators must personally run the complete analysis before finalizing the rubric); and (iii) methodology-based soft constraints are used for unenumerated paths, evaluating appropriateness, execution correctness, and interpretation soundness.

A critical validation step ensures rubric comprehensiveness: outputs from five diverse LLMs are sampled, and annotators confirm that their enumerated paths can account for all observed solution strategies. This minimizes false negatives—valid but unanticipated solutions are not unfairly penalized because the three-tier accuracy design (Appendix F.1) includes a principle-based fallback for genuinely novel approaches.

For the GSB protocol, experienced data analysts author shared scoring criteria, and baseline reports are created by combining outputs from multiple LLMs.

For DE-Impl and DE-Evol tasks, solutions are deterministic: the paper implements execution scripts that automatically validate outputs against gold repositories in DuckDB, assigning partial credit at the node/layer level via the CS metric described above (Figure 10 illustrates the difference between CS, CFS, and SR scoring on a simple pipeline).

Why manual annotation over automated generation: The hierarchical rubric design with enumerated Paths and pre-computed anchors requires deep domain expertise—annotators must understand the analytical problem, identify methodologically distinct solution strategies, compute correct reference values, and anticipate valid but unenumerated approaches. Automated generation (e.g., prompting an LLM to produce rubrics) would risk producing incomplete or incorrect rubrics, undermining the evaluation's validity. The paper's inter-annotator agreement rates (κw = 0.906) and cross-LLM-judge consistency (τb = 1.00) are evidence that the manual annotation approach produces reliable evaluation instruments.

---

Agent Frameworks and Experimental Setup

The paper evaluates agents using two frameworks, with OpenHands serving as the primary harness for both DE and DA tasks and a custom DA-Agent baseline developed specifically for analysis tasks.

OpenHands (CodeAct Agent) Configuration

OpenHands (Wang et al., 2024) is adapted with a CodeAct agent for both DE and DA tasks (Appendix B.2). The sandboxed environment supports up to 200 rounds of tool interactions, with automatic termination if the agent repeats the same action three consecutive times or if any single action exceeds a 120-second timeout. The action space consists of three tools (Table 13): BASH for shell command execution (navigation, file inspection, script running), IPYTHON for Python-based operations, and TERMINATE for signaling task completion.

For complex DE-Impl tasks, OpenHands is extended to a multi-agent setup. Each agent is assigned a specific SQL task represented by a YAML specification. Agents can reference previously generated SQL statements, ensuring consistency. A dependency graph is established based on SQL relationships, with each agent operating in order according to this graph. Upon completing each SQL file, the agent is prompted to validate its output using a testing script, enabling error correction. A separate validation agent ensures the entire pipeline runs correctly. Each SQL-writing agent is constrained to 50 steps maximum, while the validation agent is allowed up to 100 steps.

DA-Agent Baseline

For data analysis tasks, the paper develops a custom agent framework (Appendix B.1) inspired by ReAct (Yao et al., 2022). The action space consists of four operations (Table 12): BASH (shell commands for file system interaction), CREATE_FILE (create a new file with specified content), EDIT_FILE (edit or overwrite existing file content), and TERMINATE (signal task completion). The agent iterates through a thought-action-observation loop within a sandboxed interactive file system. Termination conditions mirror OpenHands: three consecutive repeated actions or 120-second timeout on any single action.

Why two frameworks: The OpenHands baseline tests the same agent architecture across both DE and DA tasks, enabling cross-task comparison. The DA-Agent baseline provides a simpler, ReAct-style alternative that may be more accessible to the community and serves as an ablation on the effect of agent framework complexity. The paper reports results for both frameworks on DA tasks (Table 5), with DA-Agent showing slightly higher scores (e.g., GPT-5: 50.84% vs. 46.99% for OpenHands), suggesting that framework design matters for analytical tasks.

Model Selection and Configuration

The paper evaluates a range of models (Section 3.1): proprietary systems (GPT-5, Gemini-2.5-Pro, o3), open-source general models (DeepSeek-V3.1, Qwen3-235B-A22B, Qwen3-8B, Doubao-Seed-1.6, Kimi-K2), and code-specialized models (Qwen3-Coder). The selection is designed to span model scales (from 8B to proprietary frontier models) and specializations (general-purpose vs. code-focused), enabling analysis of how these factors interact with DE and DA task demands.

Evaluation Protocol Details

For DA evaluation, the paper standardizes on α = 0.6 for the rubric-GSB weighting and Gemini-2.5-Flash as the LLM judge, chosen for its balance of accuracy (highest Rubric Item κw = 0.834), cost efficiency, stability, and community accessibility (Appendix F.3). All baselines are evaluated with the same judge to ensure comparability.

Why Gemini-2.5-Flash over stronger models: The validation results (Table 7) show that stronger models (Gemini-2.5-Pro, GPT-4.1) achieve slightly higher agreement, but the marginal gain is small relative to the cost increase. The paper argues that a widely accessible, cost-efficient judge enables community adoption and reproduction of the evaluation pipeline, which is essential for a benchmark intended to drive field-wide progress.

4. Key Insights and Innovations

Innovation 1: The Orchestration-is-the-Bottleneck Diagnostic Framework

The paper's most conceptually distinctive contribution is not that data engineering tasks are hard — that much is obvious — but rather the decomposition of difficulty into two separable failure modes and the development of metrics that pinpoint which one dominates. Prior data agent benchmarks conflated code-generation skill with system-integration skill because they tested only isolated outputs. A low score on BIRD or Spider 2.0 could mean the agent wrote bad SQL, or it could mean the agent misunderstood the schema — but you couldn't tell which, and you certainly couldn't measure whether the agent could handle dependencies between queries because there were none to handle.

DAComp's three-metric ladder (CS → CFS → SR) changes this. The Component Score (CS) measures pure code-generation quality by providing perfect upstream inputs, isolating the question: "Can this agent write correct SQL for this specific transformation given correct inputs?" The Cascading Failure Score (CFS) then introduces dependency realism by feeding actual (potentially incorrect) upstream outputs to downstream nodes. The gap between CS and CFS is the cost of poor orchestration — it quantifies, in percentage points, how much performance is lost not from writing bad individual queries but from failing to coordinate them into a coherent system. For GPT-5, this gap is dramatic: CS = 61.85% vs. CFS = 30.49% on Implementation (Table 4). The agent writes mostly correct individual SQL files, but half of those correct files are rendered worthless because their inputs are corrupted by upstream failures.

This is a fundamental diagnostic contribution, not an incremental metric refinement. It transforms the research question from "How good are LLMs at data engineering?" (a coarse, unactionable question) to "Where in the pipeline does failure concentrate, and is it a code-quality problem or a dependency-management problem?" The answer — that dependency errors dominate across all models, with rates exceeding 65% even for GPT-5 (Table 16) — redirects research attention from improving SQL-generation accuracy (which prior benchmarks implicitly incentivize) to developing agents that can reason about data lineage, track upstream-downstream causality, and validate pipeline integrity holistically. The paper's error analysis (Section 3.3–3.4) shows that the Core (intermediate) layer is where the sharpest drop in pipeline survival occurs (Figure 5b), and that medium-scale edits — which involve nuanced business logic rather than trivial fixes or boilerplate additions — are paradoxically the hardest (Figure 4). These findings are actionable diagnostics, not just difficulty rankings.

The conceptual significance extends beyond data engineering. Any domain where agents must coordinate multiple interdependent code artifacts — multi-file software patches, multi-step scientific workflows, multi-table data transformations — faces an analogous orchestration challenge that is invisible to single-output benchmarks. The CS/CFS/SR ladder provides a template for testing orchestration capability in any such domain: measure component quality in isolation, then measure system quality under real dependencies, and the gap is the orchestration penalty.

---

Innovation 2: Path-Enumerated Hierarchical Rubrics as a Solution to Open-Ended Evaluation

Evaluating open-ended analytical work is a known hard problem. The dominant prior approaches were either (a) constraining tasks to have deterministic, verifiable answers (as in DSBench, DABStep, DS-1000) — which sacrifices ecological validity because real analysis is rarely answerable with a single number — or (b) using unconstrained LLM-as-judge with generic quality prompts (as in some early analysis benchmarks) — which introduces scorer subjectivity, instability, and potential self-preference bias. DAComp's hierarchical rubric with enumerated solution paths represents a methodological advance that navigates between these extremes: it preserves the open-endedness of real analytical tasks while constraining evaluation to atomic, verifiable decisions that achieve human-level reliability.

The key conceptual move is path-based scoring with path selection. Rather than defining one correct answer (which penalizes valid alternatives) or defining vague quality criteria (which invites inconsistency), the rubric explicitly enumerates multiple methodologically distinct solution strategies for each analytical sub-requirement. During scoring, the LLM judge identifies which path the agent followed and applies only that path's criteria. This means the rubric rewards methodological soundness rather than conformity to a single expected approach — an agent that correctly executes a sales-based penetration analysis gets full credit even if the rubric also includes an orders-based alternative.

What makes this more than just "we wrote a detailed rubric" is the three-tier accuracy strategy (Appendix F.1) that progressively relaxes constraints based on task determinism. Tier 1 embeds numeric anchor values for exhaustively pre-computable outcomes (e.g., "West-Consumer avg ≈ 29.72%"). Tier 2 provides pseudo-code specifications for well-defined procedures with multiple equivalent derivations, constraining the computational logic without prescribing implementation. Tier 3 uses principle-based assessment (methodological appropriateness, execution correctness, interpretation soundness) for genuinely open-ended cases. This tiered approach is sophisticated because it maximizes objectivity where possible while maintaining fairness where exhaustive enumeration is infeasible — and it provides a framework that can be applied to open-ended evaluation in any domain, not just data analysis.

The validation is what elevates this from a design proposal to a demonstrated contribution. The paper doesn't just assert that the rubric works; it measures agreement at three granularities (item, case, model) against a human baseline of over 7,000 rubric items scored by expert annotators (Table 7). The chosen LLM judge (Gemini-2.5-Flash) achieves Rubric Item κw = 0.834 and Case ICC = 0.890 — near-human levels — while cross-judge consistency across six alternative judges yields τb = 1.00 (Table 8), ruling out family-specific bias. The stochastic stability test (Table 10) confirms that scores are reproducible across runs (σ < 0.35). This validation suite is more rigorous than typical LLM-judge validation in the literature, and it establishes that structured rubrics with enumerated paths and anchors can achieve reliable open-ended evaluation without requiring human judges for every task.

The practical implication is significant: this rubric design methodology can be adopted by other benchmark creators facing open-ended evaluation challenges in domains like scientific reasoning, policy analysis, legal argumentation, or creative generation — anywhere that "correctness" is multi-dimensional and multiple valid approaches exist.

---

Innovation 3: Engineering and Analysis as Distinct, Independently Difficult Capabilities

Prior work in data agents implicitly treated "data tasks" as a monolithic category, or at best distinguished between text-to-SQL and data science code generation — both of which are ultimately code-generation skills. DAComp's architecture — testing repository-level engineering AND open-ended analysis in the same benchmark — reveals that these are qualitatively distinct capabilities that do not transfer automatically.

The evidence for this claim is in the differential performance patterns across task types. Qwen3-Coder, a code-specialized model, achieves a competitive DE Score of 32.80% (Table 3), rivaling general-purpose proprietary models like Gemini-2.5-Pro (32.88%). But on DA tasks, Qwen3-Coder collapses on qualitative dimensions: Readability = 3.15%, Visualization = 1.93%, Analytical Depth = 13.42% (Table 5). The model can write correct SQL — that's what it was optimized for — but it cannot synthesize those SQL results into a readable report with meaningful visualizations. Conversely, reasoning-focused models like o3 exhibit a pattern the paper calls "calculator behavior": achieving competitive Accuracy (40.99%) and Completeness (60.73%) on DA tasks, but severely underperforming on Readability (24.63%) and Analytical Depth (13.37%) — the model computes correct numbers but fails to translate them into human-consumable insights.

This is a reframing contribution: it changes the research agenda from "build better code-generating agents for data tasks" to "build agents that can both engineer systems AND reason analytically, recognizing that these are separate skills that must be developed together." The paper's design — testing Architecture, Implementation, Evolution, AND Analysis — operationalizes this reframing. An agent developer who only optimizes for BIRD accuracy (single-query SQL) or DS-1000 accuracy (single-script data science) may be optimizing for only half the capability profile needed in real enterprise settings.

The difficulty ladder across analytical objectives (Figure 8) reinforces this point. All models perform best on Descriptive tasks ("what happened?") and worst on Strategic tasks ("what should we do?"), with a 20–30 percentage point gap between these extremes for top models. This hierarchy — Descriptive > Diagnostic > Strategic — maps onto a cognitive sophistication gradient from factual reporting through causal reasoning to prescriptive decision-making. The fact that this gradient appears consistently across all models suggests it reflects fundamental difficulty rather than model-specific weaknesses, and it provides a roadmap for capability development: improving strategic reasoning and recommendation quality is the next frontier.

---

Innovation 4: The Error Cascade as a First-Class Phenomenon Requiring Systemic (Not Local) Solutions

A finding that emerges from the detailed error analysis (Section 3.4) is that errors in data pipelines propagate and compound in specific, measurable ways — and that this propagation, rather than any single type of local error, is what makes repository-level engineering hard. This is empirically demonstrated but conceptually significant because it challenges the implicit assumption in prior benchmarks that improving per-component accuracy is the path to system-level success.

Consider three pieces of evidence. First, the calculation logic error decomposition (Table 18): for GPT-5, upstream errors are approximately 3× more prevalent than intrinsic errors across all pipeline layers. In the Marts layer, GPT-5's upstream errors account for 30.41% of total errors vs. 10.05% for intrinsic logic errors. This means that even if GPT-5 wrote perfect SQL for every Marts table, it would still have a ~30% error rate because the errors originate earlier in the pipeline and propagate forward. Second, the dependency error breakdown (Table 19) shows a balanced distribution between missing and extra dependencies across DE-Impl tasks — agents are equally likely to fail to identify necessary upstream tables AND to hallucinate unnecessary dependencies. This symmetric pattern suggests a fundamental difficulty in constructing accurate global data lineage graphs, not a specific bug that can be patched with better prompting.

Third, and most revealing, is the inversion in SQL Omission rates between Implementation and Evolution for top models (Table 17). GPT-5's SQL Omission rate is higher in Evolution (11.69%) than in Implementation (5.18%). This is counterintuitive: Evolution tasks are smaller in scope (~950 LOC vs. ~2,300 LOC), yet they prove harder for scope identification. The discriminative task of figuring out which files to change in an existing codebase is cognitively different from the constructive task of building from a blank slate, and the paper's data shows that top models are actually better at the latter. This is a negative result with positive diagnostic value: it identifies discriminative scope identification as a specific capability bottleneck that wouldn't be visible in construction-only benchmarks.

The conceptual implication is that improving data agents requires systemic solutions, not just better SQL generation. Techniques like explicit dependency graph tracking, pipeline-level verification, automated impact analysis for change propagation, and fault-tolerance mechanisms that contain upstream errors before they cascade — these are qualitatively different from the code-generation optimizations that dominate current research. The paper's CFS metric operationalizes this insight: you can't just make each component better, you have to make the system resilient to component failures, and CFS directly measures whether you're succeeding.

This insight generalizes beyond data engineering to any domain where agents must maintain and evolve complex codebases — software engineering, scientific workflow management, infrastructure-as-code — and where the interaction between components matters more than the quality of any single component.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the DAComp benchmark, consisting of 210 tasks across four types: 30 DE-Architecture, 30 DE-Implementation, 50 DE-Evolution, and 100 DA tasks. The underlying data comes from 73 enterprise-scale SaaS schemas with multi-layered transformation projects, populated with relationally consistent synthetic data (averaging ~400 columns per schema for DE tasks and ~85 columns across ~4 tables for DA tasks, per Table 2). The benchmark includes both an English version (DAComp) and a Chinese adaptation (DAComp-zh) with an identical task set. The construction pipeline involved 8 expert annotators through a multi-stage process described in Section 2.3 and Appendix E, with DA tasks further validated by sampling outputs from five diverse LLMs to confirm that enumerated rubric paths cover all observed solution strategies.

• Base model(s). The paper evaluates both proprietary frontier models (GPT-5, Gemini-2.5-Pro, o3) and open-source models across scales and specializations (DeepSeek-V3.1, Qwen3-Coder, Qwen3-235B-A22B, Qwen3-8B, Kimi-K2, Doubao-Seed-1.6). The selection spans general-purpose and code-specialized architectures to enable analysis of how specialization interacts with DE vs. DA task demands. Notably, Qwen3-Coder is a code-optimized variant that serves as a probe for whether coding ability transfers to analytical reasoning. Model scale ranges from 8B parameters (Qwen3-8B) to proprietary frontier systems, providing a capability gradient for analyzing performance scaling.

• Metrics. The evaluation framework uses distinct metrics for deterministic and open-ended tasks:

  • DE-Impl and DE-Evol: Three execution-based metrics of increasing strictness (Section 2.2, Appendix A.1). Component Score (CS) evaluates each table in isolation using gold-standard upstream inputs, checking exact schema+data equivalence in DuckDB, with tolerances for floating-point rounding (two decimal places) and exclusion of time columns. Cascading Failure Score (CFS) evaluates nodes sequentially along the DAG with actual (predicted) upstream outputs; a node scores 1 only if it and all its ancestors are correct, otherwise 0. Success Rate (SR) requires every single component to match perfectly (or CFS ≥ 80 for DE-Evol tasks under a relaxed threshold). The paper also reports Max-CFS@8 and Max-CS@8, which are the maximum values across 8 independent agent runs.
  • DE-Arch: LLM-judge rubric evaluation across three dimensions (Business Alignment & Semantic Accuracy, Technical Feasibility & Structural Completeness, Design Quality), scored item-by-item with an evidence-first policy (Appendix A.2).
  • DA: A composite score combining the LLM-judge rubric score (weighted α = 0.6) and the Good-Same-Bad (GSB) score (weighted 1−α = 0.4). The rubric decomposes each task into requirements, standards, paths, and items across six dimensions: Completeness, Accuracy, Insightfulness (rubric), Readability, Analytical Depth, and Visualization (GSB). The paper also reports per-dimension breakdowns.
  • Aggregate scores: DE Score is the mean across all DE tasks (using CFS for Implementation/Evolution and the rubric score for Architecture). DA Score is the composite rubric+GSB score. The paper does not define a single "Overall Score" beyond these per-domain aggregates, though Tab. 3 and Tab. 5 report these values separately.
  • Max-*@8 notation: For Implementation tasks, the paper reports Max-CFS@8 and Max-CS@8, representing the best score achieved across 8 independent agent runs per task, providing an upper bound on what the agent can achieve given multiple attempts.

• Baselines. The paper uses two agent frameworks as baselines:

  • OpenHands (CodeAct Agent) (Wang et al., 2024): The primary harness for both DE and DA tasks. It provides a sandboxed environment with up to 200 interaction turns, an action space of BASH, IPYTHON, and TERMINATE, and automatic termination on repeated actions or 120-second timeouts (Appendix B.2). For DE-Impl tasks, OpenHands is extended to a multi-agent configuration where individual agents handle specific SQL files guided by YAML specifications, operating in dependency order with a validation agent ensuring pipeline correctness.
  • DA-Agent: A custom baseline for DA tasks inspired by ReAct (Yao et al., 2022), with a four-action space of BASH, CREATE_FILE, EDIT_FILE, and TERMINATE (Appendix B.1). This simpler agent serves as an alternative implementation to test whether framework complexity affects DA performance. The paper reports both OpenHands and DA-Agent results for DA tasks (Table 5), enabling comparison of agent architecture effects. For DE tasks, only OpenHands results are reported (Table 3, Table 4 for Chinese), with the multi-agent extension for Implementation.

• Generation budget / compute accounting. All agents operate within sandboxed environments with interaction budgets rather than token-based limits. Each task allows up to 200 rounds of tool interactions (OpenHands) or terminates on three consecutive repeated actions or a single action exceeding 120 seconds (both frameworks). For DE-Impl multi-agent tasks, individual SQL-writing agents are constrained to 50 steps, and the validation agent to 100 steps. The paper does not report token counts or FLOPs for agent runs—the primary resource constraint is the number of interaction turns, visualized in Figure 6, which shows turn-count distributions across models and task types. This design choice means that comparisons are made at the task level rather than at equalized computational budgets, which differs from the compute-matching paradigm used in some agent benchmarks.

• Cross-validation / statistical protocol. For evaluation stability, the paper implements several statistical measures:

  • Max@8 reporting: For DE tasks, agents are run up to 8 times per task, and the maximum CS and CFS values are reported (Max-CS@8, Max-CFS@8) to account for stochastic variation in agent performance. The standard CS and CFS values represent single-run results.
  • LLM-judge variability: To assess scoring reproducibility, the paper runs 8 independent grading passes on fixed agent outputs using Gemini-2.5-Flash (Table 10), reporting means and standard deviations. Standard deviations are consistently below 0.35 across all evaluated models and task types.
  • Human-model agreement: The validation study (Section 3.6) uses 300 model responses scored by human experts against over 7,000 rubric items, with inter-rater agreement metrics (κw, ICC, τb) computed at item, case, and model levels (Appendix B.4).
  • Cross-judge consistency: Rankings are validated across 6 alternative LLM judges (GPT-4.1, Gemini-2.5-Pro, Qwen3-235B, Qwen3-30B, Kimi-K2-Thinking, DeepSeek-V3.1) using Kendall's τb (Table 8).
  • Hyperparameter sensitivity: The α weighting parameter is varied across {0.5, 0.8, 0.9} to test ranking stability (Table 9, τb = 1.00 across all settings).
  • Confidence intervals: The paper reports mean scores with standard deviations for DA-Agent baselines (± values in Table 5) but does not appear to compute statistical significance tests comparing models directly. Standard deviations for OpenHands baselines are not reported in Table 5.

---

Main Quantitative Results

DE-Architecture Results: Specification Quality

Headline result: GPT-5 achieves the highest Architecture score of 63.93 (±2.33) (Table 3), with a substantial gap over all other models. Gemini-2.5-Pro follows at 51.96 (±1.78), and open-source models cluster in the 48–53 range (Qwen3-Coder: 51.43, DeepSeek-V3.1: 52.66, o3: 48.32), while smaller models drop to 45.12 (Qwen3-8B). The ~12-point gap between GPT-5 and the next tier suggests that architectural planning—translating high-level business requirements into structured design documents—is a capability that scales non-linearly with model capability, unlike Implementation where the gaps are narrower.

The error analysis in Table 15 reveals the dominant failure modes: Function Point Omission (ranging from 26.51% for GPT-5 to 44.00% for Qwen3-8B) and Dependency Errors (17.14% to 36.59%) are the most prevalent, while Naming Inconsistencies (6.41–14.35%) and Improper Model Layering (7.21–14.35%) are secondary. This indicates that the primary challenge in Architecture tasks is completeness of specification—capturing all required business logic—rather than structural quality once the logic is identified.

The Chinese adaptation (DAComp-zh, Table 4) shows nearly identical patterns: GPT-5 achieves 63.60 (±2.14), with other models within 1–2 points of their English scores, suggesting that the Architecture task's difficulty is invariant to language for these models.

DE-Implementation Results: Pipeline Construction from Scratch

Headline result: Even the best model achieves a Success Rate of only 20.00% (GPT-5, Table 3). The three-tier metric ladder reveals the specific nature of the difficulty:

• Component Score (CS): GPT-5 achieves 61.98% (Max-CS@8: 68.77%), meaning that when given perfect upstream inputs, it writes correct SQL for roughly two-thirds of individual pipeline nodes. Qwen3-Coder and Gemini-2.5-Pro follow at 54.21% and 55.32% respectively, while Qwen3-8B collapses to 15.33%. The gap between CS and Max-CS@8 is modest (~7 percentage points for GPT-5), indicating that repeated attempts yield only incremental improvements in per-component correctness.

• Cascading Failure Score (CFS): GPT-5 drops dramatically to 30.79% (Max-CFS@8: 39.87%). The ~31 percentage-point gap between CS and CFS (61.98 → 30.79) is the paper's central diagnostic: even when individual components are mostly correct, dependency failures cascade through the pipeline, rendering correct downstream code useless because it operates on corrupted inputs. For weaker models, this gap is even more catastrophic: Qwen3-235B-A22B falls from CS 20.15% to CFS 2.43%, and Qwen3-8B from 15.33% to 1.31%.

• Success Rate (SR): GPT-5 achieves 20.00% — one in five Implementation tasks is perfectly correct end-to-end. All other models are in single digits (Gemini-2.5-Pro: 8.00%, o3: 6.00%, Qwen3-8B: 2.00%).

Comparison across models: The ranking at the CFS level is GPT-5 (30.79%) > Gemini-2.5-Pro (27.66%) > Qwen3-Coder (23.64%) > DeepSeek-V3.1 (22.33%) > o3 (15.07%), with a sharp drop to open-source general models. Notably, o3—a reasoning-specialized model—underperforms the code-specialized Qwen3-Coder (15.07% vs. 23.64% CFS), suggesting that for systematic code generation across large pipelines, code-specific optimization matters more than general reasoning capability.

Chinese adaptation (Table 4): Results are nearly identical to English across all models (GPT-5 CFS: 30.49% vs. 30.49%; Qwen3-Coder: 23.23% vs. 23.64%), confirming that the Implementation task's difficulty is language-invariant when the technical specification is provided.

Per-component difficulty analysis (Figure 4): Breaking down individual SQL file modifications by type (create vs. edit) and code scale reveals a non-monotonic difficulty curve for edit tasks. For creation tasks, there is a "sweet spot" at medium scale (20–150 lines) where models perform best, with performance declining for very large files (>150 lines). For edit tasks, medium-scale edits (5–10 lines) are paradoxically the hardest—contrary to the intuition that larger changes are more difficult. The paper attributes this to medium edits containing "the most complex and nuanced changes to business logic, aggregations, and calculations," whereas minor edits are trivial and large edits often involve repetitive, boilerplate transformations.

Pipeline layer analysis (Figure 5): The error distribution across layers reveals escalating difficulty. The staging layer has the fewest local errors (7.7% of all errors across the pipeline) and the highest task survival rate. The core (intermediate) layer accounts for the largest share of local errors (Figure 5a) and shows the sharpest drop in pipeline survival rate (Figure 5b)—this is where complex business logic and entity integration occur. The marts layer remains challenging, but its failures are increasingly driven by upstream propagation rather than local logic defects (Figure 5b shows fewer than 20% of initial tasks survive through all three layers).

Interaction patterns (Figure 6): High-performing models (GPT-5, Gemini-2.5-Pro) exhibit moderate turn counts with compact variance across both Implementation (centered around 50–100 turns) and Evolution (25–75 turns). Weaker models (Qwen3-Coder) display either excessively long and volatile traces in Implementation (extending to ~250 turns for some tasks) or unusually short traces in Evolution, where premature termination corresponds to incorrect outputs.

DE Score aggregation (Table 3, final column): When aggregated across Architecture, Implementation, and Evolution (using CFS for Implementation/Evolution), the overall DE Score ranking is: GPT-5 (43.45%) > Gemini-2.5-Pro (32.88%) ≈ Qwen3-Coder (32.80%) > DeepSeek-V3.1 (31.41%) > o3 (28.39%). The code-specialized Qwen3-Coder essentially matches Gemini-2.5-Pro despite being open-source, while o3's reasoning specialization provides no advantage for systematic code generation.

DE-Evolution Results: Modifying Existing Pipelines

Headline result: Success Rates on Evolution tasks are universally low (Table 3, SR@8 column): GPT-5 achieves 20.00%, Qwen3-Coder 12.00%, and most other models fall to 2.00–8.00%. Even with 8 attempts per task, the best model succeeds on only one-fifth of Evolution tasks.

CFS comparison: GPT-5 achieves CFS 38.75% (Max-CFS@8: 47.23%), substantially higher than Implementation CFS (30.79%), which might suggest Evolution is easier in absolute terms. However, Evolution tasks are smaller in scope (~950 LOC vs. ~2,300 LOC on average, Table 2), so the higher CFS reflects reduced scale rather than reduced difficulty per line of code.

The discriminative challenge: A critical finding in Table 17 is that for top models, the SQL Omission rate is higher in Evolution than Implementation (GPT-5: 11.69% vs. 5.18%). This reveals that identifying which files to modify is harder than building from scratch—in Implementation, the agent knows it must create every file; in Evolution, it must discriminatively determine the scope of changes, and even GPT-5 misses required modifications at a higher rate than it omits files during construction.

Dependency error patterns (Table 19): The qualitative nature of dependency errors shifts between Implementation and Evolution. In Implementation, missing and extra dependencies are roughly balanced across models. In Evolution, weaker models show a pronounced bias toward missing dependencies—for example, Qwen3-8B has 46.92% missing vs. 20.88% extra. This suggests that Evolution places a premium on context retention and lineage tracking that limited-capacity models cannot maintain, causing them to lose track of downstream consequences when making schema modifications.

Architecture-level attrition (Table 17, DE-Evol section): The layer-wise SQL Omission rates confirm that the intermediate and marts layers are the primary challenge, with GPT-5 showing 8.99% omission at the intermediate layer and 15.34% at marts (compared to Implementation's 3.66% and 9.37%). The absence of staging-layer modifications in Evolution tasks (by design, since Evolution focuses on business logic) means all the difficulty is concentrated in the layers with the most complex transformations.

DA (Data Analysis) Results: Open-Ended Analytical Reasoning

Headline result: GPT-5 achieves a DA Score of 56.14% (DA-Agent baseline, inferred from Table 8 ranking table), with the second tier clustering in the 34–42% range and smaller models falling below 15%. The full dimensional breakdown (Table 5) reveals the specific nature of the capability gap.

GPT-5 dominance across all dimensions: Under the DA-Agent baseline, GPT-5 achieves: Completeness 64.23 (±2.37), Accuracy 43.81 (±3.43), Insightfulness 56.89 (±6.48), Readability 43.59 (±6.08), Analytical Depth 76.80 (±4.91), Visualization 27.44 (±4.44), for an aggregate DA Score of 50.84 (±3.12) at α = 0.6. The OpenHands baseline (Table 5, top section) produces slightly lower scores (GPT-5: 46.99), with the largest gap in Readability (35.51 vs. 43.59) and Analytical Depth (69.8 vs. 76.80), suggesting that the simpler DA-Agent framework enables better output formatting and explanation.

The "calculator behavior" of reasoning-focused models: o3 achieves competitive Accuracy (29.54 under DA-Agent) and Completeness (40.73), but collapses on Readability (25.24), Analytical Depth (23.81), and Visualization (7.32). Similarly, Gemini-2.5-Pro achieves moderate Accuracy (30.30) but weak Insightfulness (41.45) and Depth (35.75). This confirms a pattern the paper emphasizes: computing correct numbers is necessary but insufficient for data analysis. The ability to synthesize those numbers into human-readable insights, craft decision-oriented recommendations, and generate appropriate visualizations is a distinct capability that reasoning- or code-specialized models do not automatically possess.

The code-specialization trap: Qwen3-Coder, which performs competitively on DE tasks (DE Score 32.80%, nearly matching Gemini-2.5-Pro), almost completely collapses on qualitative DA dimensions: Readability 19.37, Visualization 5.15, Analytical Depth 13.42 (DA-Agent baseline). Its DA Score of 25.13% is only marginally better than the 8B-parameter Qwen3-8B on some dimensions and far below the general-purpose DeepSeek-V3.1 (34.33%). This is the clearest evidence in the paper that engineering and analysis are distinct, non-transferable capabilities.

Small model collapse: Qwen3-8B achieves a DA Score of 4.47% under DA-Agent, with scores near zero on Readability (0.13), Analytical Depth (0.00), and Visualization (0.15). The model essentially fails to produce coherent analytical artifacts, establishing a "capability threshold" below which agents cannot engage in meaningful open-ended analysis regardless of framework quality.

Analytical objective hierarchy (Figure 8): Performance varies systematically with the primary analytical objective of the task. All models perform best on Descriptive tasks (GPT-5: ~60% DA Score, Qwen3-Coder: ~35%) and worst on Strategic tasks (GPT-5: ~52%, Qwen3-Coder: ~15%). The hierarchy is consistent: Descriptive > Diagnostic > Strategic ≈ Pattern Recognition > Profiling, with a roughly 15–25 percentage point gap between Descriptive and Strategic for each model. This confirms that tasks requiring prescriptive reasoning ("what should we do?") are fundamentally harder than tasks requiring factual reporting ("what happened?"), and that this difficulty gradient is robust across model scales.

Chinese adaptation (Table 6): The DA-zh results show similar rankings but slightly compressed scores. GPT-5 achieves 49.49 (±1.04) under DA-Agent, with other models within 2–5 points of their English scores. The dimensional patterns are preserved: code-specialized models collapse on qualitative dimensions (Qwen3-Coder: Readability 4.07, Visualization 1.75), and the small Qwen3-8B fails to produce coherent outputs (DA Score 6.33).

Error distribution in DA tasks (Figure 9): The decomposition of failures into three stages—Planning & Preparation, Execution & Calculation, and Interpretation & Synthesis—shows that Execution & Calculation Failures dominate, averaging 59.05% of all failures across models. GPT-5's error distribution is 19.9% Planning, 63.7% Execution, and 16.4% Interpretation; Gemini-2.5-Pro shows 19.3%, 59.3%, 21.4% respectively. The remaining ~40% split roughly evenly between planning failures and interpretation failures. This suggests that while execution robustness is the most pressing bottleneck (accounting for the majority of score loss), cognitive stages (planning strategy, synthesizing insights) collectively account for two-fifths of performance gaps and cannot be ignored.

LLM-Judge Validation Results

Headline result: The LLM judge (Gemini-2.5-Flash) achieves near-human agreement on rubric-based scoring (Table 7). Human inter-rater agreement establishes the ceiling: Rubric Item κw = 0.906, Rubric Case ICC = 0.925, GSB Readability κw = 0.601, GSB Professional κw = 0.751, GSB Visualization κw = 0.753. The LLM judge achieves: Rubric Item κw = 0.834, Rubric Case ICC = 0.890, GSB Readability κw = 0.604, GSB Professional κw = 0.759, GSB Visualization κw = 0.735. The judge matches or slightly exceeds human agreement on two of three GSB dimensions, with Readability showing the lowest agreement (κw ≈ 0.53–0.61) even among humans due to inherent subjectivity.

Cross-judge consistency (Table 8): Ranking correlations across six alternative judges (GPT-4.1, Gemini-2.5-Pro, Qwen3-235B, Qwen3-30B, Kimi-K2-Thinking, DeepSeek-V3.1) achieve τb = 1.00 for all except Qwen3-235B (τb = 0.90). Absolute scores vary (e.g., GPT-5 ranges from 53.72 under Qwen3-30B to 71.57 under GPT-4.1), but relative rankings are stable, confirming that judge choice does not bias the leaderboard.

Hyperparameter robustness (Table 9): Varying α across {0.5, 0.8, 0.9} preserves rankings perfectly (τb = 1.00), though absolute scores shift as expected (e.g., GPT-5 from 52.14 at α=0.5 to 60.49 at α=0.9). The α = 0.6 standard provides a balance favoring technical correctness while still rewarding presentation quality.

Stochastic stability (Table 10): Standard deviations across 8 independent grading runs on fixed outputs are consistently below 0.35 for both DE-Arch and DA scores across all five tested models. GPT-5 shows σ = 0.18 (DE-Arch) and σ = 0.16 (DA), confirming reproducible scoring.

---

Ablation Studies and Robustness Checks

Agent framework comparison (OpenHands vs. DA-Agent for DA tasks): Table 5 reports both OpenHands and DA-Agent baselines for all models on DA tasks. The DA-Agent consistently yields higher scores (GPT-5: 50.84 vs. 46.99; Gemini-2.5-Pro: 34.70 vs. 33.38; Qwen3-Coder: 25.13 vs. 24.28), with the largest gaps on qualitative dimensions (Readability, Analytical Depth, Visualization). This demonstrates that agent framework design matters for analytical tasks—the simpler ReAct-style DA-Agent may provide better scaffolding for iterative code execution and report generation than the more general OpenHands harness.

Language invariance (Chinese vs. English): Tables 3 vs. 4 (DE) and Tables 5 vs. 6 (DA) provide comprehensive cross-language results. For DE tasks, the Chinese adaptation produces scores within 1–2 percentage points of English across all models and metrics (e.g., GPT-5 DE Score: 43.45 English vs. 42.88 Chinese; Qwen3-Coder: 32.80 vs. 32.36). For DA tasks, scores show marginally larger variance but identical rankings (GPT-5 DA Score: 50.84 English vs. 49.49 Chinese). The language invariance of the benchmark's difficulty suggests that the tasks measure fundamental reasoning capabilities independent of surface-form language, though Chinese-optimized models (Doubao-Seed-1.6) do not show a comparative advantage on the Chinese version.

Model scale ablation (within Qwen3 family): The Qwen3 family provides a natural scale comparison: 8B (Qwen3-8B) vs. 235B-A22B (Qwen3-235B-A22B). On DE tasks, the 235B model achieves DE Score 20.15 vs. 19.89 for the 8B model—a marginal improvement. On DA tasks, the gap is larger but still modest: DA Score 13.25 vs. 4.47 (DA-Agent). The small improvement from ~30× parameter scaling suggests that for these complex, multi-step tasks, raw model scale is not the dominant factor—specialization (code vs. general) and architecture quality appear more important, as evidenced by Qwen3-Coder (a specialized variant) substantially outperforming the larger Qwen3-235B-A22B on both DE (32.80 vs. 20.15) and DA (25.13 vs. 13.25).

Max@8 multi-run ablation: The comparison between single-run scores (CS, CFS) and Max@8 scores (Max-CS@8, Max-CFS@8) in Tables 3 and 4 reveals the benefit of repeated attempts. For GPT-5 on Implementation, CS improves from 61.98 to 68.77 (+6.79 points), and CFS from 30.79 to 39.87 (+9.08 points). The larger gain in CFS than CS suggests that dependency management benefits more from repeated attempts than per-component correctness—likely because stochastic variation occasionally produces correct dependency graphs even when the model's modal behavior is incorrect. The SR@8 column shows that even with 8 attempts, most models still achieve very low strict success rates (GPT-5: 20.00%), confirming that consistent end-to-end correctness remains out of reach.

Difficulty analysis by task complexity (Figure 11): The paper analyzes how agent performance varies with task structural complexity. For Implementation tasks, the CS generally declines as the number of DAG nodes increases, with GPT-5 showing a significant drop on tasks with more than 50 nodes. For Evolution tasks, agents are more sensitive to total lines changed, with vulnerability concentrated in the 800–1200 line range—the "mid-to-high complexity" regime. This non-linear sensitivity suggests that there are complexity thresholds beyond which agent robustness degrades rapidly, and these thresholds differ between construction (node count) and modification (line count) tasks.

Turn count stability as a quality signal (Figure 6): The analysis of interaction turn distributions reveals that effective agents exhibit moderate, centered turn counts with compact variance. GPT-5's Implementation turn counts center around 50–100 with low variance; Qwen3-Coder shows a long tail extending beyond 200 turns, corresponding to unproductive iteration. For Evolution, weak models display unusually short traces (premature termination producing incorrect outputs). This is not an ablation in the traditional sense but serves as a robustness check on the agent interaction protocol: it confirms that the 200-turn budget and timeout/repetition termination conditions are not artificially constraining capable models while still bounding unproductive loops.

Error taxonomy validation (Figure 7, Tables 15–19): The comprehensive error classification across DE tasks provides multiple validity checks on the diagnostic framework. The consistency of error hierarchies across models (Dependency Errors always dominant, Data Type Errors always marginal) supports the claim that these are task-inherent challenges rather than model-specific weaknesses. The layer-wise analysis (Table 17) showing monotonically increasing SQL Omission from staging to marts layers validates that the DAG structure genuinely escalates difficulty. The upstream-vs-intrinsic error decomposition (Table 18) confirming that upstream propagation dominates for stronger models validates the cascading failure conceptual model underlying the CFS metric.

---

Critical Assessment

The experiments presented in Section 3 and the accompanying tables substantiate several of the paper's central claims but leave important questions unresolved. I evaluate each major claim against the evidence.

Claim: "Holistic pipeline orchestration—not merely code generation—is the core bottleneck in data engineering." This claim is strongly supported by the quantitative evidence. The gap between CS and CFS for GPT-5 on Implementation (61.85% → 30.79%, a 31-percentage-point drop) directly operationalizes "orchestration penalty" as a measurable quantity (Table 3). The dependency error analysis (Table 16) showing rates exceeding 65% for all models, combined with the decomposition of calculation errors into upstream vs. intrinsic (Table 18) showing ~3× more upstream errors for top models, provides converging evidence that the pipeline coordination problem dominates individual query accuracy. The layer-wise survival analysis (Figure 5b) showing the sharpest drop in pipeline integrity at the intermediate layer—where business logic integration occurs—further localizes the bottleneck to dependency management rather than basic SQL syntax.

However, the evidence is limited by the single benchmark and single model family. We cannot conclude that orchestration is universally the bottleneck for all LLMs on all data engineering tasks—only that it is the dominant bottleneck for the tested models on DAComp-DE. A model with near-perfect component-level accuracy but poor orchestration would confirm the claim; conversely, a model with weak component-level accuracy might still be bottlenecked by code generation, as the paper acknowledges for Qwen3-8B (Table 17, CS = 15.33%). The claim holds most strongly for capable models (GPT-5, Gemini-2.5-Pro, Qwen3-Coder) where CS is moderate but CFS collapses; for weak models, both CS and CFS are low, and the bottleneck is fundamental code generation.

Claim: "Engineering and analysis are distinct, independently challenging capabilities." The differential performance patterns provide compelling evidence. Qwen3-Coder achieves competitive DE performance (DE Score 32.80%, Table 3) but collapses on DA qualitative dimensions (Readability 3.15%, Visualization 1.93%, Table 5). Conversely, GPT-5 dominates both but with substantially different margins: its DE Score of 43.45% is only ~10 points above the second tier, while its DA Score advantage is larger. The analytical objective hierarchy (Figure 8) showing systematic performance differences across task types provides an additional axis of differentiation. These patterns are consistent across the DA-Agent and OpenHands baselines and across English and Chinese versions, strengthening the claim.

A limitation is that the paper does not present a formal statistical test for whether DE and DA performance are uncorrelated or whether the correlation is weaker than within-domain correlations. The evidence is based on qualitative pattern observation. Additionally, the claim that these are "distinct capabilities" would be strengthened by showing that improving one (e.g., through fine-tuning on DE tasks) does not improve the other (DA performance), but no such causal manipulation experiment is conducted.

Claim: "The compute-optimal test-time scaling strategy improves efficiency by more than 4×." This claim does not apply to this paper—DAComp is a benchmark, not a method paper proposing an optimization strategy. There is no test-time compute scaling experiment in DAComp. The paper's contribution is diagnostic: it measures where agents fail, not a method to improve efficiency. This is a correct scope for a benchmark paper and should not be criticized as a missing claim.

Claim: "Even state-of-the-art agents falter on DAComp, with success rates under 20% on engineering tasks and DA scores averaging below 40%." This claim is supported with nuance. GPT-5 achieves SR = 20.00% on Implementation (Table 3), meeting the "under 20%" threshold exactly for the best model. Most other models achieve single-digit SR (Gemini-2.5-Pro: 8.00%, o3: 6.00%). For DA, GPT-5 achieves a DA Score of 50.84% (Table 5, DA-Agent), which is above the "averaging below 40%" figure stated in the abstract—the average across all models is indeed below 40%, but the best model exceeds it substantially. The abstract's phrasing "scores on DA tasks also average below 40%" is technically correct as an average across all evaluated models, but it obscures the substantial variation between GPT-5 (~50%) and weaker models (<15%). This is a rhetorical imprecision rather than a factual error, but readers should note that the "state-of-the-art agents falter" framing applies more strongly to DE tasks (20% SR) than to DA tasks (50% DA Score for the best model).

Genuine weaknesses in the experimental design:

• Single benchmark, single evaluation paradigm. All claims about agent capabilities are conditioned on DAComp's specific task distribution and evaluation methodology. The paper cannot establish whether performance on DAComp-DE correlates with performance on other repository-level engineering tasks (e.g., SWE-Bench) or whether the DA rubric captures analytical quality dimensions that generalize beyond this benchmark's task types. Cross-benchmark correlation studies would strengthen the claim that DAComp measures generalizable capabilities.

• Single run vs. multiple runs inconsistency. For DE tasks, the paper reports single-run CS and CFS alongside Max@8 values, enabling assessment of stochastic stability. For DE-Arch and many DA baselines, only single runs appear to be reported (the standard deviations in Tables 3 and 4 are specified for Architecture but not for Implementation/Evolution CFS). The DA-Agent baseline includes standard deviations across runs (Table 5, ± values), but it is unclear whether these represent multiple agent runs or multiple judge runs. The difference matters: multiple agent runs capture generation stochasticity, while multiple judge runs (Table 10) capture evaluation stochasticity. Clarifying this would help readers assess the reliability of the reported scores.

• The baseline reports for GSB are LLM-generated. The GSB protocol compares agent outputs against five baseline reports "created by combining outputs from multiple LLMs" (Section 2.3, Stage 3). If these baselines are weak, the GSB score could inflate by making mediocre agent outputs appear "better than baselines." Conversely, if the baselines are strong, the GSB score could be excessively harsh. The paper does not characterize the quality of these baselines or report GSB scores for the baselines themselves, making it difficult to calibrate what a given GSB score means in absolute terms.

• Missing ablation on rubric item weighting. The 66% weight on Accuracy items (Table 2) is a design choice that heavily influences the DA Score. The paper validates that α (rubric vs. GSB weight) does not affect rankings (Table 9), but does not validate whether the internal weight distribution (Completeness 14%, Accuracy 66%, Insightfulness 20%) is robust. A sensitivity analysis varying these weights would strengthen confidence that the DA Score does not depend on arbitrary item-weighting choices.

• No test-retest reliability for agent runs. Table 10 reports scoring stability (same outputs, multiple judge runs), but the paper does not report the stability of agent outputs across multiple runs (same task, multiple agent runs, same judge). The Max@8 reporting partially addresses this by showing the best-case performance, but a full test-retest analysis (e.g., correlation between run 1 and run 2 scores for the same model on the same task) would characterize the inherent stochasticity of agent performance, which is essential for interpreting score differences between models.

Missing experiments that would strengthen the paper:

• Fine-tuning experiments. The paper identifies specific failure modes (dependency management gaps, qualitative reasoning deficiencies) but does not test whether targeted fine-tuning on these dimensions improves performance. For example, fine-tuning on dependency-tracking tasks or on analytical report generation would provide causal evidence that the identified bottlenecks are addressable and that DAComp scores respond to capability improvements.

• Human baseline for DE and DA tasks. The paper validates the LLM judge against human evaluators but does not report human performance on the tasks themselves. What DA Score would an expert human data analyst achieve? What DE Success Rate would a human data engineer achieve? Without a human ceiling, the absolute difficulty of DAComp is hard to calibrate—a 50% DA Score might represent near-human performance or far below it.

• Scaling analysis across model sizes. The Qwen3 family provides a two-point scale comparison (8B vs. 235B-A22B), but a more systematic scaling analysis (e.g., testing 1B, 8B, 70B, 235B variants) would reveal whether DAComp performance follows predictable scaling laws or exhibits emergent capability thresholds. The near-identical performance of Qwen3-8B and Qwen3-235B-A22B on DE tasks (DE Score 19.89 vs. 20.15) is surprising and warrants more investigation—it suggests that for repository-level engineering, factors other than parameter count dominate.

• Time-series analysis of agent performance. The paper does not analyze whether agent performance improves, degrades, or plateaus over the course of interaction turns. Do agents that fail on DE-Impl fail early (at the staging layer) or late (at the marts layer)? Does performance on DA tasks improve as models iterate on their analysis? Turn-count distributions (Figure 6) provide aggregate statistics but not within-task dynamics.

Conditions under which claims hold:

• The claim that orchestration is the bottleneck holds for capable models (GPT-5, Qwen3-Coder) on Implementation tasks, where CS substantially exceeds CFS. It does not hold for weak models (Qwen3-8B), where CS is already near zero and the orchestration penalty is moot.

• The claim that engineering and analysis are distinct holds across all model scales tested, but the degree of differentiation varies: for top models, the gap is large (DE Score 43.45% vs. DA Score 56.14% for GPT-5); for weak models, both scores collapse and the distinction is less informative.

• The absolute performance thresholds ("under 20% SR," "below 40% DA Score") are specific to the current generation of models evaluated. They serve as baselines for future work, not as permanent bounds on achievable performance. The paper cannot and does not claim that these thresholds represent fundamental limits—only that they represent the current state of the art.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted For in the DA Analysis

The assumption or constraint. The DAComp-DA evaluation framework requires human experts to manually construct hierarchical rubrics for each of the 100 DA tasks, enumerating valid solution paths, computing anchor values, and defining scoring criteria. This is an offline, one-time cost for benchmark construction, but it represents a fundamental constraint on the approach's scalability. Every new analytical task requires 3+ expert annotators, alignment discussions, computational verification of anchors, and validation against outputs from five diverse LLMs to ensure rubric comprehensiveness (Section 2.3, Stage 3). The paper acknowledges this explicitly only in passing—it describes the annotation pipeline as rigorous but does not quantify the person-hours required or discuss whether the methodology can scale to substantially larger task sets.

The consequence. The practical implication is that DAComp's evaluation methodology—which the paper justifiably presents as a key contribution—is not directly adoptable by practitioners who want to evaluate their own agents on custom analytical tasks. If a company wants to test whether its internal data agent can answer business questions about its proprietary data, it would need to replicate the entire rubric construction pipeline: hire domain experts, enumerate solution paths, compute anchors, run validation studies. The paper provides a template but no automation or tooling to reduce this cost. This limits DAComp's impact as an evaluation framework that others can extend, as opposed to a fixed benchmark that others can use as-is.

What evidence exists in the paper. The paper reports that DAComp was constructed by 8 experts, that at least 3 annotators worked on each DA question, and that annotators were required to "personally run the complete analysis with code to calculate all Anchor Values required for the Accuracy assessment" (Appendix E.3.3). These procedural details indirectly convey the cost but do not quantify it. Table 2 reports that DA tasks average 3.1 Requirements, 5.7 Sub-requirements, 12.7 Paths, and 22.4 Rubric Items per task—suggesting substantial annotation effort per question. The validation study (Section 3.6) confirms that the rubrics are high-quality, but does not estimate how many expert-hours were required to achieve that quality.

Mitigation status. The paper does not address this limitation or propose automation strategies for rubric construction. The three-tier accuracy strategy (Appendix F.1) suggests that for many items, numeric anchors or pseudo-code specifications can be defined deterministically, which could in principle be partially automated (e.g., by executing reference solutions and extracting computed values as anchors). However, no such automation is demonstrated, and the path enumeration step—identifying methodologically distinct solution strategies—appears to require human judgment that would be difficult to automate reliably. Future work could explore LLM-assisted rubric generation with human validation, reducing but not eliminating the expert annotation burden.

---

Engineering and Analysis Are Tested on Independent Subsets—Not as an End-to-End Workflow

The assumption or constraint. DAComp evaluates DE tasks and DA tasks as separate, self-contained problems. DE-Impl agents receive a data_contract.yaml specification and build a pipeline; DE-Evol agents receive a change request and modify an existing pipeline; DA agents receive analysis-ready semantic-layer data and produce analytical reports. What DAComp does not test is the integrated workflow where a single agent carries a business requirement through architecture design, pipeline implementation, iterative evolution under changing requirements, and finally open-ended analysis on the resulting data. The paper's formal task definitions (Section 2.1) treat each task type independently, and the experiments evaluate agents on each type in isolation.

The paper acknowledges this gap explicitly in Appendix F.4:

"At present, we evaluate these stages modularly and in a decoupled fashion to enable controlled measurement at each step. Our next key objective is to integrate them into a single, end-to-end longitudinal evaluation."

The consequence. This decoupling means that DAComp overestimates real-world agent capability in an important sense. In an enterprise setting, errors compound across stages: a flawed architecture specification leads to an incorrectly implemented pipeline, which produces subtly wrong analytical data, which causes misleading insights and bad business decisions. The paper's error analysis already demonstrates within-stage cascading (the CS→CFS gap in DE-Impl, Table 3), but it does not measure cross-stage cascading. An agent that achieves 43.45% DE Score and 56.14% DA Score on independent tasks might achieve near-zero success on an end-to-end workflow where it must design, build, and analyze—because an error in the architecture stage invalidates everything downstream, and there is no human engineer in the loop to catch it.

The modular evaluation also prevents studying an important capability: self-debugging across stages. A capable agent might detect that its DA results are inconsistent with business expectations and trace the problem back to a DE-Impl error, then fix the pipeline and re-run the analysis. This closed-loop debugging is a hallmark of skilled human data professionals, but DAComp's task decomposition makes it impossible to test.

What evidence exists in the paper. The within-stage error propagation data provides indirect evidence of how severe cross-stage cascading would be. Table 18 shows that for GPT-5 on DE-Impl, upstream errors account for 30.41% of Marts-layer calculation errors—meaning that even within a single pipeline, one-third of final-layer errors originate from earlier stages. If architecture specification errors (Table 15: GPT-5 Function Point Omission = 26.51%, Dependency Errors = 17.14%) propagated through implementation and into analysis, the compound error rate could be very high. The paper does not measure any cross-stage propagation.

Mitigation status. The paper explicitly identifies end-to-end integration as the "next key objective" (Appendix F.4) and argues it "offers substantial scientific and practical value." This is an honest acknowledgment of a scope limitation rather than an oversight. The current modular design is a reasonable first step—it enables controlled diagnosis of where agents fail—but until end-to-end evaluation is implemented, DAComp's results should be interpreted as measuring component capabilities rather than workflow-level autonomy. A practitioner reading the paper should understand that an agent scoring 50% on DA tasks may score substantially lower if it must first build the pipeline that produces the analysis-ready data.

---

The LLM Judge's Readability Assessments Show Substantially Lower Agreement Than Other Dimensions

The assumption or constraint. The DA evaluation framework relies on the LLM judge (Gemini-2.5-Flash) to assess both objective dimensions (Accuracy, Completeness, Insightfulness via the rubric) and subjective dimensions (Readability, Analytical Depth, Visualization via the GSB protocol). The validation study (Table 7) reveals that while rubric-based scoring achieves near-human agreement (κw = 0.834, ICC = 0.890), GSB-based scoring on Readability achieves only κw ≈ 0.53–0.61 across all judges, including humans. Even human inter-rater agreement on Readability is only κw = 0.601—substantially below the agreement on Professional/Depth (κw = 0.751) and Visualization (κw = 0.753).

The consequence. The Readability dimension—which accounts for part of the 40% GSB weight in the final DA Score—is inherently noisy in ways that the rubric-based dimensions are not. Two equally qualified human evaluators will disagree on whether a report is "well-organized" or "easily understandable" about 40% more often than they disagree on whether a calculation is correct. The LLM judge mirrors this noise rather than amplifying it (its κw of 0.604 is essentially identical to the human κw of 0.601), which means the problem is with the construct, not the judge.

This has two practical implications. First, small differences in DA Scores between models may not be statistically meaningful if they are driven by Readability differences. The paper does not report confidence intervals that account for GSB scoring variance, and the stochastic stability test (Table 10, σ < 0.35) measures judge variance on identical outputs, not the inherent ambiguity of the Readability construct. Second, models that optimize for Readability (e.g., by using clear Markdown formatting, structured headings, concise language) may receive inflated or deflated scores depending on the specific baseline reports they are compared against, because the GSB protocol's relative comparisons amplify variance: small differences in presentation quality can flip Good/Same/Bad judgments.

What evidence exists in the paper. Table 7 reports the full set of agreement metrics. The Readability κw values are consistently the lowest across all judges: human inter-rater κw = 0.601, GPT-4.1 κw = 0.596, Qwen3-235B κw = 0.531. By contrast, Professional/Depth and Visualization achieve κw > 0.70 for most judges. The paper acknowledges this variance in Section 3.6: "GSB Readability scores show expected variance due to subjectivity (κw ≈0.53)." However, it does not discuss the implications for model ranking reliability or propose adjustments to the DA Score formula to downweight subjective dimensions.

Mitigation status. The paper partially mitigates this by weighting the rubric component at α = 0.6, meaning the more objective Accuracy, Completeness, and Insightfulness dimensions dominate the final score. However, the GSB component still carries 40% weight, and within that 40%, Readability receives up to 4 of the 10 possible points in the GSB scoring rubric (Appendix A.3.2: Readability sub-dimensions sum to a range of −10 to +10, which is the full GSB score range for that dimension). The sensitivity analysis (Table 9) shows that varying α from 0.5 to 0.9 does not change model rankings (τb = 1.00), which is reassuring—it suggests that Readability noise is not driving leaderboard positions. However, this ranking stability may not hold for models that are closely matched on rubric dimensions but differ primarily in presentation quality. The paper does not report per-dimension score variances that would enable readers to assess whether Readability differences between specific model pairs are reliable.

---

Benchmark Scope Is Limited to Structured SQL-Centric Workflows with Deterministic Correctness Criteria for Engineering

The assumption or constraint. DAComp-DE tasks are exclusively SQL-based data transformation pipelines operating on relational data with well-defined schemas. The correctness criteria for DE-Impl and DE-Evol tasks are deterministic: exact schema+data equivalence between agent-generated and gold-standard outputs, checked in DuckDB (Appendix A.1). The DE-Arch tasks, while open-ended in specification, still assume a structured SQL pipeline will be the output. DA tasks operate over structured analytical tables and require SQL/Python for computation.

This means DAComp excludes entire categories of data intelligence work that are central to modern enterprise practice: unstructured data processing (text, images, logs), streaming data pipelines, machine learning model training and deployment, data governance tasks (access control, lineage documentation, compliance auditing), and infrastructure-as-code for data platforms. The paper positions DAComp as covering "the full data intelligence lifecycle" (title and abstract), but this claim should be understood as "the full lifecycle of structured, SQL-based data intelligence"—which is an important subset but not the whole.

The consequence. An agent that scores well on DAComp-DE has demonstrated the ability to orchestrate SQL transformation pipelines—a valuable but specific skill. It may fail on tasks that many data engineers perform regularly: parsing semi-structured JSON logs, writing streaming transformations with windowed aggregations, debugging a Spark job with data skew, or designing a feature engineering pipeline for a recommendation model. Similarly, an agent that scores well on DAComp-DA has demonstrated the ability to analyze structured business data—but not the ability to analyze customer support transcripts, product reviews, or sensor data, which require fundamentally different analytical approaches (NLP, time-series modeling, anomaly detection on unstructured signals).

The paper's framing could lead practitioners to overestimate the generality of "data agent" capabilities measured by DAComp. A hiring manager who sees that an agent achieves 50% DA Score might assume it can handle any data analysis task, when in fact it has only been tested on a specific distribution of structured business analytics problems.

What evidence exists in the paper. The task descriptions and statistics make the scope clear upon close reading. Table 2 shows DE schemas averaging 382 columns across ~32 tables—all relational, all with defined schemas. Figure 3 categorizes staging-layer cleaning operations into four types (validity, consistency, integrity, anomaly detection), all of which operate on typed columns with defined constraints. The DA task classification (Table 14) lists five analytical objectives—Descriptive, Diagnostic, Strategic, Pattern Recognition, Profiling—with example questions that all involve querying and aggregating structured business data (sales, customer segments, sales targets). Nowhere does the paper test an agent's ability to analyze free-text fields, process images, or handle schema-less data.

Mitigation status. The paper does not explicitly discuss this scope limitation or acknowledge that "full data intelligence lifecycle" is a narrower claim than it might appear. The benchmark's name ("DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle") and abstract framing ("Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights") implicitly define "data intelligence" as the structured data pipeline + analytics workflow. This is a legitimate scope choice—SQL-centric pipelines are a massive and important domain—but the paper would be stronger if it acknowledged the boundaries of this scope explicitly, rather than letting the "full lifecycle" language imply greater coverage than the benchmark delivers.

---

No Human Performance Baseline Exists to Calibrate Absolute Difficulty

The assumption or constraint. The paper evaluates 7–9 LLM-based agents on DAComp tasks and reports their scores relative to each other. It does not report the performance of expert human data engineers or data analysts on the same tasks. The LLM-judge validation study (Section 3.6) establishes that the judge agrees with humans on scoring outputs, but it does not establish what score a human producing outputs would achieve.

The consequence. Without a human ceiling, the absolute difficulty of DAComp is uncalibrated in a way that limits practical interpretation of the results. Consider three possible interpretations of GPT-5's DA Score of 56.14%:

1. Near-human performance: If expert human analysts achieve ~60%, then GPT-5 is approaching professional-level analytical capability on these tasks, and the remaining gap is modest.
2. Far below human: If expert humans achieve ~90%, then GPT-5 is far from professional competence, and the "state-of-the-art agents falter" framing is accurate but the absolute gap is much larger than the relative scores suggest.
3. Superhuman on some dimensions, subhuman on others: Humans might achieve 70% overall but with very different dimensional profiles (e.g., near-perfect Readability and Insightfulness, but lower computational Accuracy than an LLM that can flawlessly execute large-scale aggregations).

Each interpretation has different implications for how close we are to deployable autonomous data agents, and DAComp's results alone cannot distinguish between them. The paper's framing emphasizes that agents "falter" and "profound deficiencies" exist, but without a human reference point, "profound" is a judgment about the shape of the score distribution rather than about the absolute distance to acceptable performance.

The same calibration problem applies to the DE tasks. GPT-5's DE Score of 43.45% and SR of 20.00% sound low, but what SR would a human data engineer achieve on the same Implementation tasks? If humans also achieve only 40–50% SR on multi-thousand-line pipeline construction tasks (which are genuinely hard and error-prone even for experts), then the gap between LLMs and humans may be smaller than the absolute numbers suggest. If humans achieve 90%+ SR, the gap is enormous.

What evidence exists in the paper. The paper provides no human performance data on any DAComp task type. The annotation pipeline (Section 2.3, Appendix E) implies that expert humans can solve these tasks—they created the gold-standard solutions for DE tasks and computed the anchor values for DA rubrics—but they did so with unlimited time, collaboration, and iteration, which is not comparable to the single-agent, time-bounded evaluation protocol used for LLMs. The LLM-judge validation (Table 7) shows that humans agree with each other on scoring, but this is a different task from producing the outputs being scored.

Mitigation status. Not addressed. The paper does not mention the absence of a human baseline or suggest it as future work. Human baselines are standard in many agent benchmarks (e.g., SWE-Bench reports human success rates on issue resolution), and their absence here is a notable gap. A human study—even on a subset of 20–30 tasks with a few professional data engineers and analysts working under time constraints comparable to the agent evaluation—would substantially strengthen the paper's claims about the difficulty of DAComp and the capability of current agents relative to human expertise.

---

The Benchmark's Construction and Validation Depend on a Single LLM Judge That Is Still Subject to Undetectable Systematic Biases

The assumption or constraint. The DAComp-DA and DE-Arch evaluation frameworks rest on the assumption that an LLM judge (Gemini-2.5-Flash) guided by hierarchical rubrics can serve as a reliable proxy for human expert judgment. The paper validates this assumption extensively through inter-rater agreement studies (Section 3.6, Table 7), cross-judge consistency checks (Table 8), hyperparameter sensitivity analysis (Table 9), and stochastic stability tests (Table 10). These studies demonstrate that the judge is consistent with humans and consistent with other judges on the tasks and outputs tested.

The consequence. However, consistency does not guarantee accuracy in the presence of systematic biases that affect all judges similarly. The paper's validation methodology can detect random noise (through stability tests) and idiosyncratic judge biases (through cross-judge consistency), but it cannot detect shared blind spots that affect both human annotators and all evaluated LLM judges. For example:

• Verbosity bias: If longer, more elaborately formatted reports receive higher Readability scores from both humans and LLMs regardless of substantive quality, the agreement metrics will be high but the scores will systematically favor verbose outputs.
• Stylistic conformity: If both human annotators and LLM judges share expectations about what a "professional" analytical report looks like (executive summary first, methodology section, numbered recommendations), outputs that conform to this template may receive inflated Insightfulness scores even if the actual insights are shallow.
• Anchor-driven overscoring: If the rubric's anchor values make scoring too predictable—an agent that hits the anchor gets full Accuracy points regardless of whether it understood why that value is correct—the judge may award high rubric scores to agents that pattern-match rather than reason.

These biases are inherently difficult to validate because they would appear in both the human baseline and the LLM judge evaluations, producing high agreement while systematically mis-scoring certain types of outputs. The paper's agreement metrics would not detect them.

What evidence exists in the paper. The cross-judge consistency results (Table 8) provide partial reassurance: if different judge models (GPT-4.1, Gemini-2.5-Pro, Qwen variants) produce identical rankings (τb = 1.00 for five of six alternatives), it is less likely that the rankings are driven by model-specific biases. However, all of these judges are LLMs trained on similar data distributions and may share similar stylistic preferences. The GSB Readability agreement metrics (κw ≈ 0.53–0.61 across all judges and humans) suggest that even humans find Readability inherently ambiguous—which cuts both ways: it means LLM-judge noise on Readability is not worse than human noise, but it also means that Readability scores contain irreducible uncertainty that the agreement metrics cannot eliminate.

The paper does not conduct an experiment that would detect systematic biases directly. For example, intentionally degraded outputs (reports with excellent formatting but incorrect calculations, or correct calculations presented as unstructured text) could test whether the judge appropriately weights Accuracy over Readability. Outputs that use different valid analytical methods to arrive at the same conclusions could test whether the path-selection mechanism penalizes methodological diversity. Without such adversarial validation, the judge's susceptibility to systematic biases remains an open question.

Mitigation status. The paper addresses one specific form of bias—family-specific self-preference—by showing that evaluating Gemini agents with non-Gemini judges (GPT-4.1) produces identical rankings (Section 3.6). This rules out the most commonly cited LLM-judge failure mode. However, the more subtle shared biases described above are not addressed. The paper's rubric design—with explicit anchor values, path enumeration, and evidence-first scoring—is partially mitigating because it constrains the judge to make specific, verifiable decisions rather than holistic quality assessments. This reduces the surface area for bias but does not eliminate it: the judge must still make path-selection decisions and interpret whether an agent's output "matches" a path's description, which involves judgment calls that could be influenced by stylistic features. Future work could strengthen confidence through adversarial validation with deliberately constructed edge cases designed to probe specific potential biases.

7. Implications and Future Directions

How This Work Changes the Landscape

DAComp is not a method paper — it proposes no new model, algorithm, or training procedure. Its contribution is a diagnostic reorientation of the data agent field. The paper changes the research landscape by shifting the question from "Can LLMs do data tasks?" (a coarse yes/no) to "Exactly where in the data intelligence lifecycle do LLMs fail, and why?" — and by providing the instrumentation to answer that question with precision.

The magnitude of this shift is best characterized as reframing with operational teeth, not a paradigm shift. The paper does not overturn existing assumptions about LLM capabilities; it does not claim that agents are better or worse than previously thought. Rather, it reveals that the assumptions were too vague to be useful. Prior benchmarks told us that agents can write individual SQL queries with some accuracy (BIRD, Spider) and that agents can generate single data science scripts (DS-1000, DA-Code). What they did not tell us — and could not, given their design — is whether that capability composes into the systemic, multi-file, dependency-aware engineering that enterprise data work actually requires, or whether code-generation skill transfers to the open-ended reasoning, planning, and communication that data analysis demands. DAComp provides the first testbed where these questions can be asked and answered empirically.

The conceptual reframing has three components:

First, the decomposition of difficulty into separable failure modes. The paper's CS-CFS-SR ladder operationalizes the distinction between component quality and system integrity — a distinction invisible to single-output benchmarks. The finding that GPT-5 achieves 61.85% Component Score but only 30.49% Cascading Failure Score on DE-Implementation (Table 3) is not just a difficulty measurement; it is a causal diagnosis. The ~31-percentage-point gap is the cost of poor orchestration, quantified. This directly challenges the implicit assumption in prior work that improving code-generation accuracy (the dominant focus of data agent research) is the primary path to better data agents. The paper's error analysis (Table 16) confirms that dependency errors — not SQL syntax errors — dominate across all models, exceeding 65% even for GPT-5. The implication is that the field has been optimizing for the wrong bottleneck, or at least for only one of two independent bottlenecks. Research that improves single-query accuracy further (beyond GPT-5's already-high component-level performance) would yield diminishing returns on real pipelines unless it also addresses dependency management.

Second, the demonstration that engineering and analysis are distinct, non-transferable capabilities. The differential performance patterns — Qwen3-Coder achieving competitive DE scores (32.80%) while collapsing on DA qualitative dimensions (Readability 3.15%, Visualization 1.93%, Table 5), and o3 showing "calculator behavior" with competitive Accuracy but poor synthesis — are not merely interesting anomalies. They refute the implicit assumption that "good at data tasks" is a unitary capability that improves monotonically with model scale or code-optimization. This has direct consequences for research prioritization: developing a data agent requires explicit attention to both engineering-system and analytical-communication skills, and benchmarks that test only one axis (as all prior data agent benchmarks did) create a dangerous illusion of progress. An organization that selects a data agent based on BIRD or DS-1000 scores may be selecting for only half the required capability.

This finding also reconciles a latent tension in the literature. Prior work could be read as suggesting either that LLMs are promising data analysts (because they score well on text-to-SQL) or that they are insufficient (because they fail on multi-step reasoning). DAComp's dimensional analysis resolves this: both perspectives are partially correct, but they are measuring different things. SQL generation is necessary but not sufficient for analysis; analytical reasoning is necessary but not sufficient for engineering. The field's contradictory intuitions were a scope problem, not an evidence problem.

Third, the validation of a rubric-based evaluation methodology that achieves human-level reliability on open-ended tasks. This is a methodological contribution with implications beyond data agents. The hierarchical rubric design — with enumerated solution paths, pre-computed anchor values, and path-selection scoring — provides a template for evaluating any domain where tasks are open-ended but quality is multi-dimensionally assessable: scientific reasoning, policy analysis, legal argument evaluation, medical diagnosis justification, educational assessment. The paper's demonstration that an LLM judge using such rubrics achieves Rubric Item κw = 0.834 and Case ICC = 0.890 against human experts (Table 7) — and that rankings are stable across six alternative judges (Table 8, τb = 1.00) — establishes that structured evaluation protocols can overcome the reliability problems that have limited LLM-as-judge adoption. This reduces the barrier to creating open-ended benchmarks in other domains, because the paper provides both a design methodology and a validation framework that others can replicate.

What becomes more attractive, and less so. The paper's results make several research directions more attractive:

• Dependency-aware agent architectures. The finding that dependency errors dominate (Table 16) and that the CFS penalty is severe (Table 3) makes explicit dependency graph tracking, pipeline-level verification, and fault-containment mechanisms high-priority research targets — not optional enhancements.
• Multi-capability agent design. The engineering-analysis distinction implies that research on data agents should evaluate both capability axes and develop architectures that explicitly support both — for example, separating system-building and insight-communicating modules, or training on both engineering and analytical tasks.
• Rubric-based evaluation in other domains. The demonstrated reliability of hierarchical rubrics with path enumeration makes this evaluation paradigm newly credible for adoption in other open-ended task domains.
• Discriminative scope identification. The finding that top models find Evolution scope identification harder than Implementation construction (GPT-5 SQL Omission: 11.69% vs. 5.18%, Table 17) identifies a specific cognitive capability — understanding what to change in an existing system — as an under-studied bottleneck.

Conversely, several directions become less attractive as primary research investments:

• Incremental improvements in single-query SQL generation. With GPT-5 already achieving 61.85% CS on DAComp-DE, further improvements in per-component accuracy — absent improvements in dependency management — face diminishing returns on real pipeline tasks, where the CFS gap dominates.
• Generic "bigger model" scaling for data agent capability. The Qwen3 family comparison (8B vs. 235B-A22B) shows near-identical DE Scores (19.89 vs. 20.15) and only modest DA Score improvement (4.47 vs. 13.25). Raw scale is not the bottleneck; architecture, specialization, and training objective matter more for these complex, multi-step tasks.

Follow-Up Research This Work Enables

End-to-end longitudinal evaluation: a single agent through Architecture → Implementation → Evolution → Analysis. DAComp currently tests each stage independently, but the paper acknowledges (Appendix F.4) that integrated evaluation is the "next key objective." A concrete experiment would chain the four task types: give an agent a business requirement (DE-Arch), have it produce a specification, then implement that specification (DE-Impl), then modify the pipeline under a new requirement (DE-Evol), and finally analyze the resulting data to answer a business question (DA). The key measurement would be cross-stage error propagation — what fraction of Architecture errors cause Implementation failures? What fraction of Implementation errors produce incorrect DA insights? The paper's within-stage cascading data (Table 18: GPT-5's upstream Marts-layer errors are ~3× its intrinsic errors) suggests cross-stage cascading could be severe. A failure to achieve non-trivial end-to-end success would confirm that current agents are far from autonomous data professionals; moderate success would establish a new capability threshold. The experiment would require solving the "handoff" problem — how the agent's Architecture output becomes the Implementation input — ideally with the agent's own output feeding the next stage without human cleanup.

Targeted fine-tuning on dependency tracking and impact analysis. The paper identifies dependency errors as the dominant failure mode (Table 16, >65% across all models) and discriminative scope identification as a specific weakness in Evolution tasks (Table 17). A natural follow-up is to fine-tune a base model on a synthetic dataset of dependency-tracking tasks: given a partial DAG specification, predict which tables depend on which, or given a schema change, identify all downstream files that need modification. The training data could be generated from the 73 SaaS schemas in DAComp by programmatically extracting dependency graphs. The test would be whether fine-tuned models close the CS-CFS gap on DAComp-DE relative to untuned baselines. A positive result (CFS improvement without CS degradation) would provide causal evidence that dependency reasoning is trainable and that the CFS metric responds to targeted capability improvement. A negative result (no improvement despite training on dependency tasks) would suggest that the bottleneck is architectural (e.g., context window limitations for large dependency graphs) rather than skill-based, redirecting research toward external memory or structured state-tracking mechanisms.

Adversarial validation of the LLM judge to detect shared human-model biases. The paper's validation (Section 3.6) demonstrates consistency between the LLM judge and human experts and across different judge models, but it cannot detect biases shared by both humans and LLMs — for example, verbosity bias (longer reports scored higher regardless of substance) or template-conformity bias (reports following an expected structure receiving inflated Insightfulness scores). A diagnostic experiment would construct adversarial output pairs: (a) a report with flawless calculations but unstructured, hard-to-read presentation vs. one with elegant formatting but subtly incorrect calculations; (b) a report using an unconventional but valid analytical method vs. one using a standard method with shallow reasoning; (c) reports of varying length with identical analytical content. If the LLM judge (and human annotators) systematically prefer the well-formatted-but-wrong or the conventional-but-shallow outputs, this reveals biases that the agreement metrics miss. Such biases are important because they would mean DAComp's DA Score rewards surface features over substantive quality — and because any rubric-based evaluation in any domain faces analogous risks. The experiment would also test whether rubric items that explicitly require "evidence from output" (as the DAComp rubric prompt mandates) provide sufficient protection against these biases.

Cross-benchmark correlation study: does DAComp score predict real-world data engineering performance? DAComp establishes internal validity (the metrics measure what they claim to measure) but not external validity (performance on DAComp predicts performance on actual enterprise data tasks). A correlation study would deploy the same agents tested on DAComp on a set of proprietary enterprise data engineering and analysis tasks (e.g., building a pipeline for a company's internal Salesforce data, analyzing customer churn), scored by professional data engineers and analysts. The key question: does the DAComp leaderboard (Tables 3, 5) predict the enterprise-task leaderboard? Strong correlation (τb > 0.8) would validate DAComp as a proxy for real-world capability; weak correlation would suggest that the benchmark's task distribution or evaluation methodology does not capture the skills that matter in practice, or that the gap between synthetic and real data is a confound. This experiment is expensive (requiring enterprise partners and professional evaluators) but essential for establishing DAComp's practical value beyond academic benchmarking.

Human performance baseline on DAComp tasks. The absence of a human ceiling (Section 6) makes it impossible to calibrate the absolute difficulty of DAComp. A human study — even on a 20–30 task subset — with professional data engineers attempting DE-Impl and DE-Evol tasks and professional data analysts attempting DA tasks, under time constraints comparable to the agent evaluation protocol (~1–2 hours per task), would provide the missing reference point. The key measurements: (a) What CS/CFS/SR do humans achieve on DE tasks? If human SR is also ~20–40%, then DAComp-DE is genuinely hard even for experts, and the paper's framing of LLM performance should be tempered. If human SR is >80%, the capability gap is enormous and the benchmark is appropriately calibrated as a stretch goal. (b) What dimensional profile do humans show on DA tasks? The paper finds that LLMs exhibit systematic weaknesses on Insightfulness and Analytical Depth relative to Accuracy (Table 5); if humans show the opposite pattern (lower computational Accuracy but higher Insightfulness), this would clarify the qualitative difference between human and machine analytical reasoning. (c) Do human errors show the same cascading pattern (Table 18) or do humans contain errors more effectively? This would reveal whether cascading failure is an inherent property of complex pipelines or a specific weakness of current LLM-based agents.

Automated rubric generation from task specifications and reference solutions. The paper's rubric construction pipeline (Section 2.3, Stage 3) requires 3+ expert annotators per task, compute anchor values manually, and validate against five LLM outputs. This cost limits the benchmark's scalability and prevents practitioners from creating custom rubrics for their own analytical tasks. A follow-up could explore semi-automated rubric generation: given a task specification and a set of reference solutions (produced by human experts or diverse LLMs), can an LLM (a) identify methodologically distinct solution paths by clustering the reference solutions by approach, (b) extract computational steps and anchor values by executing the reference solutions and recording intermediate outputs, and (c) draft rubric items that a human validator can review and correct? The evaluation would compare automatically generated rubrics against the gold-standard human-authored rubrics on a subset of DAComp tasks, measuring both scoring agreement (do the auto-generated rubrics produce similar scores when used by the same LLM judge?) and path coverage (do the auto-generated rubrics enumerate all valid paths that humans identified?). A successful demonstration would dramatically increase the practical adoptability of rubric-based evaluation.

Practical Applications and Downstream Use Cases

Agent selection and capability diagnosis for enterprise data platform teams. An organization evaluating whether to deploy LLM-based agents for internal data engineering or analysis can use DAComp as a capability audit. Rather than relying on vendor claims or single-metric benchmarks (BIRD accuracy, DS-1000 pass rate), the team can run candidate agents on DAComp and inspect the dimensional breakdown. The DE results (Table 3) reveal whether an agent's failure is primarily code-generation (low CS) or orchestration (large CS-CFS gap) — which dictates whether the remediation is better prompting/SQL training or infrastructure investment in dependency-tracking middleware. The DA results (Table 5) reveal whether an agent is a "calculator" (high Accuracy, low Insightfulness/Readability — like o3 or Qwen3-Coder) or a "communicator" (balanced profile — like GPT-5), which determines whether the agent can produce stakeholder-ready deliverables or requires human analysts to interpret raw outputs. The concrete benefit is replacing a binary "deploy or don't" decision with a differentiated deployment strategy: use the agent for tasks matching its dimensional strengths, supplement with human review for its weaknesses, and track dimensional scores over time as the agent (or its successor) improves.

Curriculum design for data agent training. The paper's error taxonomy (Section 3.4, Tables 15–19) and difficulty hierarchy (Figure 5, Figure 8) provide a capability roadmap for training better data agents. A research team developing a new data agent can use DAComp's layer-wise and objective-wise breakdown to prioritize training data collection. The finding that dependency errors dominate (Table 16) and that the intermediate (core) layer is the sharpest failure point (Figure 5b) suggests that training data should emphasize multi-table joins, business logic integration, and lineage tracking — not just single-table transformations. The finding that Strategic tasks are systematically harder than Descriptive tasks across all models (Figure 8) suggests that training on prescriptive reasoning (generating recommendations from data) should be prioritized over training on descriptive reporting. The finding that medium-scale edits are paradoxically hardest (Figure 4) suggests that training on targeted, logic-dense code modifications is more valuable than training on either trivial fixes or large-scale boilerplate. The concrete benefit is replacing uniform data collection ("more data tasks") with difficulty-aware collection that targets the specific bottlenecks DAComp diagnoses.

Benchmarking-as-a-service for the data agent ecosystem. As multiple organizations develop proprietary data agents (cloud providers, database vendors, enterprise AI platforms), DAComp provides a standardized, third-party evaluation that can serve a function analogous to SWE-Bench for coding agents or MMLU for general knowledge. An organization releasing a new data agent can report DAComp scores (DE Score, DA Score with dimensional breakdown, SR on Implementation/Evolution) as a standardized capability label. The cross-judge consistency (Table 8, τb = 1.00) and hyperparameter robustness (Table 9, τb = 1.00) mean that scores are reproducible across evaluation runs and evaluator choices, making them suitable for public leaderboards. The Chinese adaptation (DAComp-zh, Tables 4 and 6) with near-identical results enables apples-to-apples comparison for Chinese-optimized agents. The concrete benefit is reducing information asymmetry in the data agent market — purchasers can compare agents on a common, realistic benchmark rather than relying on vendor-curated demonstrations.

Internal evaluation pipeline for continuous agent improvement. A team iterating on a data agent (e.g., improving its prompt, fine-tuning on new data, upgrading the base model) can integrate DAComp into their CI/CD pipeline as a regression and progression test. After each change, running the agent on DAComp (or a representative subset) and tracking the dimensional scores reveals whether the change improved the targeted capability (did Accuracy go up?) without regressing on others (did Readability or Insightfulness go down?). The cascading metrics (CFS vs. CS) are particularly valuable here — an improvement that raises CS but lowers CFS (because it introduces new dependency errors) is a net regression for real-world deployment, even though a CS-only benchmark would report progress. The concrete benefit is catching capability tradeoffs early, before they manifest in production failures. The Max@8 reporting protocol provides a template for statistical reliability: running the agent multiple times per task and reporting both mean and maximum scores accounts for stochastic variation and reveals whether improvements are consistent or lucky.