ArXiv: 2509.25084

🎯 Pitch

An open-source 14B model trained on only 12K synthesized trajectories surpasses GPT-5 and DeepSeek-V3.1 on multi-turn data analysis by combining SFT and RL with a dynamic loss schedule. The key insight is that self-consistency filtering during data curation matters far more for final performance than simply selecting the single best trajectory.

1. Executive Summary

This paper introduces DATAMIND, a scalable data synthesis and agent training recipe for constructing generalist data-analytic agents that process diverse-format data files through multi-turn code generation. The approach addresses three core challenges—insufficient data resources, improper training strategy, and unstable code-based multi-turn rollout—by combining a fine-grained task taxonomy with recursive easy-to-hard query composition, knowledge-augmented trajectory sampling with self-consistency filtering, a dynamically adjustable training objective blending SFT and RL losses (with a cosine-annealed coefficient γ that decays from 0.9 to 0.05), and a memory-frugal asynchronous rollout framework with chunked code maintenance and per-trajectory sandboxing. Trained on the curated DATAMIND-12K dataset (11,707 high-quality trajectories spanning 18 task categories across .csv, .xlsx, and .sqlite formats), DATAMIND-14B achieves a state-of-the-art average score of 71.16% across DABench, TableBench, and BIRD, outperforming proprietary baselines including DeepSeek-V3.1 and GPT-5, while DATAMIND-7B attains 68.10% as the best open-source model. The analysis establishes that self-consistency filtering is more critical than best-trajectory selection for data quality, that SFT loss acts as an effective stabilizer for RL training yet can cause entropy collapse when dominant throughout training, and that RL narrows but cannot reverse the performance ordering between base models of different capacities.

2. Context and Motivation

The Core Problem: Open-Source Models Cannot Perform Real-World Data Analysis

The fundamental gap this paper addresses is deceptively simple to state but enormously difficult to solve: open-source language models cannot reliably perform the multi-step, code-driven data analysis that real-world users need. Current open-source agents can handle toy table understanding tasks—questions about small tables that fit entirely within a prompt—but they break down dramatically when confronted with the messy realities of actual data work: diverse file formats (CSV, Excel, SQLite), large-scale files that cannot be inlined into a context window, and long-horizon reasoning chains requiring multiple rounds of code generation, execution, and reflection.

This gap is not a minor performance difference. The paper's own baselines quantify the severity in Table 1: a vanilla Qwen-2.5-Coder-7B running a standard ReAct prompt achieves only 11.26% average accuracy across three data analysis benchmarks. On DABench, which features large-scale CSV files, it scores 15.05%. On BIRD's multi-table SQL analysis, it drops to 7.02%. This is not "room for improvement"—it is outright unusability for any practical purpose. The model cannot reliably load data, cannot reason about schema, and cannot recover from execution errors.

The significance of closing this gap extends far beyond benchmark scores. Automated data analysis is positioned by the paper as "an essential pillar of AI for scientific research" and a "critical role in realizing Innovating AI" (Section 1). The vision is that LLM agents should accelerate scientific discovery by autonomously processing datasets, generating insights, and supporting evidence-based decision-making—tasks that currently consume enormous human analyst time. Without open-source solutions that can handle real data formats and multi-step workflows, this vision remains locked behind proprietary APIs, limiting reproducibility, customization, and cost-sensitive deployment.

Why This Problem Is Hard: Three Interlocking Challenges

The paper structures its motivation around three specific challenges that collectively explain why no prior work had produced a competent open-source data-analytic agent. Understanding these challenges is essential because the DATAMIND pipeline is engineered specifically to address each one.

Challenge 1: Insufficient data resources. Training a specialized agent requires large-scale collections of tasks paired with step-by-step solution trajectories. However, publicly available data analysis benchmarks overwhelmingly provide only evaluation sets—they include questions and ground-truth answers for testing, but no executable code trajectories showing how an agent should arrive at those answers. Even when trajectories exist, they are typically produced by proprietary models (GPT-4, etc.) and are therefore scarce, expensive to scale, and potentially biased toward the generating model's idiosyncrasies.

The paper is explicit about this scarcity: "publicly available data analysis benchmarks often only provide a limited test set for evaluation purposes and lack step-by-step trajectory annotations, making it infeasible to assemble an effective training corpus from off-the-shelf resources" (Section 1). This is not merely an inconvenience—it is a structural barrier. Without trajectories, there is nothing to supervise on. The few existing trained models (TableLLM, Table-R1) inherited the limitations of their training data: they learned from small-scale tables that fit in prompts and never encountered the multi-turn code-execution loops that real analysis demands.

Challenge 2: Improper training strategy. Even when training data exists, the standard SFT-then-RL paradigm that has proven effective for math and code reasoning does not obviously transfer to agentic data analysis. The paper notes that "in a new scenario, it remains unclear how to stabilize long-horizon agent training and how to allocate training steps across SFT and RL to achieve optimal performance" (Section 1).

This is a deeper problem than it sounds. In math reasoning, the model generates a single chain of thought and arrives at an answer. In data analysis, the model interacts with an external environment—a code interpreter—that introduces compounding errors across multiple turns. A code error on turn 3 means all subsequent turns are conditioned on garbage state. The standard SFT-then-RL pipeline, where SFT provides a warm start and RL continues from there, was designed for single-turn or short-horizon settings. The paper's experiments (Table 2) confirm that this naive transfer fails: pure SFT achieves 62.54%, SFT-then-RL adds marginal gains to 63.42%, but zero-RL (RL without SFT pre-training) actually underperforms SFT alone at 58.03%. The training instability is severe enough that the authors "need to run many trials and select a relatively good checkpoint that performs best on the validation set" for both zero-RL and SFT-then-RL.

Challenge 3: Unstable code-based multi-turn rollout. This is an engineering challenge with profound training implications. During RL training, the agent must interact with a code execution environment across thousands of rollouts simultaneously. Each rollout involves file I/O, Python/SQL execution, and state management. The paper identifies that "massive concurrent file I/O and code execution can easily lead to environment crashes, especially with limited memory resources" (Section 4, "Agentic Code-based Multi-turn Rollout"). If the environment crashes, rollouts are lost, gradients become noisy or degenerate, and training destabilizes. This is not a theoretical concern—it is the practical bottleneck that the paper's memory-frugal rollout framework (asynchronous interaction, chunked code maintenance, security sandboxing) is specifically designed to address.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work, each with distinct limitations that DATAMIND aims to overcome.

Prompt-engineered agents on proprietary models. The dominant approach in data analysis agents (DS-Agent, AutoKaggle, Data-Copilot, Data Interpreter, AgenticData) relies on carefully crafted prompts and multi-agent scaffolds wrapped around closed-source models like GPT-4. These systems achieve impressive results—Data Interpreter reports 94.93% on DABench with GPT-4o—but their performance is fundamentally tied to the underlying proprietary model. The prompt engineering and workflow orchestration contribute, but they cannot compensate for a weaker base model. As the paper demonstrates in Table 5, Data Interpreter's scaffold degrades when applied to open-source models: on Qwen-2.5-Coder-7B, it achieves 54.09% on DABench (better than vanilla ReAct's 15.05%, but far below DATAMIND's 77.30%), and on TableBench it reaches only 27.99% versus DATAMIND's 67.60%. The scaffolds are brittle, model-specific, and do not transfer.

Moreover, these prompt-engineered systems are not trainable. They treat the model as a black box and add complexity outside it (workflow graphs, case-based reasoning, multi-agent decomposition). This means they cannot benefit from scaling laws in training data or compute—the only way to improve them is better prompts or better proprietary models. The DATAMIND approach inverts this: train the model itself to internalize the reasoning patterns, making prompt scaffolds unnecessary.

Open-source trained models on limited tasks. TableLLM and Table-R1 represent the state of prior open-source training efforts for tabular data. Both are trained on TableInstruct, a dataset of ~20K table QA instances. Their fundamental limitation is that they were designed for tables small enough to fit in a prompt—the model receives the entire table as text and reasons over it in a single pass. When confronted with DABench's large-scale CSV files (which cannot be inlined) or BIRD's multi-table database schemas (which require SQL generation and iterative query refinement), these models fail catastrophically. TableLLM-7B achieves 11.99% on BIRD and 36.71% on DABench. Table-R1-7B is even worse on BIRD at 10.69%. The paper explicitly notes this: "TableLLM and Table-R1 are limited to small-scale tables. When evaluated on DABench's large-scale tables, they fail to generalize, and their accuracy deteriorates even further on BIRD's multi-table analysis" (Section 5.2).

This limitation is baked into their training data. TableInstruct contains exclusively small-table QA pairs with no code execution, no multi-turn interaction, and no environment feedback. The models never learn to write code, interpret execution results, or recover from errors because they never encountered these situations during training.

SQL-specialized models that overfit to one format. OmniSQL and SQL-R1 are trained on massive Text-to-SQL corpora (2.5M instances for OmniSQL) and achieve strong performance on BIRD—57.11% for OmniSQL-7B, competitive with much larger models. However, this specialization comes at a severe cost. When evaluated on DABench's CSV-based analysis tasks (even after converting tables to SQLite format), OmniSQL-7B drops to 26.46%. The paper's diagnosis is clear: "SQL-oriented models still underperform... the breadth of query types and file formats covered by DATAMIND-12K" is what enables generalization, and specialized models "degrade sharply when confronted with unseen data" (Section 5.2).

This pattern reveals a deeper issue with specialization in agent training. Training narrowly on one task format (SQL generation) produces models that cannot generalize to related but different tasks (CSV analysis with Python). The models have learned format-specific heuristics rather than general data analysis reasoning. DATAMIND's training data intentionally spans .csv, .xlsx, and .sqlite formats with 18 task categories to force the model to learn transferable reasoning patterns.

How This Paper Positions Itself Relative to Existing Work

DATAMIND positions itself at the intersection of three research trajectories: agent training through SFT, reinforcement learning for reasoning models, and automated data synthesis. The paper is not proposing a fundamentally new algorithm—it uses established techniques (DAPO for RL, model-as-judge for evaluation, ReAct for agent architecture). Rather, its contribution is an integrated systems recipe that makes these techniques work for a challenging domain where naive applications fail.

Relative to agent training work (Agent-FLAN, AgentTuning, FireAct): These prior efforts established that SFT can equip open-source models with agentic capabilities—tool use, multi-turn interaction, planning. They focused primarily on SFT alone and on general-purpose agent tasks (web navigation, API calling). DATAMIND extends this lineage by showing that domain-specific agent training requires (1) domain-specific data synthesis at scale, (2) careful trajectory quality control through self-consistency filtering, and (3) a hybrid SFT+RL objective that stabilizes training. The paper also provides evidence that pure SFT (62.54%) leaves substantial performance on the table compared to the hybrid approach (68.10%), suggesting that prior SFT-only agent training work may have under-exploited RL's potential.

Relative to RL-for-reasoning work (DeepSeek-R1, DAPO, GRPO variants): The explosion of RL-trained reasoning models demonstrated that RL can elicit complex reasoning behaviors from base models. However, these methods were developed and validated on math and code tasks with clean, verifiable reward signals (exact string match, unit tests). DATAMIND extends them to a setting with fuzzy rewards (model-as-judge for descriptive answers), multi-turn environmental interaction, and distributional drift from execution feedback. The paper's findings—that SFT loss is an "effective stabilizer" but can also "be the culprit of unstable training" (Section 5.4), and that RL "narrows the performance gap between different base models, but can hardly reverse the order" (Section 5.4)—add to the broader understanding of when and how RL helps agent training.

Relative to data synthesis work (OpenThoughts, STaR, ReST): Prior work on synthetic data for reasoning focused on generating correct answers and filtering for correctness. DATAMIND introduces two key refinements specific to agent trajectories: (1) knowledge-augmented sampling, where human-crafted procedural knowledge steers the trajectory generation process, and (2) self-consistency filtering with a reflection loop, where inconsistent trajectories are fed back to the model with judge critique to produce improved versions. The paper's ablation (Figure 4) demonstrates that self-consistency filtering is more important than selecting the "best" trajectory among consistent ones—a finding that challenges the prevailing emphasis on trajectory quality over diversity.

The paper's central positioning claim is that it is "the first to systematically investigate the scaling of agent post-training data and multi-turn RL training in the data-analytic scenario" (Section 6, extended related work). This is a specific but defensible claim: prior work either trained agents with SFT-only, applied RL to non-agentic reasoning, or built prompt-engineered systems on proprietary models. No prior work combined large-scale trajectory synthesis, self-consistency filtering, hybrid SFT+RL training with dynamic weighting, and memory-frugal multi-turn rollout into a unified pipeline that produces open-source agents competitive with the strongest proprietary models.

A subtle but important aspect of the positioning: the paper does not claim its individual components are novel. The task taxonomy builds on Wu et al. (2025b), the self-consistency filtering parallels rejection sampling, the dynamic SFT-RL weighting follows Zhang et al. (2025c), and the DAPO algorithm is from Yu et al. (2025). The novelty lies in (1) integrating these components into a working system for a domain where prior integration attempts failed, (2) identifying why naive combinations fail (entropy collapse, trajectory collapse, environment crashes), and (3) providing empirical insights (the "three insights" highlighted in the abstract and Section 5.4) that generalize beyond the specific system.

3. Technical Approach

3.1 Reader Orientation

The paper describes a full data synthesis and agent training pipeline—called DATAMIND—that starts from raw data files scraped off the internet and ends with a 7B or 14B language model that can autonomously analyze spreadsheets and databases by writing and executing code across multiple turns. The problem it solves is that no open-source model can competently perform real-world data analysis, and the solution's shape is a three-stage pipeline: (1) programmatically generate diverse, difficult queries and corresponding expert trajectories from collected data files, (2) filter those trajectories through self-consistency and rule-based checks to produce a clean training set, and (3) train a model using a dynamically-weighted combination of supervised fine-tuning and reinforcement learning losses with a custom-built stable multi-turn rollout environment.

3.2 Big-Picture Architecture (Diagram in Words)

The DATAMIND system consists of five major components connected in a linear pipeline with one feedback loop:

  1. File Collection — scrapes and filters .csv, .xlsx, and .sqlite files from Kaggle, BIRD, and OmniSQL to create a diverse corpus of raw data files (5,914 files retained after filtering).

  2. Query Synthesizer — extracts metadata from each file, uses a fine-grained 18-category task taxonomy with few-shot examples to prompt DeepSeek-V3 to generate diverse queries, then applies recursive easy-to-hard composition (chaining 2–5 task types) to increase difficulty.

  3. Trajectory Synthesizer — takes each query and its data file, uses knowledge-augmented prompting (human-written procedural workflows per task category) to guide DeepSeek-V3.1 to generate multi-turn code-execution trajectories, samples three independent trajectories per query, and runs them through a GPT-4o-mini judge for self-consistency checking. Inconsistent trajectories trigger a reflection loop where the judge's critique is fed back to the model for re-generation.

  4. Trajectory Filter — applies model-based filtering (keep only self-consistent trajectories, optionally select the best one) and rule-based filtering (format compliance, answer length ≤ 1,024 tokens, linguistic integrity) to produce the final DATAMIND-12K dataset of 11,707 trajectories.

  5. Agent Trainer — takes a base model (Qwen-2.5-Coder-7B or 14B) and trains it with a combined SFT + DAPO reinforcement learning objective, where a coefficient γ (cosine-annealed from 0.9 down to 0.05) dynamically balances exploitation of expert data against exploration through RL. The training runs inside a custom stable multi-turn environment with asynchronous model generation and code execution, chunked code maintenance, and per-trajectory sandboxing.

Information flows sequentially: raw files → synthetic queries → expert trajectories → filtered training set → trained agent. The only feedback loop is the self-consistency reflection step, where judge critique on inconsistent trajectories feeds back into trajectory re-generation.

3.3 Roadmap for the Deep Dive

  • First, the data file collection and filtering process, because the files are the foundation on which all queries and trajectories are built. Understanding what files are included—and excluded—explains the diversity and difficulty range of the final training data.

  • Second, the query synthesis mechanism (task taxonomy + recursive composition), since queries determine what reasoning patterns the agent will learn. The taxonomy's 18 categories and the easy-to-hard composition scheme are the primary drivers of training set diversity and difficulty.

  • Third, the knowledge-augmented trajectory sampling and self-consistency filtering pipeline, because this is where raw model outputs become training data. The reflection loop, judge model, and filtering criteria are the quality-control mechanisms that distinguish DATAMIND-12K from naively generated trajectory sets.

  • Fourth, the training objective design (SFT loss, DAPO RL loss, dynamic γ scheduling), since this is the core algorithmic contribution for stabilizing long-horizon agent training. The void-turn filtering and cold-start strategy complete the training stability picture.

  • Fifth, the multi-turn rollout infrastructure (asynchronous interaction, chunked code maintenance, security sandboxing), because RL training for code-executing agents cannot function without a stable environment. This is the engineering foundation that makes the algorithmic contributions feasible.

  • Sixth, the reward design (format reward, answer reward via model-as-judge, length penalty), since the reward signal determines what behaviors RL reinforces. The length penalty in particular addresses a subtle failure mode—reward hacking through verbose outputs—that is specific to descriptive-answer tasks.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a carefully designed data synthesis pipeline, combined with a dynamically-balanced SFT+RL training objective and a stable multi-turn execution environment, can produce open-source data-analytic agents that match or exceed proprietary frontier models.

Data File Collection and Filtering

The data synthesis pipeline begins by constructing a large, diverse corpus of raw data files from which queries can be generated. The paper draws files from three sources with distinct formats and domains, applying format-specific filtering to ensure usability.

Source 1: Kaggle spreadsheets. Using the official Kaggle API, the authors crawl "a diverse subset of files spanning multiple domains" from Kaggle's repository of tens of thousands of .csv and .xlsx files (Section 3.1). Each downloaded file passes through three exclusion filters. First, files that cannot be loaded at all—corrupted formats, encoding errors, missing data—are discarded. Second, files that are too small (fewer than 20 rows) or too large (more than 1,000 rows) are discarded. The lower bound excludes degenerate files that provide no analytical depth; the upper bound excludes files whose scale would make trajectory synthesis computationally prohibitive or whose content exceeds reasonable context windows. Third, files containing anomalous data types—presumably binary blobs, nested structures, or non-tabular formats—are discarded. After filtering, the pipeline retains 3,400 .csv files and 560 .xlsx files.

Source 2: Database files from Text-to-SQL benchmarks. For .sqlite database files, the paper draws from the training sets of BIRD (Li et al., 2023b) and OmniSQL (Li et al., 2025a). Both are high-quality corpora widely used in the Text-to-SQL community, meaning they contain realistic database schemas with multiple interrelated tables. The authors "sample from these sources and apply an analogous filtering pipeline," though the exact filtering criteria for database files are not specified separately. The result is 1,954 .sqlite files.

The combined corpus of 5,914 files spans three file formats (.csv, .xlsx, .sqlite) and multiple domains (Kaggle's diverse topical coverage plus the domain-specific databases in BIRD and OmniSQL). This format diversity is essential for training a generalist agent: a model that only sees .csv files will never learn SQL generation, and a model that only sees SQLite schemas will never learn pandas-based CSV analysis.

Design rationale for file size bounds. The 20–1,000 row filter for spreadsheets is a practical engineering choice, not a statistical one. Files below 20 rows are trivial—they contain so little data that analysis questions become trivially answerable by inspection, providing no training signal for multi-step reasoning. Files above 1,000 rows are not inherently impossible but strain the trajectory synthesis pipeline: the expert model (DeepSeek-V3.1) must process the file in a context window, and each trajectory turn generates code that executes against the full dataset. The paper does not discuss whether larger files could be handled through chunking or sampling strategies—this is implicitly left as future work.

Query Categorization and Synthesis

With raw files in hand, the next step is generating specific analysis questions (queries) for each file. The paper designs this process to maximize two properties: diversity (queries should span many different types of analytical reasoning) and difficulty (queries should range from simple single-step operations to complex multi-hop chains).

Step 1: Metadata extraction. An automated script processes each data file to extract meta-information, denoted $d$ in the problem definition. This includes table headers, column names, data types for each column, and representative rows—essentially everything needed to understand the file's structure and content without seeing every row. For .sqlite files, the script extracts the database schema (table names, column names, foreign key relationships). This metadata serves as the input to the query generation model: the model never sees the raw file, only its structural description, which prevents it from simply reading answers off the data and forces it to formulate questions that require actual computation.

Step 2: Fine-grained task taxonomy. To ensure diversity, the paper refines a taxonomy from Wu et al. (2025b) into 18 fine-grained task categories (shown in Figure 1a). The categories span a wide spectrum:

  • Basic operations: Aggregation, Counting, Comparison, Arithmetic Calculation, Ranking
  • Statistical reasoning: Statistical Analysis, Distribution Analysis, Correlation Analysis, Causal Analysis
  • Contextual analysis: Time-based Calculation, Impact Analysis, Anomaly Detection, Fact Checking
  • Structural operations: Feature Engineering, Data Preprocessing, Descriptive Analysis
  • Complex reasoning: Multi-hop Numerical Reasoning, Domain Specific Numerical Reasoning

For each category, the authors manually curate 4–6 exemplar queries that "vary in complexity and domains" to serve as few-shot demonstrations. These exemplars encode two things simultaneously: the format of a well-formed query in that category (phrasing patterns, typical syntactic structures) and the type of reasoning expected (what kind of answer the query demands). The prompt for query synthesis (Appendix I.2) includes these exemplars along with the file metadata and a strict instruction to generate questions that "primarily focus on the given type."

The query synthesis model is DeepSeek-V3 (the original V3, not V3.1, which is used later for trajectory generation). The taxonomy-based generation means each file yields up to 18 queries, one per category, though the actual number varies because not all categories are applicable to every file (e.g., a file without time columns cannot generate Time-based Calculation queries).

Step 3: Recursive easy-to-hard composition. The 18 individual task categories generate queries of moderate difficulty—each requires reasoning in one category, with a single analytical operation. To create genuinely hard queries that demand multi-step reasoning, the paper introduces a recursive composition scheme. The mechanism works as follows:

  1. Generate an initial query $q_1$ for a specific task type (e.g., Aggregation: "What is the average salary by department?").
  2. The output of answering $q_1$ becomes an input to a second query $q_2$ of a potentially different type (e.g., Comparison: "Which department's average salary is highest?").
  3. Repeat the chaining process, feeding the output of each step as input to the next.

The paper chains 2–5 task types iteratively, "progressively amplifying the difficulty" and creating "multi-hop analytic challenges that go well beyond the capability required by any single task type." The exact prompt for composition is not shown, but the mechanism is conceptually equivalent to building a dependency graph of sub-questions where each subsequent question depends on the answer to the previous one.

Why recursive composition rather than manually writing hard queries? The recursive scheme leverages the existing taxonomy and exemplars to automatically generate hard queries without requiring human experts to craft complex multi-step questions for each file. It also ensures that the difficulty comes from genuine analytical chaining (you must compute X before you can answer about Y) rather than from obscure domain knowledge or trick phrasing. This is a scalable approach to difficulty: the same few-shot exemplars that produce simple queries can, through compositional iteration, produce queries that require coordinating multiple reasoning steps across different task types.

Design choice: why 18 categories rather than fewer? The taxonomy granularity determines the diversity of reasoning patterns in the training data. Too few categories (e.g., 5 broad types) would collapse distinct reasoning skills—Correlation Analysis and Causal Analysis require fundamentally different statistical thinking, and training only on one would leave the model unprepared for the other. Too many categories (e.g., 50 highly specific types) would fragment the data, with too few training examples per category to learn generalizable patterns. Eighteen categories balances coverage with per-category data volume.

Problem Formalization and Agent Architecture

Before proceeding to trajectory synthesis, the paper formalizes the data analysis task and the agent architecture. This formalization defines the input-output contract that trajectory generation must satisfy.

A data analysis task is a quadruple:

u=(q,f,d,a)u = (q, f, d, a)

where $q$ is the user query (the question text), $f$ is the data file (in .csv, .xlsx, or .sqlite format), $d$ is an optional data description (the metadata extracted during query synthesis), and $a$ is the answer (generated during trajectory synthesis through the judge model's best-trajectory selection).

What this formulation does: it establishes that the input to the agent is not just a question but a question paired with a specific data file and its structural description. This is non-trivial because in real-world data analysis, the same question worded identically can have different answers for different datasets—the file context is part of the task specification, not just background knowledge.

The agent follows the ReAct (Reasoning + Acting) paradigm. At each interaction round, the agent produces a Thought (reasoning and reflection conditioned on current context) and an Action (code invocation or final answer generation). The environment returns an Observation (execution feedback from a code interpreter). Formally, the agent's trajectory up to time step $t$ is:

ht=(u,τ0,α0,o0,τ1,α1,o1,…,τt−1,αt−1,ot−1)h_t = (u, \tau_0, \alpha_0, o_0, \tau_1, \alpha_1, o_1, \ldots, \tau_{t-1}, \alpha_{t-1}, o_{t-1})

where $\tau_i$ is the thought at step $i$, $\alpha_i$ is the action at step $i$ (code or answer), and $o_i$ is the observation at step $i$ (execution output from the interpreter).

What this notation captures: the trajectory is a sequence of interleaved mental reasoning and environmental feedback. Each thought conditions on all previous thoughts, actions, and observations—meaning an error in early code execution (captured in $o_2$) can propagate through all subsequent reasoning. This is exactly why multi-turn stability is hard: the model must learn to interpret execution errors and recover, not just generate correct code on the first try.

The agent's policy $\pi_\theta(\tau_t, \alpha_t | h_t)$ generates the next thought and action conditioned on the history. A trajectory terminates either when the agent emits an answer (final action) or when a predefined maximum number of rounds $T = 10$ is reached.

The paper simplifies notation for the training sections: the input $x$ includes everything provided to the agent at the start (query $q$, file $f$, description $d$), and the trajectory $y \sim \pi_\theta(\cdot | x)$ includes the full Thought-Action-Observation sequence plus the final answer $a$.

Knowledge-Augmented Trajectory Sampling

Trajectory synthesis is where the paper generates the actual training data—the multi-turn code-execution sequences that the agent will later be trained to imitate. This is the most complex data generation step because producing a correct, executable, well-reasoned trajectory is substantially harder than producing a query: the model must not only understand what analysis to perform but also generate valid code, interpret execution results, and decide when to stop.

Step 1: Procedural knowledge injection. For each of the 18 task categories, the authors "manually craft a high-level workflow $k$ that encodes procedural knowledge and steers the model during trajectory synthesis" (Section 3.2). The paper does not show these workflows directly, but they are described as encoding the typical steps an analyst would follow for that category. For example, a workflow for Correlation Analysis might encode: inspect data types → select numeric columns → check for missing values → compute correlation matrix → identify strongest correlations → report findings. This procedural knowledge is not the same as a rigid script; it is a flexible guide that gives the model a structured approach without constraining it to a single path.

Why inject procedural knowledge rather than let the model discover its own approach? The expert trajectory generator is DeepSeek-V3.1, which is a powerful model but still makes errors in multi-step code generation. The procedural knowledge serves as a scaffold: it reduces the search space for the model by providing a high-level plan, making it more likely to generate correct trajectories on the first attempt (or within the reflection loop). Without this scaffold, the model might skip critical steps (e.g., forget to check for null values before computing correlations) or approach the problem in a disorganized way that produces correct answers by accident rather than through sound methodology.

Step 2: Self-consistency sampling. For each query, the trajectory generator samples $N = 3$ independent trajectories from the expert policy:

{yi}i=1N∼πθexpert(⋅∣k,x)\{y_i\}_{i=1}^{N} \sim \pi_{\theta_{\text{expert}}}(\cdot | k, x)

where $\pi_{\theta_{\text{expert}}}$ is DeepSeek-V3.1 prompted with the procedural knowledge $k$ and the task input $x$. Sampling three trajectories rather than one increases the probability that at least some trajectories are correct, and—crucially—provides a signal for quality filtering: if three independent attempts converge to the same answer, that answer is more likely to be correct than if they diverge.

Step 3: Consistency judging. A judge model $M$ (GPT-4o-mini) evaluates whether the three trajectories' final answers are consistent:

{c,s,y}=M({yi}i=1N)\{c, s, y\} = M(\{y_i\}_{i=1}^{N})

where $c$ is the chain-of-thought reasoning of the judge model explaining its decision, $s \in \{0, 1\}$ is the binary consistency verdict, and $y$ is the selected best trajectory among the consistent ones. The judge's evaluation criteria, specified in the prompt (Appendix I.4), include:

  1. Semantic equivalence: for descriptive questions, all answers must express the same meaning even if phrased differently.
  2. Numerical agreement: for numerical questions, values must be within 3% of each other, with units converted for comparability.
  3. Completeness: the answer must address all aspects of the original question, including implicit sub-questions.
  4. Source traceability: the judge identifies which of the three answers "most clearly and accurately represents the final synthesized answer."

The trajectory selection rule is:

y={yi∈{yi}i=1N,s=1none,s=0y = \begin{cases} y_i \in \{y_i\}_{i=1}^{N}, & s = 1 \\ \text{none}, & s = 0 \end{cases}

What this means operationally: if the judge determines all three answers are semantically equivalent and correct ($s=1$), the judge also selects which single trajectory is the "best" (most concise and accurate) among them—this becomes the training instance $y$. If the judge determines the answers are inconsistent ($s=0$), no trajectory is retained from this sampling round, and the process moves to the reflection step.

Step 4: Reflection loop for inconsistent trajectories. When the consistency check fails ($s=0$), the paper does not simply discard the trajectories. Instead, the judge model's chain-of-thought critique $c$ is fed back to the expert model as external feedback, prompting it to reflect and revise:

{yireflected}i=1N∼πθexpert(⋅∣k,x,{yi}i=1N,c)\{y_i^{\text{reflected}}\}_{i=1}^{N} \sim \pi_{\theta_{\text{expert}}}(\cdot | k, x, \{y_i\}_{i=1}^{N}, c)

The reflected trajectories are then fed back into the judge model for a second consistency check and trajectory selection. This "rescue loop" serves two purposes: it "salvages additional usable data" (increasing the effective yield of the synthesis pipeline) and it "enriches the diversity of thinking patterns embedded in the trajectory pool" because reflected trajectories may exhibit different reasoning paths than the originals.

The bias toward easy queries. The paper explicitly acknowledges a built-in bias: "this pipeline inherently biases us toward easier queries whose answers are more likely to coincide" (Section 3.2). Queries where the expert model consistently produces the correct answer are by definition easier than queries where it struggles. To counteract this, the authors refine the procedural knowledge $k$ into "more granular, step-by-step instructions for categories that exhibit low inter-trajectory consistency." This makes the scaffold more detailed and constraining for harder task types, steering the model more strongly toward correct solutions and thereby increasing the consistency rate.

Expert model choice. The trajectory generator is DeepSeek-V3.1 (the 2025 release, not the original V3 used for query synthesis). This is a stronger model, chosen specifically because trajectory generation—multi-turn code generation with execution feedback—is a harder task than query generation and demands a more capable model. Using a strong proprietary model for data synthesis is a pragmatic choice: the goal is to produce the highest-quality trajectories possible, and the cost is a one-time data generation expense, not a per-inference cost.

Rule-Based Trajectory Filtering

After the model-based consistency filtering, three additional rule-based stages remove trajectories that are formally valid but would destabilize training.

1. Format compliance. The paper drops any trajectory that "deviates from the ReAct format, ensuring that every remaining trajectory can be losslessly converted into our target training schema" (Section 3.2). The ReAct format requires specific XML-like tags: thinking... and response for reasoning, <code> and </code> for code blocks, <interpreter> and </interpreter> for execution output, and <answer>...</answer> for final answers. Trajectories where the expert model fails to properly enclose code in tags, omits required tags, or nests tags incorrectly are discarded because they cannot be parsed into the structured training format.

2. Length control. Trajectories whose final answer exceeds 1,024 tokens are filtered out. The stated reason is precise: this prevents "the model from exploiting spurious hallucinations to artificially hit the correct string" (Section 3.2). The concern is specific to the model-as-judge evaluation paradigm. If the agent learns that verbose outputs—containing many possible answer formulations—tend to score well because the judge finds some substring that matches the ground truth, it will produce increasingly verbose answers rather than learning to be precise. The 1,024-token cap enforces conciseness during training.

3. Linguistic integrity. Trajectories "containing garbled text or intermingled natural languages" are removed to eliminate "samples that could destabilize the agent training" (Section 3.2). Code-switching or corrupted text in the training data would introduce noise into the language modeling objective, potentially causing the model to produce mixed-language outputs or degenerate text.

After the full filtering pipeline (consistency check + rule-based stages), the final training set contains 11,707 trajectories, named DATAMIND-12K (the "12K" is a rounded figure).

Design choice: why rule-based filtering after model-based? The model-based consistency filter checks semantic correctness—do the trajectories converge to the same right answer? The rule-based filters check syntactic validity—are the trajectories well-formed enough to use as training data? A trajectory could be semantically correct (consistent, accurate answer) but syntactically invalid (malformed tags, excessively long answer), and such trajectories would break the training pipeline even though they represent "correct" solutions. The two-stage filtering separates concerns: first ensure correctness, then ensure usability.

Supervised Fine-Tuning Loss Design

The first component of the training objective is a standard supervised fine-tuning (SFT) loss, but with one critical modification: tokens produced by the environment are explicitly masked out.

Given the training dataset $\mathcal{D}$ of expert trajectories, the SFT loss is:

LSFT(θ)=−E(x,y)∼D[∑t=1∣y∣I(yt∉o)⋅log⁡πθ(yt∣x,y<t)]\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(x,y) \sim \mathcal{D}} \left[ \sum_{t=1}^{|y|} \mathbb{I}(y_t \notin o) \cdot \log \pi_\theta(y_t | x, y_{<t}) \right]

where $\theta$ is the model parameters, $(x, y)$ is an input-trajectory pair from the dataset, $|y|$ is the token length of the trajectory, $y_t$ is the $t$-th token, $y_{<t}$ is all tokens before position $t$, $\mathbb{I}(y_t \notin o)$ is an indicator function that returns 1 when $y_t$ is not produced by the environment (i.e., not part of an observation $o$), and $\pi_\theta(y_t | x, y_{<t})$ is the model's predicted probability for token $y_t$ given the input and preceding tokens.

What it computes: standard next-token prediction cross-entropy loss, summed over all tokens in the trajectory that were generated by the agent (thoughts, code, answers), while skipping tokens that came from the code interpreter's execution output. The expectation over the dataset means this loss is averaged across all training trajectories.

Why mask environment tokens: the code interpreter's output ($o_t$) is determined by the execution environment, not by the agent's policy. Training the agent to predict what the interpreter would have said is wasted capacity—the model cannot control execution output, and learning to mimic it would consume model capacity that should be spent on learning to reason about execution output. The indicator function $\mathbb{I}(y_t \notin o)$ zeros out the loss contribution for any token that sits inside <interpreter> tags, ensuring the model only learns from tokens it actually produces during inference.

Why standard cross-entropy: the SFT stage is essentially behavioral cloning—the model is trained to imitate the expert trajectories token-by-token. Cross-entropy is the maximum-likelihood objective for this task. Alternative losses like policy gradient would add unnecessary complexity during the SFT phase, where the goal is simply to absorb knowledge from the expert data efficiently.

Reinforcement Learning Loss Design (DAPO)

For the RL component, the paper adopts the Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) algorithm from Yu et al. (2025). DAPO is a variant of the GRPO (Group Relative Policy Optimization) family, which uses groups of trajectories sampled from the current policy to compute advantages relative to the group mean, avoiding the need for a separate value function.

The DAPO objective is:

LDAPO(θ)=−E(x,y)∼D,{yi∗}i=1G∼πθold(⋅∣x)[1∑i=1G∣yi∗∣∑i=1G∑t=1∣yi∗∣min⁡(ri,t(θ)A^i,t,  clip(ri,t(θ),1−εlow,1+εhigh)A^i,t)]\mathcal{L}_{\text{DAPO}}(\theta) = -\mathbb{E}_{(x,y) \sim \mathcal{D}, \{y^*_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{\sum_{i=1}^{G} |y^*_i|} \sum_{i=1}^{G} \sum_{t=1}^{|y^*_i|} \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \; \text{clip}\left(r_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right) \hat{A}_{i,t} \right) \right]

subject to the constraint:

0<∣{yi∗∣is_equivalent(y,yi∗)}∣<G0 < \left| \{ y^*_i \mid \text{is\_equivalent}(y, y^*_i) \} \right| < G

Symbol breakdown:

  • $\mathcal{D}$ is the training dataset containing expert trajectories $y$.
  • $\{y^*_i\}_{i=1}^{G}$ is a group of $G = 4$ trajectories sampled from the agent's current (old) policy $\pi_{\theta_{\text{old}}}$.
  • $|y^*_i|$ is the token length of the $i$-th sampled trajectory.
  • $r_{i,t}(\theta) = \frac{\pi_\theta(y^*_{i,t} | x, y^*_{i,<t})}{\pi_{\theta_{\text{old}}}(y^*_{i,t} | x, y^*_{i,<t})}$ is the per-token importance sampling ratio—how much more (or less) likely the current policy $\pi_\theta$ is to produce token $y^*_{i,t}$ compared to the old policy $\pi_{\theta_{\text{old}}}$ that actually generated the trajectory.
  • $\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_i\}_{i=1}^{G})}{\text{std}(\{R_i\}_{i=1}^{G})}$ is the group-normalized advantage for trajectory $i$, where $R_i$ is the scalar reward for the $i$-th trajectory computed from the reward function.
  • $\varepsilon_{\text{low}} = 0.2$ and $\varepsilon_{\text{high}} = 0.28$ are the asymmetric clipping bounds.
  • $\text{is\_equivalent}(y, y^*_i)$ checks whether the sampled trajectory's answer is equivalent to the expert answer (receives reward 1).

What it computes: a clipped policy gradient objective that increases the probability of trajectories with above-average reward and decreases the probability of trajectories with below-average reward, while constraining how much the policy can change per update. The $\min$ operation between the unclipped ratio $r_{i,t}(\theta) \hat{A}_{i,t}$ and the clipped version implements PPO-style trust-region clipping: if the advantage is positive (good trajectory), the objective is clipped at $1 + \varepsilon_{\text{high}}$ to prevent the policy from increasing probability too aggressively; if the advantage is negative (bad trajectory), the objective is clipped at $1 - \varepsilon_{\text{low}}$ to prevent excessive probability decrease.

Why asymmetric clipping bounds ($\varepsilon_{\text{low}} = 0.2, \varepsilon_{\text{high}} = 0.28$): standard PPO uses symmetric clipping (typically $\varepsilon = 0.2$). The paper's asymmetric bounds, inherited from DAPO, allow slightly more aggressive policy improvement (higher upper clip) while maintaining conservative policy deterioration (standard lower clip). This asymmetry is motivated by the observation that in RL training, overly conservative clipping on the positive side can unnecessarily slow convergence.

The filtering constraint ($0 < |\{y^*_i \mid \text{is\_equivalent}(y, y^*_i)\}| < G$) discards trajectory groups that provide no meaningful learning signal. If all $G$ trajectories are equivalent to the expert (all rewards are 1), the advantage $\hat{A}_{i,t}$ is zero for all trajectories (since all $R_i$ equal the group mean), and the gradient update contributes nothing—these groups are skipped. Similarly, if no trajectory is equivalent to the expert (all rewards are 0), the advantage is also zero. The constraint ensures that each training batch contains at least one correct and one incorrect trajectory, providing a meaningful gradient signal.

Why DAPO over standard PPO: DAPO is a GRPO-family algorithm, meaning it normalizes advantages within groups rather than using a learned value function. This eliminates the need to train a separate critic network—a significant practical advantage for agent training where the state space (full trajectory history) is complex and high-dimensional. DAPO's decoupled clipping is a further refinement that the paper adopts without modification. The choice is pragmatic: DAPO is a state-of-the-art GRPO variant that the authors found to work well for their setting.

What tokens are included in the RL objective: as with SFT, "any tokens emitted by the environment are discarded when computing the objective" (Section 4). Only the agent-generated tokens (thoughts, code, final answer) contribute to the policy gradient, since the environment tokens are not under the policy's control.

Dynamic SFT+RL Objective and γ Scheduling

The core algorithmic contribution of the training methodology is the joint optimization of SFT and RL losses with a dynamically scheduled mixing coefficient. Rather than the conventional SFT-then-RL pipeline, the paper trains with a combined objective:

LFinal(θ)=γLSFT(θ)+(1−γ)LDAPO(θ)\mathcal{L}_{\text{Final}}(\theta) = \gamma \mathcal{L}_{\text{SFT}}(\theta) + (1 - \gamma) \mathcal{L}_{\text{DAPO}}(\theta)

where $\gamma \in [0, 1]$ varies dynamically throughout training.

What this computes: a weighted sum of the SFT loss and the DAPO RL loss. When $\gamma$ is large (close to 1), the model predominantly learns from expert demonstrations. When $\gamma$ is small (close to 0), the model predominantly learns from its own rollouts through RL. The combined gradient updates push the model parameters in a direction that simultaneously imitates expert behavior and optimizes the reward signal.

The γ schedule: $\gamma$ is initialized to a large value and annealed to a small value using cosine decay:

  • Peak value: $\gamma = 0.9$ — at the start of training, 90% of the gradient comes from SFT and only 10% from RL. The model first "acquires knowledge from expert data via the SFT loss" (Section 4).
  • Valley value: $\gamma = 0.05$ — by the end of training, 95% of the gradient comes from RL and only 5% from SFT. The model has "matured" and is encouraged to explore beyond expert trajectories.
  • Schedule type: cosine decay, meaning $\gamma$ decreases smoothly from 0.9 to 0.05, spending more time at intermediate values than linear decay would.

Why dynamic scheduling works—the "raising a child" analogy. The paper provides an explicit intuition in Section 5.4: "During early childhood, constant parental guidance (a large $\gamma$) is indispensable to keep the child from going astray. As the child grows up, excessive supervision stifles the child's innate drive for self-directed exploration. At that stage, judiciously letting go (a small $\gamma$) enables the child to discover their true capabilities through the feedback from the surrounding world."

This analogy maps directly to the training dynamics. Early in training, the model's randomly initialized (or cold-start) policy produces poor trajectories with near-zero rewards. RL on these trajectories provides no useful signal—the agent has no idea what a good trajectory looks like. The strong SFT signal ($\gamma = 0.9$) rapidly teaches the model the basic format, common code patterns, and reasoning structures from the expert data. As training progresses, the model becomes capable enough that its own rollouts sometimes succeed, making the RL signal informative. Gradually reducing $\gamma$ allows the model to explore variations beyond the expert trajectories, discovering strategies that may be better than what the expert model (DeepSeek-V3.1) produced.

Why not fixed γ or SFT-then-RL: The paper's analysis (Section 5.4, Figure 6) empirically validates this design. With $\gamma = 0$ (pure RL, no SFT), the answer reward "declines almost monotonically" because the 7B model's "limited multi-step reasoning capability makes it difficult to roll out high-quality trajectory groups for effective learning." With a small fixed $\gamma = 0.2$, the reward "initially rises despite large oscillations, yet the SFT loss remains too weak to prevent the policy from eventually drifting away and collapsing." With a large fixed $\gamma = 0.8$ (Figure 7), the reward "rises briefly, followed by a gradual decline" because "over-fitting to the SFT loss traps the policy in the rigid thinking patterns embedded in the expert trajectories" and causes "entropy collapse." The dynamic schedule avoids all three failure modes: strong SFT early prevents initial collapse, weak SFT late prevents entropy collapse, and the smooth transition avoids the abrupt distribution shift of SFT-then-RL.

The cold start exception. For trajectories filtered out by the DAPO constraint (all-reward-1 or all-reward-0 groups), only the SFT loss is computed. This ensures that batches with no diverse rewards still provide a stable learning signal. The paper also performs a cold start using DATAMIND-12K before the joint SFT+RL training begins—this means running a few epochs of pure SFT to initialize the model at a reasonable performance level before the RL component activates. The effect of cold start duration is analyzed in Section 5.4 (Figure 8): more cold start epochs reduce the marginal gain from subsequent RL, suggesting that SFT and RL partially substitute for each other in teaching the same capabilities.

Void Turn Filtering

A specific failure mode in multi-turn agentic RL training is trajectory collapse, where the model produces a turn that fails to generate valid code or answer, and this failure cascades through subsequent turns, eventually producing degenerate trajectories full of error messages or empty outputs. The paper observes this phenomenon directly and implements a simple countermeasure.

A void turn is defined as "an agentic loop that fails to produce a valid code snippet or answer" (Section 4). When a trajectory contains any void turn, the entire loss contributed by that trajectory is masked out—set to zero for both SFT and RL components.

Why mask the entire trajectory rather than just the void turn: the intuition is that once a void turn occurs, all subsequent turns are conditioned on invalid state. The model's reasoning after a void turn is based on a context that does not reflect actual code execution, so training on those subsequent turns would reinforce behaviors that are correct only in a corrupted context. Masking the full trajectory is a conservative approach that errs on the side of discarding potentially useful data to avoid poisoning the model with spurious patterns.

Why void turns cause trajectory collapse in RL: in RL training, the model generates trajectories, receives rewards, and updates its policy to increase the probability of high-reward trajectories. If void turns become common (which they do when the policy distribution drifts), the average reward drops, and the policy receives noisy or negative gradients. This can create a vicious cycle: lower-quality trajectories → worse gradients → worse policy → even lower-quality trajectories. The void-turn filter breaks this cycle by removing the worst trajectories from the gradient computation entirely.

Agentic Code-Based Multi-Turn Rollout Infrastructure

The engineering challenge underlying the entire training pipeline is maintaining a stable environment where thousands of parallel rollouts can execute code safely and efficiently. The paper implements three optimizations to prevent environment crashes:

1. Asynchronous interaction. The paper "asynchronizes model generation and code execution for different data samples, which can decouple peak GPU and CPU memory demands and avoid simultaneous file I/O and code-execution spikes" (Section 4). In a naive synchronous implementation, all $G = 4$ trajectories in a group would generate their next action, then all four would execute their code simultaneously, then all four would generate again. This creates correlated spikes in resource usage: GPU spikes during generation, CPU/memory spikes during execution. The asynchronous approach interleaves generation and execution across samples: while sample A is executing code on CPU, sample B can be generating on GPU, smoothing resource utilization.

Why this matters for stability: simultaneous code execution across many trajectories can overwhelm the file system (many concurrent reads of different data files) and memory (each execution environment loads libraries and data). If the environment crashes due to resource exhaustion, all in-progress rollouts are lost, and the training step fails. Asynchronization reduces peak resource demand, making crashes less likely.

2. Chunked code maintenance. The paper implements "a light-weight, notebook-style code generation strategy" (Section 4). Rather than requiring the model to regenerate its entire code history at each turn (as a typical notebook kernel would), the model "only needs to produce the code snippet required for the current reasoning step." At runtime, "we concatenate the active snippet with its predecessors, yielding the same global execution effect without the memory overhead."

What problem this solves: in standard notebook-based code execution (like Jupyter), the kernel maintains a global variable pool that grows with each executed cell. All variables defined earlier remain in memory. For long trajectories with large intermediate datasets, this memory accumulation can exhaust available RAM—especially problematic when running many trajectories in parallel. The chunked maintenance strategy avoids storing runtime state across turns. Instead, each turn's code execution starts from a clean state and re-executes all previous code chunks in sequence before executing the new chunk. This is computationally more expensive (re-execution) but dramatically reduces memory pressure (no persistent variable pool).

Why re-execution is acceptable: the cost of re-running pandas/SQL operations on the datasets used in DATAMIND (files with 20–1,000 rows) is negligible compared to the cost of model inference. The tradeoff trades a small increase in CPU time for a large decrease in memory usage and crash risk.

3. Security control. Three measures isolate each trajectory and prevent malicious or accidental harmful code:

  • Environment isolation: each trajectory runs in a "sandboxed" environment—likely a separate Python subprocess or container—so that one trajectory's code cannot interfere with another's.
  • Resource caps: per-trajectory limits on CPU time and peak memory prevent infinite loops or memory leaks from consuming shared resources.
  • Function filtering: any code snippet containing "insecure function calls" is filtered before execution. The paper does not enumerate which functions are blocked, but the category includes file system manipulation, network access, and subprocess spawning—anything that could escape the sandbox.

Additionally, an automatic package-installation mechanism dynamically checks for and installs missing Python packages. This is necessary because the expert trajectories may use arbitrary libraries (pandas, numpy, scipy, sklearn, statsmodels, etc.), and not all environments will have all libraries pre-installed. The mechanism prevents trajectory failures from ModuleNotFoundError without requiring manual environment configuration.

Reward Design

The RL component requires a scalar reward signal for each trajectory. The paper designs a reward function with three components, combining environment-verifiable signals with model-based judgment.

Format reward ($r_{\text{format}}$). This binary reward checks whether the trajectory adheres to the required structural format: reasoning enclosed in thinking... and response tags, code enclosed in <code> and </code> tags, and final answer wrapped in <answer>...</answer>. The reward is 1 if all format requirements are met, and 0 otherwise. This is a purely syntactic check that can be verified by regex matching without any model involvement.

Answer reward ($r_{\text{answer}}$). This binary reward assesses whether the final answer is correct. Because "many answers are descriptive and thus resist rule-based verification," the paper uses GPT-4o-mini as a judge model (the same model architecture used for consistency checking during trajectory synthesis, but with a different prompt). The judge compares the predicted answer against the ground-truth answer using criteria similar to the consistency judge: numerical answers within 3% are considered correct, semantic equivalence is accepted for descriptive answers, and completeness is required. The reward is 1 if the judge deems the answer correct, and 0 otherwise.

Length reward ($r_{\text{length}}$). This is a continuous penalty designed to prevent the agent from hacking the answer reward by producing excessively verbose outputs that happen to contain the correct answer string. The length reward is:

rlength={1,l≤lminlmax−llmax−lmin⋅0.5+0.5,lmin<l≤lmax0.5,lmax<lr_{\text{length}} = \begin{cases} 1, & l \leq l_{\text{min}} \\ \frac{l_{\text{max}} - l}{l_{\text{max}} - l_{\text{min}}} \cdot 0.5 + 0.5, & l_{\text{min}} < l \leq l_{\text{max}} \\ 0.5, & l_{\text{max}} < l \end{cases}

where $l$ is the token length of the final answer, $l_{\text{min}} = 256$, and $l_{\text{max}} = 1024$.

What this computes: a piecewise linear function of answer length. For answers of 256 tokens or fewer, the length reward is 1.0 (no penalty). Between 256 and 1,024 tokens, the reward decays linearly from 1.0 down to 0.5—longer answers receive progressively lower rewards, but never below 0.5. Above 1,024 tokens, the reward is a fixed 0.5.

Why this form: the length reward interacts with the answer reward through multiplication (see below), so a reward of 1.0 preserves the full answer-reward signal, while a reward of 0.5 halves it. The design ensures that:

  1. Short, correct answers receive the maximum possible reward.
  2. Long but correct answers receive a reduced but still positive reward (0.5 × 1.0 = 0.5).
  3. The penalty never reduces a correct answer's reward below 0.5, so correctness always dominates length in determining the reward sign.

Alternative that would be wrong: subtracting a length penalty directly from the answer reward (e.g., $R = r_{\text{answer}} - \lambda \cdot l$) could make long correct answers receive negative rewards, which would train the model to avoid answering correctly if doing so requires a long explanation. The multiplicative form ensures that correctness is always rewarded, just more so for concise answers.

Final reward composition:

R={rlength⋅ranswer,ranswer=10,rformat=1,ranswer=0−0.1,rformat=0,ranswer=0R = \begin{cases} r_{\text{length}} \cdot r_{\text{answer}}, & r_{\text{answer}} = 1 \\ 0, & r_{\text{format}} = 1, r_{\text{answer}} = 0 \\ -0.1, & r_{\text{format}} = 0, r_{\text{answer}} = 0 \end{cases}

What this computes: a three-tier reward structure.

  • Tier 1 (correct answer): the reward is the length-penalized answer reward, ranging from 0.5 to 1.0. The format reward is irrelevant because correctness implies the answer was properly formatted.
  • Tier 2 (incorrect answer, correct format): the reward is exactly 0. The model followed instructions but got the wrong answer—it is not penalized (no negative reward), but receives no positive reinforcement.
  • Tier 3 (incorrect format, incorrect answer): the reward is -0.1, a small negative signal. The model failed to follow basic formatting instructions, indicating a potentially degenerate trajectory.

Why differentiate tier 2 from tier 3: a zero reward for "tried but wrong" tells the model that this trajectory is neutral—not worth imitating but not worth actively avoiding. A negative reward for "didn't even try correctly" tells the model that format violations are counterproductive regardless of answer quality. The small magnitude (-0.1 vs. potentially +1.0) ensures that format penalties do not overwhelm correctness signals—a correctly formatted wrong answer scores 0, while an incorrectly formatted wrong answer scores -0.1, a difference small enough that format and correctness remain the primary discriminators.

Why -0.1 specifically: the value is small enough that it does not dominate the gradient—a single correctly formatted correct trajectory (reward 0.5–1.0) outweighs 5–10 format-violating trajectories in terms of total reward. This prevents the model from overfitting to format at the expense of correctness.

The integration of these components—the SFT+RL objective with dynamic γ, void-turn filtering, the stable rollout environment, and the three-tier reward—constitutes the training methodology. The key design principle throughout is stability: every component (SFT loss, clipping, void-turn masking, environment sandboxing, reward shaping) is engineered to prevent the degenerate behaviors (reward collapse, entropy collapse, trajectory collapse, environment crashes) that the paper's exploratory experiments (Section 5.4) showed are the default outcomes of naive training approaches.

4. Key Insights and Innovations

Innovation 1: Self-Consistency Filtering Is More Important Than Selecting the "Best" Trajectory

The field's default assumption in synthetic trajectory generation has been that data quality means correctness: generate candidate trajectories, pick the one that scores highest on some quality metric, and train on that. This assumption manifests in rejection sampling (keep only the best), reward-model ranking (pick the top-scoring response), and judge-model selection (choose the cleanest chain-of-thought). It is so natural that most papers never question it—the whole point of filtering is to remove low-quality data, and "best" selection seems like the logical endpoint.

DATAMIND's analysis in Figure 4 overturns this intuition for multi-turn agent trajectories. The paper compares four trajectory curation strategies: (1) con-select, the full pipeline with self-consistency filtering AND best-trajectory selection by a judge model; (2) non-con, which skips self-consistency entirely and uses all trajectories including inconsistent ones; (3) random-select, which applies self-consistency filtering but randomly picks a consistent trajectory rather than asking the judge to select the best; and (4) non-select, which applies self-consistency but keeps ALL consistent trajectories without selection.

The headline finding is that removing self-consistency filtering (non-con) causes the most severe performance degradation—worse than any variation in selection strategy. But the more surprising finding is that random selection often matches or outperforms judge-based "best" selection, and keeping all consistent trajectories (non-select) yields the largest gains. Random-select achieves higher pass@1 on DABench than con-select, and non-select consistently dominates across all three benchmarks.

What makes this a genuine conceptual contribution rather than a parameter-tuning observation is the diagnosis: trajectory diversity matters more than per-trajectory quality, once a correctness floor is established. Self-consistency filtering establishes that floor—it guarantees that all retained trajectories converge to the same (presumably correct) answer. But within that pool of correct trajectories, the judge model's preference for "concise and accurate" answers inadvertently collapses diversity. The judge selects trajectories that follow similar reasoning patterns, use similar code structures, and phrase answers similarly. Training on a narrower distribution of reasoning paths—even if each individual trajectory is higher quality—produces a less capable model than training on a broader distribution of correct trajectories.

The paper provides supporting evidence through pass@3 scores: random-select's pass@3 is "on par with or superior to those of con-select across all three datasets." This indicates that the randomly-selected trajectories preserve a diversity of problem-solving strategies that the judge inadvertently prunes. When the model is sampled multiple times at inference (pass@3), that diversity translates directly into higher coverage of correct answers.

This finding is a significant reframing of the synthetic data quality problem for agent training. The dominant paradigm—filter for quality, keep only the best—is appropriate when correctness is rare and contamination by incorrect data is the primary risk. But when a strong expert model (DeepSeek-V3.1) can generate correct trajectories with reasonable consistency, the bottleneck shifts from correctness to coverage. A model trained on a single reasoning path to each answer learns brittle pattern-matching; a model trained on multiple correct reasoning paths to the same answer learns transferable analytical skills. This insight parallels findings in the reasoning literature (e.g., OpenThoughts, Guha et al., 2025) but extends them to the multi-turn agentic setting where trajectory diversity encompasses both reasoning paths AND code implementation strategies.

Innovation 2: SFT Loss as a Training Stabilizer That Can Become the Destabilizer

Prior work on combining SFT and RL for language model training has overwhelmingly treated SFT as a warm-start mechanism—something you do first, then switch off. The SFT-then-RL pipeline (used in DeepSeek-R1, Table-R1, and most RL-for-reasoning work) assumes that SFT provides initial capabilities and RL takes over for refinement. The paper's results challenge this assumption at a fundamental level by showing that SFT and RL interact dynamically throughout training, and that the schedule of their interaction determines whether training succeeds or collapses.

The diagnostic evidence is Figure 6 and Figure 7, which together map out the failure modes of different SFT-RL allocation strategies. Pure RL (γ = 0, no SFT loss) exhibits monotonic reward decline—the 7B model cannot generate good enough rollouts for RL to provide a meaningful learning signal. A small fixed SFT weight (γ = 0.2) initially improves rewards but eventually collapses as the weak SFT signal fails to prevent policy drift. A large fixed SFT weight (γ = 0.8) shows a different failure mode: rewards rise briefly then decline as the policy's entropy collapses (tracked in Figure 7), indicating that the model has overfit to the expert trajectory patterns and lost the ability to explore. Only the dynamic schedule—strong SFT early, weak SFT late—maintains stable reward improvement AND preserves policy entropy.

The conceptual contribution is identifying SFT loss as playing two opposing roles in RL training, with the balance between them shifting over time. Early in training, SFT acts as a stabilizer: it provides a strong gradient signal that pulls the policy toward a region of parameter space where trajectories are well-formed, code executes successfully, and answers are at least plausible. Without this signal (pure RL), the model never escapes the regime where most rollouts are garbage and RL provides no useful gradient. Late in training, SFT acts as a constraint: it anchors the policy to the expert distribution, preventing it from discovering strategies that differ from—and potentially improve upon—the expert. The entropy collapse under fixed-high-γ is direct evidence of this constraining effect becoming pathological.

This dual-role diagnosis is more nuanced than the standard "SFT provides a good initialization" narrative. It implies that the optimal SFT-RL allocation is not a single decision (how many SFT epochs before RL?) but a temporal schedule that changes as the model's capabilities improve. The cosine decay from γ = 0.9 to γ = 0.05 implements this insight algorithmically, but the conceptual point extends beyond this specific parameterization: any training procedure that combines imitation learning and reinforcement learning for agents must account for the fact that the value of imitation changes over the course of training.

The "raising a child" analogy (Section 5.4) is more than rhetorical flourish—it encodes a genuine theoretical claim about why the balance must shift. Early in training, the model lacks the competence to benefit from autonomous exploration (the child has no basis for evaluating which actions lead to good outcomes). The SFT signal substitutes for this missing competence by directly specifying correct behavior. As competence develops, the SFT signal becomes redundant for well-learned patterns and restrictive for novel situations. The optimal policy gradually withdraws SFT guidance, allowing the model to develop its own strategies while retaining the foundational skills that SFT provided.

Innovation 3: Reinforcement Learning Can Narrow but Not Reverse the Performance Gap Between Base Models

This finding, presented in Figure 8, addresses a question with substantial practical and theoretical implications: if you apply the same RL training pipeline to different base models, does the initial capability ordering persist, or can RL "catch up" a weaker model? The paper's results show a clear pattern: as the amount of cold-start SFT training increases, the marginal gain from subsequent RL diminishes, and the post-RL performance ordering matches the pre-RL ordering. The 14B model consistently outperforms the 7B model at equivalent training stages, and RL never reverses this gap.

The conceptual significance is in constraining what RL can and cannot do for agent training. Over the past two years, the success of DeepSeek-R1 and other RL-trained reasoning models has created an implicit narrative that RL is the primary driver of capability, with SFT merely providing a base to build on. This narrative implies that with enough RL training, a smaller model might match a larger one—the "RL as great equalizer" hypothesis. The paper's results directly contradict this: "the bulk of knowledge is acquired during SFT, whereas RL primarily serves to unlock latent potential rather than explicitly push the model beyond its inherent capacity boundary" (Section 5.4).

This finding parallels results from the reasoning literature (Yue et al., 2025a; Chu et al., 2025) but is established here for a substantially different domain—multi-turn code-executing agents rather than single-turn math/code reasoning. The consistency of the finding across domains strengthens its generality: RL is a capability amplifier, not a capability creator. It can help a model make better use of what it already knows, but it cannot teach fundamentally new knowledge that wasn't present in the SFT data or base pretraining.

The practical implication is clear: if you want a better agent, invest in better SFT data (higher-quality trajectories, more diverse queries, broader coverage of task types) rather than hoping that more RL training will close the gap with a larger model. This doesn't diminish RL's value—Figure 8 shows that RL provides meaningful gains over cold-start alone at every training stage—but it recalibrates expectations about what those gains represent and where they come from.

Innovation 4: A Fully Automated Pipeline That Synthesizes Multi-Turn Code-Execution Trajectories Without Human Annotation

This is primarily a systems engineering contribution but one with conceptual implications for how the field thinks about training data for interactive agents. Prior to DATAMIND, training an open-source data-analytic agent required one of two resources that are fundamentally unscalable: (1) human-annotated trajectories (expensive, slow, domain-specific) or (2) trajectories scraped from existing benchmarks (limited in quantity, narrow in format coverage, and often lacking executable code). The paper's pipeline breaks this dependency by demonstrating that a stronger proprietary model (DeepSeek-V3.1), guided by human-written procedural knowledge per task category and filtered through self-consistency, can generate training trajectories that are good enough to train a smaller model that surpasses the teacher on downstream benchmarks.

The conceptual contribution is the demonstration that teacher-student distillation for multi-turn agentic tasks works even when the "teacher" trajectories contain errors, provided the filtering mechanism distinguishes consistently-correct from inconsistently-correct trajectories. This is not obvious a priori. In standard knowledge distillation for language models, the teacher's outputs are treated as ground truth—even incorrect teacher outputs are used as training targets. In DATAMIND, the self-consistency filter discards teacher trajectories that fail to converge, and the reflection loop uses judge feedback to improve trajectories that are inconsistent. The result is a training set where every trajectory is self-consistent—the expert model's independent samples agree—which provides a stronger correctness signal than any single trajectory could.

The pipeline is also notable for what it does NOT require: no human-labeled answers, no manually verified code, no execution-traced ground truth. The only human input is the procedural knowledge k for each task category (a few sentences of high-level workflow guidance) and the 4–6 exemplar queries per category. Everything else—file collection, query generation, trajectory sampling, consistency checking, reflection, filtering—is fully automated. At a time when agent training is bottlenecked by data scarcity, this automation recipe is arguably more valuable than any single trained model, because it can be adapted to new domains and new data formats without requiring new human annotation effort.

The significance of this is amplified by the paper's data scaling results (Figure 3): model performance improves monotonically with training set size from 2K to 12K trajectories, and the paper only stopped at 12K due to computational constraints rather than saturation. The implication is that the pipeline could generate substantially more data—limited only by file collection breadth and expert-model inference cost—and continue to improve the trained agent. This changes the framing from "how do we get enough data to train an agent?" to "how do we scale data synthesis to produce the best possible agent?"

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on three benchmarks: DABench (Hu et al., 2024) with 257 challenges across 52 CSV files spanning 7 question categories; TableBench (Wu et al., 2025b) with real-world table reasoning questions across 18 fields and four major categories, where .json tables were converted to .csv and trend forecasting/chart generation questions were filtered out (no explicit gold answers); and BIRD (Li et al., 2023b), a widely-used Text-to-SQL benchmark where the validation set is used as the testbed (since the official test set requires leaderboard submission). The BIRD evaluation uses exact-match comparison of materialized CSV results against gold labels; DABench and TableBench use model-as-judge (GPT-4o-mini) to compare predicted answers against ground truth.

  • Base model(s). The primary backbone is Qwen-2.5-Coder at two scales: 7B and 14B parameters (Hui et al., 2024). The paper argues this family exhibits "stronger reasoning capacity and higher plasticity during RL training" (Appendix G.1). To verify cross-family generalization, Llama-3.1-8B-Instruct is also trained on DATAMIND-12K (Appendix G.1, Table 4). All open-source models use the Instruct version.

  • Metrics. The primary metric is accuracy (fraction of questions answered correctly), reported as both pass@1 (average of three independent trials) and pass@3 (success on any of the three trials counts as correct). For DABench and TableBench, correctness is determined by GPT-4o-mini comparing the predicted answer against the gold label. For BIRD, SQL execution results are materialized to CSV and compared via exact match. The paper additionally validates the judge model's reliability by re-scoring with Qwen-2.5-72B as an external judge (Figure 5), finding a Pearson correlation of 0.96, and by comparing with Rouge-L on descriptive TableBench tasks (Appendix G.4, Table 7), where rankings align but Rouge-L fails to capture true correctness.

  • Baselines. The paper compares against five proprietary models: GPT-4o (gpt-4o-2024-0806), o4-mini (o4-mini-2025-04-16), DeepSeek-R1 (deepseek-r1-2025-0528), DeepSeek-V3.1 (deepseek-v3.1-nothinking), and GPT-5 (gpt-5-2025-08-07). Four open-source untrained models are included: QwQ-32B, Qwen-2.5-Coder-32B, Llama-3.3-70B, and Qwen-2.5-72B, all evaluated via zero-shot ReAct prompting. Four explicitly trained open-source baselines are compared: TableLLM (Wu et al., 2025b), trained via SFT on TableInstruct (~20K instances); Table-R1 (Wu et al., 2025c), using region-enhanced SFT followed by table-aware GRPO; OmniSQL (Li et al., 2025a), trained on SynSQL-2.5M (2.5M instances); and SQL-R1 (Ma et al., 2025), further RL-trained on a 5K subset of SynSQL-2.5M. All trained baselines are reproduced using their official code and data for Qwen-2.5-Coder-7B and 14B backbones where official models were unavailable. The vanilla ReAct baseline (zero-shot prompting of the base model with the ReAct format) serves as the untrained lower bound.

  • Generation budget / compute accounting. For training, each experiment runs on 8 × 80G A100 GPUs within 2 days (Section 5.1). The RL batch size is 32 with mini-batch size 4, rollout group size G = 4, and maximum interaction rounds T = 10. At inference, all models use temperature 0.7, top-p 0.95, and batch size 5. The paper does not report inference FLOPs or wall-clock time for the baselines—comparisons are based purely on accuracy metrics, not compute-matched inference budgets. For the proprietary models, only zero-shot ReAct prompting is used (no best-of-N, no majority voting), which the paper implicitly acknowledges as a conservative baseline setup.

  • Cross-validation / statistical protocol. Three independent trials are run for each model on each benchmark. The average across trials is pass@1; the union (success on any trial) is pass@3. No confidence intervals, standard deviations, or statistical significance tests are reported. The paper does not mention cross-validation for strategy selection (unlike the reference example's compute-optimal policy selection). The data contamination analysis (Appendix H, Table 8) computes header-overlap ratios between DATAMIND-12K files and benchmark files, finding 0.00% overlap for TableBench and 0.02% for BIRD (both effectively zero), and 1.29% for DABench—a negligible figure the authors deem acceptable.

Main Quantitative Results

Headline Benchmark Performance (Table 1)

The central quantitative claim is that DATAMIND-14B achieves a state-of-the-art average score of 71.16% across DABench, TableBench, and BIRD (averaging pass@1 across the three benchmarks), outperforming all proprietary and open-source baselines. DATAMIND-7B achieves 68.10%, ranking best among all open-source models.

Breaking down by benchmark:

  • DABench (CSV analysis): DATAMIND-14B achieves 80.29% pass@1 (88.72% pass@3), surpassing DeepSeek-V3.1 at 81.32% pass@1? Wait—this requires careful reading. Table 1 shows DeepSeek-V3.1 at 81.32% pass@1 on DABench versus DATAMIND-14B at 80.29%. So DeepSeek-V3.1 outperforms DATAMIND-14B on DABench specifically. The "outperforms all proprietary models" claim in the abstract is qualified by the average across all three benchmarks, not per-benchmark superiority. DATAMIND-7B achieves 77.30% pass@1, compared to the vanilla ReAct 7B baseline at 15.05%—a 5.1× improvement from training. Compared to the best open-source baseline (Qwen-2.5-72B at 75.33%), DATAMIND-7B gains ~2 percentage points with 10× fewer parameters.

  • TableBench (table QA): DATAMIND-14B achieves 70.95% pass@1 (81.81% pass@3), compared to DeepSeek-V3.1 at 72.52% pass@1. Again, DeepSeek-V3.1 leads on this individual benchmark. DATAMIND-7B achieves 67.60% pass@1 versus the next-best open-source model Qwen-2.5-72B at 65.44%.

  • BIRD (Text-to-SQL): DATAMIND-14B achieves 62.23% pass@1 (70.21% pass@3), now surpassing DeepSeek-V3.1 at 57.89% and GPT-5 at 60.17%. This is the benchmark where DATAMIND shows the clearest advantage over proprietary models. DATAMIND-7B achieves 59.41% pass@1, competitive with Qwen-2.5-72B at 60.30% and substantially ahead of the next-best trained baseline OmniSQL-7B at 57.11%.

The average across the three benchmarks produces the 71.16% figure for DATAMIND-14B versus DeepSeek-V3.1 at 70.58% and GPT-5 at 69.44%. The margin is modest (+0.58% over DeepSeek-V3.1) but the claim of superiority over proprietary models is numerically accurate for the average, even though DATAMIND loses on two of three individual benchmarks.

Performance of Specialized vs. Generalist Trained Models (Table 1)

A striking pattern in Table 1 is the catastrophic specialization of models trained narrowly. OmniSQL-7B achieves 57.11% on BIRD (strong SQL performance) but drops to 26.46% on DABench and 39.95% on TableBench—despite the paper converting all tables in those benchmarks to .sqlite format for fair evaluation. This 30+ percentage point gap between in-domain and out-of-domain performance demonstrates that OmniSQL learned format-specific heuristics rather than general data analysis reasoning. TableLLM-7B shows even starker failure: 36.71% on DABench and 11.99% on BIRD—the model trained for small-table QA cannot handle either large-scale CSV analysis or multi-table SQL. Table-R1-7B performs similarly poorly on BIRD (10.69%).

The DATAMIND models exhibit balanced performance: 77.30% (DABench), 67.60% (TableBench), 59.41% (BIRD) for the 7B version—a spread of ~18 percentage points across benchmarks, compared to OmniSQL's ~30-point spread and TableLLM's ~25-point spread. This balanced profile is direct evidence that DATAMIND-12K's multi-format, multi-category training data produces a generalist rather than a specialist.

Data Scaling Behavior (Figure 3)

The paper trains the 7B model on random subsets of 2K, 4K, 8K, and 12K trajectories from DATAMIND-12K and reports average performance across all benchmarks. The pass@1 curve shows monotonic improvement from ~55% at 2K to ~68% at 12K, with no visible plateau—suggesting further data scaling would yield additional gains. The pass@3 curve similarly rises from ~70% to ~79%.

An intriguing secondary observation: the gap between pass@1 and pass@3 narrows as data volume increases (from roughly 15 percentage points at 2K to roughly 11 points at 12K). The paper interprets this through the lens of Yue et al. (2025a): "RL training tends to increase the likelihood of generating correct samples, yet contributes little to expanding the model's coverage of solvable problems." In other words, more data makes the model more reliable on problems it can already solve (narrowing pass@1–pass@3 gap) but the set of solvable problems expands more slowly.

Training Strategy Comparison (Table 2)

The paper ablates four training strategies on the 7B model:

  • Pure SFT: 62.54% pass@1, 73.74% pass@3. This is the baseline—SFT alone on DATAMIND-12K raises performance from 11.26% (vanilla ReAct) to 62.54%, capturing the majority of the total gain.
  • Zero-RL (RL without SFT cold start): 58.03% pass@1, 71.72% pass@3. RL alone underperforms SFT alone, confirming that the 7B model cannot bootstrap effective exploration from random initialization—the initial rollouts are too poor to provide meaningful RL gradients.
  • SFT-then-RL (conventional pipeline): 63.42% pass@1, 75.46% pass@3. This adds only +0.88 percentage points over pure SFT on pass@1—a marginal gain that the paper attributes to training instability ("we need to run many trials and select a relatively good checkpoint").
  • SFT-and-RL (dynamic γ, the DATAMIND approach): 68.10% pass@1, 79.07% pass@3. This represents a +5.56 percentage point gain over pure SFT and +4.68 points over SFT-then-RL. The hybrid objective is the only configuration that realizes substantial gains from RL.

The pass@3 gains are particularly informative: SFT-and-RL achieves 79.07% versus SFT-then-RL's 75.46% (+3.61 points), indicating that the dynamic objective not only improves average-case performance but also preserves (or enhances) generation diversity.

Cross-Family Generalization (Appendix G.1, Table 4)

Training Llama-3.1-8B-Instruct on DATAMIND-12K yields 65.56% average pass@1 and 77.73% pass@3. This places the Llama-8B agent above Llama-3.3-70B (61.45% pass@1) and competitive with Qwen-2.5-72B (67.02%). The pass@3 score of 77.73% approaches DeepSeek-V3.1 (79.76%) and DATAMIND-7B (79.07%). This result demonstrates that the DATAMIND pipeline transfers across model families—the gains are not specific to Qwen architectures. However, the Llama-8B underperforms DATAMIND-7B by ~2.5 percentage points on pass@1, suggesting that base model quality (Qwen-2.5-Coder vs. Llama-3.1) matters even with identical training data and procedure.

Difficulty-Generalization: QRData Results (Appendix G.3, Table 6)

On QRData (Liu et al., 2024), a benchmark featuring "intricate causal-reasoning tasks" that rely on "domain commonsense knowledge," DATAMIND-14B achieves 62.04% pass@1 (77.62% pass@3), outperforming DeepSeek-V3.1 at 60.75% and Qwen-2.5-72B at 60.50%. DATAMIND-7B achieves 57.66% pass@1—competitive but not dominant. The paper notes that the 7B model "lacks the domain-specific knowledge required" for QRData's causal reasoning, acknowledging a parameter-count limitation that the training pipeline cannot fully overcome.

Evaluation Robustness: Alternative Judges and Metrics (Figure 5, Appendix G.4)

When re-scoring DABench and TableBench results with Qwen-2.5-72B instead of GPT-4o-mini, the ranking of models remains "virtually identical" (Figure 5). The Pearson correlation between the two judges' scores is 0.96, indicating near-perfect linear agreement. On TableBench's descriptive-generation tasks (Table 7), Rouge-L scores produce the same relative ordering as the model-as-judge: DATAMIND-7B achieves 23.41 Rouge-L versus DeepSeek-V3.1 at 19.64 and Qwen-2.5-72B at 17.93. However, the paper correctly notes that Rouge-L "over-emphasizes surface lexical and sentence overlap with the gold label rather than answer correctness," making the absolute scores uninformative (even the top model scores only 23.41 on a 0–100 scale).

Ablation Studies and Robustness Checks

Self-consistency filtering vs. best trajectory selection (Figure 4): The paper compares four trajectory curation strategies via SFT on the 7B model. Removing self-consistency filtering entirely (non-con) causes the largest performance degradation across all three benchmarks. Within the set of self-consistent trajectories, random-select (randomly picking one consistent trajectory) achieves 61.92% pass@1 on DABench versus con-select (judge-based best selection) at 59.65%—random selection actually outperforms the judge's preference on DABench. The most effective strategy is non-select (keeping all consistent trajectories without any selection), which achieves the highest scores on all benchmarks (e.g., 64.84% on BIRD vs. 63.67% for con-select). This non-obvious finding establishes that trajectory diversity within the consistent set matters more than per-trajectory quality, once a correctness floor is guaranteed by self-consistency.

SFT loss weight (γ) during RL training (Figures 6 and 7): The paper tracks answer reward dynamics across training steps for different γ settings without cold start. With γ = 0 (pure RL, no SFT loss), the answer reward "declines almost monotonically"—the 7B model's initial rollouts are too poor for RL to provide a useful learning signal. With a small fixed γ = 0.2, the reward initially rises but eventually collapses as the weak SFT signal fails to prevent policy drift. With a large fixed γ = 0.8 (Figure 7), the reward rises briefly then declines, accompanied by a pronounced entropy collapse—the policy overfits to expert trajectory patterns and loses exploration capability. The dynamic γ schedule (cosine annealed from 0.9 to 0.05) maintains both stable reward improvement and high policy entropy throughout training, avoiding all three failure modes.

Cold start duration for RL training (Figure 8): Training with varying numbers of cold-start SFT epochs before the joint SFT+RL phase reveals that more cold start reduces the marginal gain from RL. At zero cold-start epochs, RL provides the largest relative improvement over the baseline. By three cold-start epochs, the slope of improvement from RL is substantially diminished. This pattern holds for both 7B and 14B models, with the 14B model consistently outperforming the 7B at equivalent training stages—RL narrows the performance gap between models but does not reverse the ordering. The paper interprets this as evidence that "the bulk of knowledge is acquired during SFT, whereas RL primarily serves to unlock latent potential" (Section 5.4).

Data contamination analysis (Appendix H, Table 8): Header-overlap ratios between DATAMIND-12K files and benchmark files are 1.29% for DABench, 0.00% for TableBench, and 0.02% for BIRD—all effectively zero or negligible. The paper acknowledges the DABench files were "harvested from GitHub" while TableBench tables were "extracted from Wikipedia," with neither overlapping with the Kaggle and Text-to-SQL corpus sources used for DATAMIND.

Data Interpreter scaffold comparison (Appendix G.2, Table 5): The paper attempts to reproduce Data Interpreter, an existing prompt-engineered scaffold, but cannot replicate its reported 94.93% GPT-4o score on DABench. When applied to DeepSeek-V3.1, Data Interpreter achieves 67.70% on DABench (vs. 81.32% for vanilla ReAct—the scaffold degrades the stronger model). On Qwen-2.5-Coder-7B, Data Interpreter achieves 54.09% (vs. 15.05% for ReAct—a large improvement, but far below DATAMIND-7B's 77.30%). This negative result demonstrates that prompt engineering scaffolds are model-specific and brittle: "prompts engineering for one model may fail to generalize to others" (Appendix G.2).

Alternative evaluation metrics (Appendix G.4, Table 7): Rouge-L scores on TableBench descriptive tasks produce correct relative rankings (DATAMIND > DeepSeek-V3.1 > Qwen-72B) but fail to capture absolute model quality—even DeepSeek-V3.1 scores only 19.64 Rouge-L. This ablation validates the choice of model-as-judge over surface-form metrics while acknowledging the judge's limitations for descriptive answers.

Critical Assessment

Claim: DATAMIND-14B outperforms the strongest proprietary baselines

This claim holds for the average across three benchmarks (71.16% vs. DeepSeek-V3.1 at 70.58%, GPT-5 at 69.44%), but the per-benchmark picture is more nuanced. DeepSeek-V3.1 outperforms DATAMIND-14B on DABench (81.32% vs. 80.29%) and TableBench (72.52% vs. 70.95%). DATAMIND-14B's average advantage comes entirely from BIRD (62.23% vs. 57.89%), where the margin is substantial (+4.34 points). The claim "outperforms all proprietary models" is technically true for the average but obscures that the model loses on two of three individual benchmarks. A more precise characterization would be: DATAMIND-14B is competitive with the strongest proprietary models, with particular strength in SQL-based database analysis.

The proprietary baselines are also evaluated under zero-shot ReAct prompting only—no majority voting, no best-of-N, no compute-optimal scaling, no search. Given that these models (especially GPT-5 and DeepSeek-V3.1) likely have internal reasoning capabilities that could be amplified by simple test-time strategies, the comparison likely understates what these models could achieve with minimal engineering. The paper's own Data Interpreter experiment (Table 5) shows that scaffolds can sometimes harm performance, so the zero-shot baseline is not obviously unfair, but a best-of-3 or majority-vote baseline would have been informative.

Claim: DATAMIND-7B performs best among all open-source models

This claim is strongly supported. At 68.10% average pass@1, DATAMIND-7B outperforms the next-best open-source model (Qwen-2.5-72B at 67.02%) while using ~10× fewer parameters. It surpasses all trained baselines, including models trained on substantially larger corpora (OmniSQL's 2.5M instances vs. DATAMIND's 12K). The per-benchmark results are consistent: 77.30% on DABench (vs. 75.33% for Qwen-72B), 67.60% on TableBench (vs. 65.44%), and 59.41% on BIRD (vs. 60.30% for Qwen-72B—the one benchmark where it trails, by less than 1 point).

Claim: Self-consistency filtering is more important than best trajectory selection

Supported by Figure 4. Removing self-consistency produces the largest performance drops across all benchmarks. Random selection within consistent trajectories matches or outperforms judge-based best selection. The caveat is that these experiments use only the 7B model with SFT-only training—the finding may not generalize to the full SFT+RL pipeline or the 14B model. The non-select strategy (keeping all consistent trajectories) achieves the highest scores, which confounds trajectory quality with training data volume (non-select provides more training examples than con-select). The paper acknowledges this: "we cannot fully rule out the contribution of the larger training volume introduced by this unfiltered approach" (Section 5.4). A proper ablation would control for data volume by downsampling non-select to match con-select's size.

Claim: SFT loss stabilizes RL training but can also destabilize it

Well-supported by the training dynamics in Figures 6 and 7, which show three distinct failure modes for non-dynamic γ schedules. The evidence is qualitative (reward curves, entropy curves) rather than quantitative (no statistical comparisons between schedules), but the patterns are clear and interpretable. A limitation: these dynamics experiments were conducted "without a cold start" (Section 5.4), while the final DATAMIND training includes cold start—the stabilization benefit of the dynamic schedule might differ when cold start is present. The final model uses both cold start AND dynamic γ, making it difficult to attribute gains to either component independently.

Claim: RL narrows but cannot reverse the performance gap between base models

Supported by Figure 8, which shows the 14B model consistently outperforming the 7B at all training stages, with the gap narrowing (the dashed RL lines have shallower slope than the solid cold-start lines) but never closing. The experiment uses only 3,843 training samples (balanced on query types) and 240 test samples—a fraction of the full DATAMIND-12K and benchmark sizes. Whether this finding holds at full scale is untested. The paper also does not test whether a 7B model trained on more data could close the gap with a 14B model trained on less data—a data-compute tradeoff that would be informative.

Genuine weaknesses in the experimental design

  1. No confidence intervals or statistical testing. All results are reported as point estimates from three trials, with no standard deviations, confidence intervals, or significance tests. For a paper claiming state-of-the-art performance, the absence of uncertainty quantification makes it impossible to assess whether margins like 71.16% vs. 70.58% (a 0.58-point difference) are statistically meaningful.

  2. Single model family for primary results. Most experiments use Qwen-2.5-Coder backbones. The Llama-3.1-8B result (Appendix G.1) is a single data point that partially addresses generalizability, but the 14B scale and full pipeline are only validated on Qwen. It is unknown whether the dynamic γ schedule, void-turn filtering, and reward design would transfer to, say, a DeepSeek or Mistral backbone.

  3. Test set sizes are moderate. DABench has 257 questions, TableBench's size after filtering is not reported (the original has multiple categories, some removed), and BIRD's validation set is the testbed. The paper does not report exact per-benchmark test sizes, but 257 + (unknown) + (BIRD dev) suggests a total of perhaps 500–800 questions. This is adequate but not large, and the averaged results are sensitive to per-benchmark weighting (the paper appears to weight benchmarks equally in the average, despite potentially different test set sizes).

  4. The data contamination analysis (Appendix H) addresses only header overlap. Headers being disjoint does not guarantee that questions and answers are not memorized from pretraining. A proper decontamination analysis would check for n-gram overlap between DATAMIND-12K questions/answers and benchmark questions/answers, and ideally would evaluate on a version of the benchmarks with perturbed numbers or entities.

  5. No ablation of the reward function components. The paper describes a carefully designed reward with format, answer, and length components, but never ablates them (e.g., removing length penalty, changing the -0.1 format penalty, testing alternative length-penalty functions). The reward design is presented as a fixed recipe with no empirical validation of its components.

  6. The reflection loop's contribution is not isolated. The trajectory synthesis pipeline includes a reflection step for inconsistent trajectories, but the paper never reports what fraction of trajectories required reflection, what fraction were rescued by reflection, or what training performance looks like without reflection. This component could be essential or irrelevant—the experiments do not distinguish.

  7. Missing baseline: model trained on uncurated data at larger scale. The paper emphasizes that DATAMIND-12K's quality enables outperforming models trained on 2.5M instances (OmniSQL). But a direct comparison at matched data volume—e.g., 12K OmniSQL-style trajectories vs. 12K DATAMIND trajectories, or 2.5M naively generated trajectories vs. DATAMIND—would isolate the contribution of the curation pipeline from the contribution of data volume. The current comparison confounds quality and quantity.

  8. Cold start + dynamic γ are confounded in the final model. The best model uses both cold start (SFT-only pre-training) and the dynamic SFT+RL objective. Figure 8 shows cold start matters; Figures 6–7 show dynamic γ matters. But there is no experiment showing the combination is better than either alone, or whether the dynamic schedule's benefit persists when cold start is present.

  9. No evaluation of agentic behaviors beyond answer correctness. The paper evaluates only final answer accuracy. Important agentic capabilities—error recovery rate, efficiency (number of turns to solution), code quality, ability to recognize when an answer is unattainable—are not measured. A model could achieve high accuracy by brute-force code generation with many retries while being significantly less efficient than baselines.

Experiments that would have strengthened the paper

  • A compute-matched inference budget comparison with proprietary models, giving them best-of-N or majority voting to match the training investment DATAMIND received.
  • Per-category analysis of the 18 task types to identify which types benefit most from training and which remain difficult.
  • Error analysis showing common failure modes (execution errors, logical errors, incomplete answers) and how they change across training stages.
  • Scaling beyond 12K trajectories to identify whether the data scaling trend (Figure 3) continues or plateaus.
  • An ablation of the void-turn filtering mechanism—what fraction of trajectories are filtered, and what training performance looks like without this component.
  • A direct comparison of trajectory synthesis with vs. without procedural knowledge $k$ to quantify the contribution of the human-written workflows.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Not Accounted for in the Synthesis Pipeline

The assumption or constraint. The trajectory synthesis pipeline assumes that generating three independent expert trajectories per query and running them through a GPT-4o-mini judge model (with a reflection loop for inconsistent cases) is a one-time cost that does not require amortization analysis. The paper never quantifies the total inference cost of producing DATAMIND-12K—how many DeepSeek-V3.1 API calls were made, how many GPT-4o-mini judge calls were consumed, or what fraction of initial trajectories required the expensive reflection loop. The only cost metric provided is training cost (8 × 80G A100 GPUs within 2 days, Section 5.1), which excludes synthesis entirely.

The consequence. A practitioner wanting to adapt DATAMIND to a new domain (e.g., healthcare data, financial analysis, scientific datasets) must replicate the full synthesis pipeline, not just the training recipe. This involves: crawling and filtering domain-specific data files (with unknown yield rates), generating queries from the 18-category taxonomy (potentially requiring new exemplars for domain-specific task types), synthesizing trajectories using a proprietary model at least as capable as DeepSeek-V3.1 (the paper found that DeepSeek-V3 was sufficient for query generation but a stronger model was needed for trajectory generation), running self-consistency checks with a judge model, and applying the reflection loop. The total API inference cost for 11,707 final trajectories—when each trajectory requires up to 3 initial samples plus potentially 3 reflected samples, plus 1–2 judge calls per group—could easily reach millions of tokens of proprietary model inference. For a small research lab without access to DeepSeek-V3.1-level models, this cost may be prohibitive. The paper's claim of advancing open-source agents is thus partially contingent on access to proprietary models for data synthesis.

What evidence exists in the paper. The paper provides no cost analysis for synthesis. We know that N = 3 samples per query, a reflection loop exists (Equation 3), and the judge model is GPT-4o-mini, but we do not know: the total number of queries generated (only that 11,707 trajectories survived filtering), the consistency rate (what fraction of queries passed the first consistency check vs. required reflection), the reflection success rate (what fraction of reflected trajectories subsequently passed consistency), or the total number of DeepSeek-V3.1 and GPT-4o-mini API calls. Section 3.2 notes that the pipeline "inherently biases us toward easier queries whose answers are more likely to coincide," implying that harder queries are disproportionately filtered out or require reflection—but no quantitative breakdown is provided.

Mitigation status. The paper does not address this limitation and does not propose cheaper alternatives for trajectory synthesis (e.g., using a smaller model for initial sampling, reducing N below 3, or eliminating the reflection loop). The procedural knowledge k is described as "manually crafted" (Section 3.2), adding human labor cost that is also unquantified. The paper frames data synthesis as a one-time contribution—"we have released DATAMIND-12K"—but any domain adaptation requires re-incurring these costs.

6.2 Hard Problems and Non-Tabular Tasks Remain Out of Scope

The assumption or constraint. The paper constrains its problem definition to tabular data files (.csv, .xlsx, .sqlite) and textually-expressed queries, explicitly excluding several categories of data analysis work:

"At present, we only incorporate reasoning-oriented data-analysis tasks; training, predictive, and data-visualization tasks are deliberately excluded and reserved as our important future work." (Appendix B)

"The current version of DATAMIND only accepts tabular data files and textual questions. In the future, we will extend DATAMIND to additional modalities." (Appendix B)

Additionally, the maximum file size is capped at 1,000 rows during collection (Section 3.1), and the maximum answer length is capped at 1,024 tokens during filtering (Section 3.2).

The consequence. The trained agent cannot perform several classes of tasks that are central to real-world data analysis: (1) predictive modeling (training machine learning models, evaluating their performance, making forecasts), (2) data visualization (generating plots, charts, dashboards), and (3) large-scale data processing (files with >1,000 rows are excluded from training, so the model has never encountered the memory management and chunking strategies needed for big data). The 18-category task taxonomy (Figure 1a) contains no categories for model training, hyperparameter tuning, feature selection for ML, or visualization design. A data analyst who asks "build a random forest classifier to predict customer churn and show me the feature importance plot" will receive no useful response, because the model was never trained on trajectories involving scikit-learn model fitting or matplotlib plotting.

The file size constraint is particularly limiting. The 1,000-row cap at collection time means DATAMIND-12K contains only small-to-medium datasets. Real-world enterprise data analysis routinely involves files with hundreds of thousands or millions of rows. The model's strategies for data inspection (using df.head(3) and df.columns, as prescribed in the prompt) may be appropriate for 100-row files but are inadequate for understanding the distribution of a million-row dataset where sampling, aggregation, and memory-efficient operations are essential.

What evidence exists in the paper. The explicit exclusions are stated in Appendix B. The file size filter (<20 rows and >1,000 rows discarded) is documented in Section 3.1. The 18-category taxonomy (Figure 1a) visibly lacks predictive modeling, visualization, and big-data processing categories. The QRData results (Appendix G.3, Table 6) show that the 7B model underperforms on tasks requiring "domain commonsense knowledge" (57.66% vs. DeepSeek-V3.1 at 60.75%), hinting that knowledge-intensive analysis is a weakness even within the supported task types. The BIRD results (62.23% pass@1 for the 14B model) leave substantial room for improvement, and BIRD represents multi-table SQL analysis—a core supported task—so harder unsupported tasks would presumably fare worse.

Mitigation status. The paper explicitly acknowledges these exclusions as limitations and identifies them as "important future work" (Appendix B). No mitigation is attempted within the current system. The design choices that produced these limitations—excluding large files for trajectory synthesis tractability, excluding visualization because model-as-judge evaluation cannot assess plot quality, excluding ML training because reward design for model performance is non-trivial—are practical and defensible for a first-generation system, but they bound the agent's applicability.

6.3 The Reward Function Relies on a Proprietary Judge Model with Unverified Reliability

The assumption or constraint. The RL training reward and the primary evaluation metric both depend on GPT-4o-mini acting as a judge to assess answer correctness. The reward function (Section 4, Equation 8) assigns r_answer = 1 when the judge deems the answer correct and 0 otherwise. The evaluation protocol (Section 5.1, Appendix D) uses the same judge model to compare predicted answers against ground truth for DABench and TableBench. The paper validates this choice by showing high correlation (Pearson's r = 0.96) with an alternative judge (Qwen-2.5-72B) and with Rouge-L rankings (Appendix G.4), but these validations have specific limitations.

The consequence. A model trained to maximize a specific judge's approval may learn to exploit that judge's particular biases—producing answers that the judge accepts as correct but a human analyst would reject. This is the well-documented "reward hacking" problem in RLHF, here applied to a domain (descriptive data analysis answers) where correctness is genuinely ambiguous. For example, the judge's prompt (Appendix I.4) specifies that "for numerical questions, any result within 3% of the ground truth answer is considered correct" and that "semantic equivalence" is accepted for descriptive answers. A model that learns to produce answers that are approximately correct (within 3% tolerance) but miss important nuance, or that use verbose language to create the appearance of completeness, would score well under the judge without actually solving the analysis problem rigorously.

The validation with Qwen-2.5-72B as an alternative judge (Figure 5) shows high correlation in rankings but does not verify that both judges agree on absolute correctness for individual answers. Two judges can rank models identically while both systematically overrating or underrating certain types of answers. The Rouge-L validation (Table 7) is informative about ranking consistency but, as the paper itself notes, Rouge-L "over-emphasizes surface lexical and sentence overlap with the gold label rather than answer correctness" (Appendix G.4)—so it validates that the judge model's rankings are better than surface-form metrics, not that the judge model is accurate.

What evidence exists in the paper. The judge validation experiments (Figure 5, Table 7) provide partial evidence. The length penalty in the reward function (Equation 8) is designed specifically to prevent one form of judge exploitation—"the agent hacking the answer reward by hallucinating excessive tokens" (Section 4)—which indicates the authors are aware of the risk. However, the paper does not report any experiment that directly measures judge exploitation: no adversarial evaluation where deliberately verbose, semi-correct answers are tested against the judge; no human evaluation of a sample of model answers to verify judge accuracy; no analysis of whether the model's answers improve on metrics the judge does not explicitly reward (e.g., conciseness, actionability, clarity).

Mitigation status. The paper partially mitigates this through external validation (alternative judge, Rouge-L comparison) and through the length penalty in the reward function. However, these mitigations address whether the judge is consistent with other automated metrics, not whether the judge is correct relative to human judgment. The fundamental circularity—the same model architecture (GPT-4o-mini) used for training reward also serves as the evaluation judge—is not addressed. A more robust approach would use human evaluation for at least a subset of answers, or train on a judge model and evaluate with human annotators, but neither is attempted.

6.4 Generalization Beyond Qwen Architectures and the Three Benchmarks Is Unverified

The assumption or constraint. The primary experiments use exclusively Qwen-2.5-Coder backbones (7B and 14B) and evaluate on three benchmarks: DABench, TableBench, and BIRD. The paper claims to produce "generalist data-analytic agents" and provides one cross-family validation point (Llama-3.1-8B, Appendix G.1) and one additional benchmark (QRData, Appendix G.3), but these are limited in scope and reported in the appendix rather than the main results.

The consequence. Three specific generalizability questions remain unresolved:

Model family transfer: The Llama-3.1-8B result (65.56% average pass@1, Table 4) demonstrates that DATAMIND training works on non-Qwen architectures, but the 8B scale is only validated—there is no Llama-14B or Llama-70B result to test whether the gains scale with model size in the same way across families. Different model families have different pretraining data mixtures, tokenizers, and architectural inductive biases. The procedural knowledge k, prompt templates, and reward function were all developed and tuned on Qwen models. The paper provides no evidence about whether DeepSeek, Mistral, or Gemma backbones would benefit similarly.

Benchmark coverage: The three main benchmarks, while diverse in format (.csv, .json-converted-to-.csv, .sqlite), all evaluate tabular data analysis with structured ground-truth answers. None test the agent's ability to handle: (a) truly messy real-world data (missing values, inconsistent formatting, encoding errors) beyond what appears in curated benchmarks, (b) streaming or incrementally arriving data, (c) multi-file analysis where data must be joined across heterogeneous sources, or (d) open-ended exploratory analysis where there is no single correct answer. The QRData benchmark partially addresses (a) and tests causal reasoning, but it is a single additional data point.

Task type coverage: The paper emphasizes that DATAMIND models exhibit "balanced performance across all datasets" (Section 5.2) in contrast to specialized models that collapse on out-of-domain formats. However, "balanced" here means a spread of ~18 percentage points between best and worst benchmark (77.30% to 59.41% for the 7B model). This is better than OmniSQL's ~30-point spread, but it still indicates that the model is substantially more capable on CSV-based analysis than on SQL-based analysis—the training data's format distribution (3,400 .csv + 560 .xlsx vs. 1,954 .sqlite files) may encode a format bias that persists in the trained model.

What evidence exists in the paper. The Llama-3.1-8B result (Table 4) is a single model at one scale on the main three benchmarks. The QRData result (Table 6) tests the 7B and 14B models on one additional benchmark with positive results (the 14B model leads all baselines). The Data Interpreter experiment (Table 5) shows that a prompt-engineering scaffold designed for GPT-4o fails to transfer to other models, indirectly supporting the paper's claim that training is more robust than prompt engineering. But no experiment systematically varies the base model family across the full pipeline.

Mitigation status. The paper does not claim universality—the limitations section (Appendix B) acknowledges that "not all mainstream benchmarks are covered in our evaluation suite" and that "our experimental backbone is restricted to the Qwen family, with model scale capped at 14B." These are honest disclosures, but they leave the generalizability question open. The release of DATAMIND-12K and the trained models enables the community to conduct cross-family and cross-benchmark evaluations, which is a partial mitigation through openness rather than through experimental design.

6.5 The Dynamic SFT+RL Schedule Was Developed Without Systematic Ablation of Cold Start Interaction

The assumption or constraint. The final DATAMIND training recipe combines two mechanisms: (1) a cold start phase of pure SFT training before the joint SFT+RL phase begins, and (2) a cosine-annealed dynamic γ schedule during joint training. The paper analyzes these mechanisms independently—Figure 6 and Figure 7 study γ dynamics without cold start, and Figure 8 studies cold start duration with the dynamic γ schedule—but never reports an experiment that isolates their interaction. The amount of cold start (number of SFT-only epochs before RL begins) is swept in Figure 8, but the dynamic schedule parameters (γ peak = 0.9, γ valley = 0.05, cosine decay) are held fixed across all cold start conditions.

The consequence. It is impossible to determine from the reported experiments whether the dynamic γ schedule provides benefit above and beyond what an appropriately chosen fixed γ would provide when cold start is present. The paper's analysis of fixed γ failure modes (γ = 0 collapses, γ = 0.2 collapses, γ = 0.8 causes entropy collapse) was conducted without cold start. With cold start, the model enters joint training at a substantially higher capability level—cold start SFT alone achieves 62.54% (Table 2). At this higher initial capability, the fixed γ failure modes may not manifest: a model that already produces well-formed trajectories might not collapse under pure RL, and the entropy collapse observed at γ = 0.8 might be less severe.

This matters because the dynamic schedule adds complexity (an additional hyperparameter, the cosine decay rate, and the interaction with cold start duration). If a simpler fixed-γ schedule with appropriate cold start achieves comparable performance, the paper's headline claim about the importance of dynamic scheduling would be weakened. The "raising a child" analogy (Section 5.4) is intuitively appealing but does not constitute evidence that the dynamic schedule is necessary rather than merely sufficient.

What evidence exists in the paper. Figure 8 shows RL performance after 0, 1, 2, and 3 cold start epochs with the dynamic schedule. The gap between cold start and RL narrows as cold start increases, which is evidence about cold start's effect but not about whether dynamic γ is needed. Table 2 shows that SFT-and-RL (68.10%) outperforms SFT-then-RL (63.42%)—but SFT-then-RL uses a different training regime (discrete switch rather than continuous blending), so this is evidence for continuous blending over sequential training, not for dynamic scheduling over fixed blending. The paper does not report a fixed-γ SFT-and-RL baseline with cold start that would directly test whether the schedule matters.

Mitigation status. The paper does not acknowledge this as a limitation and does not report the necessary ablation. The training stability analysis (Section 5.4) is described as providing "insights gained from our exploratory trials" and is presented as empirical guidance rather than rigorous ablation. The lack of a cold-start + fixed-γ experiment is a gap in the evidence for the dynamic schedule's necessity.

6.6 Latency and Serial Dependency of Multi-Turn Agent Execution

The assumption or constraint. The DATAMIND agent operates in a multi-turn ReAct loop where each turn requires: (1) the model to generate a thought and action, (2) the code interpreter to execute the action and return results, (3) the model to process the results and decide on the next action. This loop is inherently serial—the model cannot plan turn t+1 until it receives the execution feedback from turn t. The paper sets a maximum of T = 10 interaction rounds per query (Section 5.1) and implements asynchronous interaction during training to decouple GPU and CPU demands across different samples (Section 4), but within a single sample, the execution is strictly sequential.

The consequence. At inference time, the agent's latency per query is proportional to the number of turns it takes to arrive at an answer, with each turn incurring the full cost of model generation plus code execution. For a query requiring 5 turns, the user experiences 5 serial generation steps plus 5 code execution waits. This is fundamentally different from single-pass models (like the vanilla ReAct baselines in Table 1) that generate a complete answer in one forward pass. The paper reports only accuracy metrics and provides no latency measurements—tokens generated per query, wall-clock time per query, or distribution of turn counts across the test set.

This matters for deployment scenarios where users expect interactive response times. A 7B model generating 2,000 tokens across 5 turns at typical inference speeds might take 10–20 seconds of generation time, plus code execution overhead. This is acceptable for batch analytics but prohibitive for exploratory analysis where users iterate rapidly. For comparison, the proprietary baselines (DeepSeek-V3.1, GPT-5) are evaluated in a zero-shot ReAct setting where they may produce fewer turns (the paper does not report turn counts), potentially achieving lower latency even if their accuracy is similar.

What evidence exists in the paper. None. The paper reports no latency or throughput measurements. The training section (Section 5.1) mentions that the maximum number of interaction rounds T is set to 10, but the actual distribution of turn counts during inference is not reported. The inference batch size is 5 (Section 5.1), suggesting some parallelization across queries, but per-query latency is unaddressed.

Mitigation status. The paper does not discuss latency as a limitation. The chunked code maintenance optimization (Section 4) reduces memory pressure but does not reduce serial dependency. The asynchronous training design (decoupling generation and execution across samples) is a training-time optimization that does not affect inference latency. The length penalty in the reward function (Equation 8) incentivizes concise answers, which indirectly reduces generation tokens, but does not address the fundamental serial bottleneck of the multi-turn architecture.

7. Implications and Future Directions

How This Work Changes the Landscape

DATAMIND shifts the conversation around agent training from prompt engineering over proprietary models toward scalable supervised-and-reinforcement learning recipes that make open-source models competitive. Before this work, the dominant approach for data-analytic agents was to build increasingly elaborate scaffolds (multi-agent architectures, dynamic workflow graphs, case-based reasoning) around closed-source models like GPT-4. The implicit assumption was that open-source models lacked the fundamental reasoning capability for real data analysis, and that the best we could do was engineer around their limitations with clever prompting. DATAMIND falsifies this assumption: a 14B open-source model, trained on 12K synthetically generated trajectories with a carefully designed SFT+RL recipe, matches or exceeds GPT-5 and DeepSeek-V3.1 on average across three diverse data analysis benchmarks (71.16% vs. 70.58% vs. 69.44%, Table 1).

This is not just an incremental improvement over prior open-source trained models—it represents a qualitative category shift from "these models cannot do the task at all" to "they are competitive with the best available systems." The vanilla ReAct 7B baseline scores 11.26% (Table 1). TableLLM-7B, the prior state-of-the-art trained model for tabular reasoning, scores 29.90%. DATAMIND-7B scores 68.10%. This ~6× improvement over the untrained baseline and ~2.3× improvement over the prior trained baseline is not a matter of tuning hyperparameters—it reflects a fundamentally different approach to agent construction.

The paper also resolves a contradiction in the broader agent training literature. Prior work on reinforcement learning for language agents has produced mixed signals. Some studies (DeepSeek-R1, Kimi K2, various GRPO applications) showed dramatic gains from RL. Others, particularly in multi-turn settings, found that RL training collapses or adds marginal benefit over SFT alone. The paper's dynamic SFT+RL analysis (Figures 6, 7, and Table 2) provides a diagnostic framework for understanding when RL helps and when it fails: pure RL collapses because initial rollouts are too poor (γ=0 case), weak SFT fails to prevent policy drift (γ=0.2 case), strong SFT causes entropy collapse (γ=0.8 case), and only a schedule that transitions from high to low SFT weighting maintains both stability and exploration (dynamic γ). This is not a new algorithm but a new understanding of an existing algorithm's failure modes, and it provides actionable guidance for practitioners applying RL to other multi-turn agent domains.

The finding that data quality control (self-consistency filtering) matters more than data curation (best-trajectory selection) (Figure 4) challenges a deeply held assumption in synthetic data generation. The standard approach in the field—generate candidates, score them, keep the best—implicitly prioritizes per-instance quality over distributional diversity. DATAMIND shows that within a self-consistent set (all trajectories converge to the same answer), randomly selecting a trajectory or keeping all of them outperforms judge-based "best" selection. This finding, if it generalizes, implies that a substantial amount of effort currently spent on reward modeling, scoring, and selection in synthetic data pipelines could be redirected toward cheaper consistency checks and diversity preservation.

The work also redirects research attention from agent architecture design to agent data and training design. Prior work on data-analytic agents (DS-Agent, AutoKaggle, Data-Copilot, Data Interpreter, AgenticData) invested heavily in workflow engineering—decomposing tasks into sub-tasks, orchestrating multiple specialized agents, building dynamic planning graphs. DATAMIND shows that a simple ReAct loop, when the underlying model is properly trained, outperforms elaborate scaffolds on weaker models. The Data Interpreter comparison (Table 5) is particularly revealing: Data Interpreter's sophisticated scaffold degrades DeepSeek-V3.1's performance compared to vanilla ReAct (67.70% vs. 81.32% on DABench), and while it helps the 7B model (54.09% vs. 15.05%), it falls far short of DATAMIND-7B's trained performance (77.30%). The implication is that scaffold complexity is a poor substitute for model capability, and investment in training pipelines yields higher returns than investment in prompt engineering.

Follow-Up Research This Work Enables

Directly combining PRM-based search with the DATAMIND training recipe for inference-time scaling. The paper establishes that DATAMIND-14B is competitive with DeepSeek-V3.1 and GPT-5 using only pass@1 evaluation—a single sampled trajectory per query. But the pass@3 results (Table 1) show substantial headroom: DATAMIND-14B achieves 80.25% pass@3 versus 71.16% pass@1, a gap of ~9 percentage points. This gap indicates that the model generates correct answers on many queries where its first sample is wrong. The natural next step is to apply process reward model (PRM) guided search or best-of-N weighted selection at inference time, using either the GPT-4o-mini judge from training or a fine-tuned outcome reward model. A concrete experiment: train an ORM on DATAMIND-12K trajectories (which already have correctness labels from the consistency judge), apply best-of-N weighted selection at N=4, 8, 16, and measure how much of the pass@1–pass@3 gap can be recovered through test-time compute. The finding in the reference scaling-laws paper—that compute-optimal test-time strategies yield 4× efficiency gains over best-of-N—suggests that a few additional inference samples with intelligent selection could push DATAMIND's pass@1 close to its pass@3 ceiling without any further training.

Training a difficulty-aware compute allocation policy for data analysis queries. The paper's data ablation (Figure 3) shows that model performance varies substantially across queries—the pass@1–pass@3 gap of ~11 points at 12K trajectories means the model's first-attempt reliability is non-uniform. Some queries are "easy" for the model (high probability of correct first attempt) and others are "hard" (low probability). Building on the compute-optimal test-time scaling framework from the reference paper, a natural extension is: (1) use DATAMIND's GPT-4o-mini judge to label query difficulty by measuring consistency across multiple model samples, (2) train a lightweight difficulty classifier on question text and data file metadata, (3) implement an adaptive inference policy that allocates more samples (best-of-N or beam search) to hard queries and fewer to easy ones, under a fixed total inference budget. The key metric would be: at matched total inference cost, does difficulty-aware allocation outperform uniform allocation? The cross-benchmark evaluation (DABench, TableBench, BIRD) provides natural difficulty variation—BIRD queries (59.41% for 7B) are systematically harder than DABench queries (77.30% for 7B), suggesting that format-based difficulty is a strong prior.

Extending the training recipe to predictive modeling and visualization tasks. The paper explicitly excludes training, predictive, and data-visualization tasks (Appendix B), limiting the agent to descriptive and inferential analysis. Extending DATAMIND to these categories would require: (1) expanding the task taxonomy to include categories like "Model Training," "Hyperparameter Tuning," "Feature Selection," "Model Evaluation," "Visualization Design," (2) writing procedural knowledge k for each new category (e.g., for model training: inspect data types → handle missing values → encode categoricals → split train/test → select model class → fit → evaluate → report metrics), (3) generating queries that require scikit-learn model fitting and matplotlib/seaborn plotting (the trajectory synthesis prompt in Appendix I.3 currently says "Don't use visualization libraries like matplotlib or seaborn, as the user will not be able to see the plots"—this would need modification), (4) designing a reward function that can evaluate plot quality (model-as-judge with multimodal capabilities, or rule-based checks on figure properties), and (5) handling the increased computational cost of model training within the sandboxed execution environment. A strong follow-up would evaluate on DSBench (Jing et al., 2025) or DataSciBench (Zhang et al., 2025a), both of which include ML training tasks, and compare against prompt-engineered systems like Data Interpreter that already support some ML workflows.

Stress-testing the dynamic γ schedule against simpler alternatives with proper cold start. Section 6.5 of this analysis identified a confound: the paper demonstrates dynamic γ's benefits without cold start (Figures 6, 7), while the final model uses both cold start AND dynamic γ, making it impossible to attribute gains to the schedule alone. A rigorous follow-up would run the full DATAMIND training pipeline (including cold start) with three ablations: (1) dynamic γ as in the paper (cosine annealed 0.9→0.05), (2) fixed γ = 0.2 throughout joint training, and (3) fixed γ = 0.5 throughout joint training. The hypothesis: with sufficient cold start, the fixed-γ failure modes (collapse for γ=0, entropy collapse for γ=0.8) may not occur because the model enters joint training at high capability (62.54% from SFT alone, Table 2). If fixed γ with cold start matches dynamic γ performance, the "raising a child" analogy is an appealing narrative but not a necessary training strategy. If dynamic γ still outperforms, the schedule's benefit is robust. Either outcome provides valuable guidance for practitioners adapting the recipe to new domains.

Systematic comparison of trajectory synthesis strategies at matched data volume. The paper shows that 12K high-quality DATAMIND trajectories outperform 2.5M OmniSQL trajectories, but this confounds quality and quantity. A controlled experiment would: (1) generate trajectories at varying quality levels by ablating the DATAMIND synthesis pipeline (e.g., remove procedural knowledge k, remove self-consistency filtering, remove the reflection loop), (2) train models on matched data volumes (e.g., 5K, 10K, 20K trajectories) from each quality tier, (3) measure performance scaling curves. The hypothesis: high-quality trajectories (full DATAMIND pipeline) produce steeper scaling curves than medium-quality (no reflection) or low-quality (no consistency filtering), meaning the quality advantage compounds with data volume. If instead all quality tiers produce parallel scaling curves (same slope, different intercept), then quality and quantity are independent levers and the optimal strategy is to maximize whichever is cheaper. This experiment would quantify the return on investment for each component of the curation pipeline, enabling practitioners to make cost-aware decisions about which filtering steps to include.

Cross-family and cross-scale replication of the full training recipe. The paper demonstrates DATAMIND on Qwen-2.5-Coder 7B and 14B, with a single Llama-3.1-8B validation point (Appendix G.1). A comprehensive replication would apply the full pipeline to: (1) a different architecture family (DeepSeek-Coder, Mistral, Gemma), (2) a significantly larger scale (32B, 70B) to test whether the dynamic γ schedule and void-turn filtering scale with model size, and (3) a significantly smaller scale (1–3B) to identify the minimum viable model size for multi-turn code-executing agents. The key question: do the training dynamics (reward curves, entropy curves, optimal γ schedule) transfer across architectures, or are they specific to Qwen's pretraining? The Llama-8B result (65.56%, Table 4) is promising but only tests the final performance, not the training trajectory. A negative result—e.g., Llama models require different γ scheduling or experience different failure modes—would refine our understanding of when and why the DATAMIND recipe works.

Practical Applications and Downstream Use Cases

Low-cost, reproducible data analysis for scientific research labs. Academic research groups in biology, social sciences, and other data-rich fields routinely face the bottleneck of having datasets that require analysis but lacking dedicated data scientists. A deployed DATAMIND-14B agent could serve as an on-premise, zero-marginal-cost analytics assistant: a researcher uploads their CSV or Excel file, asks natural language questions ("What is the correlation between treatment dosage and patient outcome, controlling for age?"), and receives analyzed answers with executable code that can be inspected and verified. The 80.29% DABench accuracy (Table 1) on CSV-based analysis tasks means the agent is reliable for the majority of routine analytical queries, and the open-source nature means the model can run on institutional hardware without sending sensitive research data to proprietary APIs. The pass@3 score of 88.72% suggests that running three independent analyses and taking the majority answer would push reliability above 85%—comparable to a junior data analyst—without human intervention.

Training data generation for domain-specific analytical agents. Organizations with proprietary datasets in specialized domains (healthcare records, financial transactions, manufacturing sensor data) cannot use off-the-shelf data analysis agents trained on public data—the file schemas, terminology, and analytical conventions are domain-specific. The DATAMIND synthesis pipeline provides a template: (1) collect domain-specific data files (even a few hundred will suffice, given the paper's data scaling in Figure 3 shows strong performance at 2K trajectories), (2) write domain-specific exemplars for the 18 task categories (or adapt the categories to domain needs—e.g., "Clinical Trial Analysis" might replace "Domain Specific Numerical Reasoning"), (3) generate queries and trajectories using a strong proprietary model guided by procedural knowledge, (4) filter through self-consistency, (5) train a smaller open-source model on the domain-specific trajectories. The paper shows that SFT alone achieves 62.54% (Table 2), so even organizations without RL training infrastructure can produce useful agents. The cost is the proprietary model inference for trajectory synthesis, which is a one-time expense amortized over all future analyses.

Automated SQL report generation from enterprise databases. DATAMIND's BIRD performance (62.23% pass@1, 70.21% pass@3 for the 14B model, Table 1) demonstrates competence on multi-table SQL analysis. In an enterprise setting, business analysts routinely ask questions like "Show me monthly revenue by product category for the last quarter, broken down by region, with year-over-year growth." These queries require multi-table joins, temporal aggregations, and window functions—exactly the skills tested by BIRD. A deployed DATAMIND agent connected to an enterprise data warehouse could accept natural language questions, generate and execute SQL queries, and return formatted results, reducing the bottleneck on SQL-proficient analysts. The 70.21% pass@3 rate means that with three independent attempts, the agent correctly answers ~70% of complex SQL questions—not production-reliable for mission-critical reporting, but sufficient for exploratory analysis where a human can verify results, and substantially faster than writing queries from scratch.

GitHub-integrated data analysis bot for open-source projects. Many open-source projects distribute datasets alongside code (benchmark results, performance measurements, community survey data) that are rarely analyzed systematically. A GitHub Action integrating DATAMIND could automatically generate descriptive analyses and trend reports whenever new data is committed: it reads the updated CSV file, generates summary statistics, identifies notable changes from previous versions, and posts a structured analysis comment on the pull request. This use case leverages DATAMIND's strength on DABench-style tasks (80.29% for 14B, Table 1) and the fact that the model runs on consumer-grade hardware (14B parameters at 8-bit quantization fits on a single GPU). The accuracy is sufficient for automated flagging of interesting patterns, with the understanding that humans make final decisions—a human-in-the-loop deployment where the agent amplifies rather than replaces analyst attention.

When to Prefer This Method

The paper articulates an implicit tradeoff between training open-source agents via the DATAMIND recipe versus building prompt-engineered scaffolds around proprietary models. The choice turns primarily on three factors: the availability of domain-specific training data, the inference cost sensitivity, and the need for reproducibility and customization.

Prefer the DATAMIND training recipe when:

  • You have access to domain-specific data files (even a few hundred) and can afford the one-time cost of trajectory synthesis using a proprietary model. The paper shows that performance scales from 2K to 12K trajectories with no plateau (Figure 3), so modest data volumes yield useful agents, and the synthesis pipeline is fully automated once procedural knowledge and exemplars are written.
  • Inference cost per query matters (high-volume analytics, edge deployment, or budget-constrained environments). A 7B model running on a single GPU costs a fraction of a cent per query, while proprietary API calls for GPT-5 or DeepSeek-V3.1 can cost orders of magnitude more for multi-turn trajectories.
  • Reproducibility, auditability, or data privacy is required. The open-source model can be run on-premise with full control over data flow, and the training recipe is fully documented for independent verification.
  • You need consistent performance across diverse file formats. Specialized models (OmniSQL on SQL, TableLLM on small tables) collapse outside their training distribution, while DATAMIND's multi-format training data produces balanced performance (Table 1).

Prefer prompt-engineered proprietary agents when:

  • The task domain includes capabilities DATAMIND explicitly excludes—predictive modeling, data visualization, large-scale file processing. These require scaffolds (for visualization, tool orchestration) or base model capabilities (for ML training) that the current DATAMIND training data does not cover.
  • Inference latency is the dominant constraint and you have access to low-latency proprietary APIs. The paper provides no latency data, but a proprietary model generating a complete analysis in one pass may be faster than a 7B model running 5–10 serial ReAct turns.
  • You need to handle queries that require extensive world knowledge or domain commonsense beyond what's in the training data. DATAMIND-7B underperforms on QRData (57.66% vs. 60.75% for DeepSeek-V3.1, Table 6), suggesting that smaller open-source models have knowledge gaps that larger proprietary models fill through pretraining scale.
  • The analytic task is so easy that even an untrained model handles it reliably (e.g., single-step aggregation on a small table), in which case the training investment is unnecessary and any capable model with a simple prompt suffices.