DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI

ArXiv: 2512.16676

🎯 Pitch

A unified 10K-sample dataset produced by this framework enables base models to outperform counterparts trained on 1M Infinity-Instruct instances, while the system’s PyTorch-style abstractions turn ad-hoc LLM data preparation into composable, reproducible pipelines.

1. Executive Summary

This paper introduces DataFlow, a unified LLM-driven framework for data preparation that elevates model-in-the-loop synthesis to a first-class, programmable abstraction through a PyTorch-style pipeline construction API. Built around nearly 200 reusable operators organized into four functional categories—generation, evaluation, filtering, and refinement—the system provides six domain-general pipelines spanning text, mathematical reasoning, code, Text-to-SQL, agentic RAG, and large-scale knowledge extraction, with all pipelines adhering to a common generate–evaluate–filter–refine paradigm rather than ad-hoc scripting. The framework achieves up to +3% execution accuracy in Text-to-SQL over the 2.5M-sample SynSQL corpus while using under 0.1M training examples, +7% average improvements on code benchmarks, and 1–3 point gains on MATH, GSM8K, and AIME over curated synthetic baselines, with a unified 10K-sample multi-domain corpus enabling base models to surpass counterparts trained on 1M Infinity-Instruct instances. The paper establishes that principled, composable dataflow abstractions can produce training data that matches or exceeds both human-curated datasets and large-scale synthetic corpora, though the agentic orchestration layer's ability to synthesize entirely new operators from natural-language specifications degrades substantially under under-specified queries, dropping from a 0.92 LLM-Judge score on explicit descriptions to 0.60 on high-level requirements.

2. Context and Motivation

The Core Problem: Data Preparation for LLMs Remains Fragmented and Unstandardized

The fundamental gap this paper addresses is that LLM data preparation—despite being arguably the most critical factor in model quality—lacks principled abstractions, standardized workflows, and first-class support for LLM-driven synthesis. As the paper states in Section 1:

"most practitioners still rely on ad-hoc scripts and loosely standardized workflows, which lack explicit dataflow abstractions, well-defined atomic operators, or any form of pipeline-level optimization."

This isn't merely an inconvenience. It creates concrete problems that reverberate throughout the LLM development lifecycle:

• Reproducibility failures: Two teams attempting the same data preparation strategy often produce different results because their ad-hoc scripts implement subtly different logic (different ordering of filtering steps, inconsistent handling of edge cases, undocumented hyperparameters). There is no shared protocol for expressing "this is how I prepared my data" that another team can faithfully replicate.

• Composability is absent: A filtering technique developed for code data cannot be cleanly reused for mathematical reasoning data without modification, because operators are not defined with standardized input/output semantics. Each pipeline becomes a monolith.

• No optimization surface: Because pipelines aren't expressed as explicit dataflow graphs, there's no way to analyze dependencies, detect redundant computation, parallelize independent operations, or checkpoint intermediate states for resumption. Every pipeline execution is effectively a fresh run.

• The verification gap: With ad-hoc scripts, there is no standard way to validate that a pipeline is structurally sound before execution. Missing field dependencies, type mismatches, and circular dataflows are discovered only at runtime—often after expensive LLM API calls have already been made.

This fragmentation matters because data preparation has become the dominant cost and complexity driver in LLM development. As the paper notes in Section 2.1, scaling-law studies consistently demonstrate that data quality and quantity are central to model performance, and the trend toward fine-grained post-training tasks (instruction tuning, chain-of-thought generation, function calling) has made semantic accuracy in data preparation essential for achieving precise task-level model behavior.

Why This Problem Is Important: The Shift from Data Consumers to Data Producers

The paper identifies a tectonic shift that makes standardized data preparation frameworks urgently necessary: LLMs are no longer only consumers of data, but also producers. Section 1 frames this explicitly:

"LLMs are no longer only consumers of data, but also producers. Because large-scale human annotation is prohibitively expensive, recent work heavily leverages LLM-based data synthesis workflows to construct high-quality corpora at scale. Multiple recent reports show that, in many regimes, carefully synthesized data can outperform even high-quality selected data, further underscoring the importance of LLM-driven generation workflows."

This shift has profound implications for what a data preparation system must support:

Pre-LLM era data preparation was dominated by extract–transform–load (ETL) patterns: download raw text (Common Crawl, Wikipedia), apply heuristic filters (language detection, deduplication, length filtering), and produce a cleaned corpus. The computation was largely rule-based and embarrassingly parallel. Systems like Apache Spark, Dask, and Hadoop were designed for exactly this pattern.

Post-LLM era data preparation is fundamentally different. Modern pipelines involve iterative, model-in-the-loop workflows where an LLM generates candidate outputs, a verifier (often another LLM) scores them, filters remove low-quality samples, and refinement stages polish the survivors—potentially feeding back into another round of generation. The computation is heterogeneous (LLM inference + rule-based filtering + embedding-based deduplication), stateful (scores and metadata accumulate across stages), and semantically complex (prompt templates must be carefully constructed and versioned).

The paper argues that existing systems fail to support this new paradigm:

"These frameworks can, in principle, run semantic cleaning by calling LLMs or embedding models as user-defined functions, but they provide no native support for model-in-the-loop processing, GPU-efficient batching, or token-level text operations." (Section 2.2)

This isn't just about developer convenience. It's about whether the field can systematically improve data preparation or whether progress will remain locked in scattered, irreproducible repositories. When every lab reinvents filtering, synthesis, and refinement from scratch, the field cannot accumulate knowledge about what works. A standardized framework would enable controlled experiments ("does this new filtering operator improve downstream performance when inserted at position 3 in a known-good pipeline?") rather than whole-pipeline comparisons where the source of improvement is unclear.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work and articulates specific limitations for each:

1. General Big-Data Engines (Spark, Dask, Hadoop)

These systems (Section 2.2) were designed for structured data processing in the pre-LLM era. Their limitations are fundamental, not superficial:

• No native model-in-the-loop support: While you can call an LLM API from a Spark UDF, the framework provides no batching, rate-limiting, retry logic, or GPU-aware scheduling. Every team must reimplement this infrastructure.

• Operators designed for structured data: Built-in operators focus on numerical aggregations, joins, and string operations. Essential LLM-specific operations—tokenization, language detection, document segmentation, semantic deduplication, safety filtering—must be implemented as ad-hoc UDFs, which are harder to test, version, and compose.

• Abstraction mismatch: The row-at-a-time transformation model of these frameworks doesn't naturally express operations that need to see all candidates simultaneously (e.g., deduplication, diversity filtering, inter-sample scoring), or operations that generate new rows conditionally (e.g., rejection sampling with regeneration).

"This leads to significant overhead and engineering complexity, making general big-data engines inadequate for the large-scale, semantics-heavy pipelines required for LLM corpus construction." (Section 2.2)

2. LLM-Specific Data Curation Frameworks (NeMo Curator, Data-Juicer)

The paper acknowledges that NeMo Curator and Data-Juicer represent significant progress—they "substantially improve the efficiency and quality of LLM data preparation" (Section 2.3)—but identifies a critical architectural limitation: they remain fundamentally extraction- and filtering-oriented.

Table 1 in Section 2.3 makes this distinction explicit. NeMo Curator is characterized as providing "Minimal" LLM integration focused "mainly [on] filtering." Data-Juicer provides "Partial" integration with "some gen ops." Both are described as "largely configuration-centric toolkits" whose "abstractions provide limited support for expressing iterative, model-in-the-loop generative workflows with fine-grained semantic control" (Section 1).

What does "configuration-centric" mean concretely? In these systems, pipelines are typically specified through YAML files that list a sequence of pre-built operators with their parameters. This works well when the operator library covers your needs. But when you need to:

• Synthesize new data using a specific prompt strategy not covered by existing operators
• Refine existing data through multiple rounds of LLM critique and revision
• Implement a custom evaluation that requires calling multiple LLMs in sequence
• Condition generation on results from previous filtering stages in non-trivial ways

...you quickly run into the limits of what can be expressed through configuration alone. The paper's core thesis is that LLM-driven synthesis must be elevated to a first-class, programmable abstraction—not tacked on as a special case of a filtering-oriented system.

3. Ad-Hoc Synthesis Scripts and Dataset-Specific Pipelines

The paper observes that many of the best-performing synthetic datasets in recent literature (Open-R1, Synthetic-1, SynSQL, Infinity-Instruct) were produced by purpose-built scripts that are:

• Tightly coupled to specific models (e.g., DeepSeek-R1 for CoT generation) and specific tasks
• Not designed for reuse across domains
• Difficult to inspect, reproduce, or extend without access to the original authors' environment

The result is that each new domain or task requires rebuilding infrastructure from scratch. The paper positions DataFlow as a substrate where these domain-specific synthesis strategies can be expressed as compositions of reusable operators rather than monolithic scripts, enabling knowledge accumulation across projects.

How This Paper Positions Itself

DataFlow positions itself not as an incremental improvement to existing frameworks but as a conceptual reorientation of what a data preparation system should be:

From configuration-centric to code-first. DataFlow adopts a PyTorch-like programming interface where pipelines are Python classes with __init__() (resource allocation and operator configuration) and forward() (execution logic with explicit key bindings). This is framed as superior to YAML-based configuration for two reasons: (1) it enables IDE-native development (code completion, navigation, type checking), and (2) it makes conditional logic, loops, and dynamic behavior expressible naturally within the pipeline definition rather than requiring framework-specific template languages.

From filtering-oriented to synthesis-first. The generate–evaluate–filter–refine paradigm (Section 4.3) makes synthesis the first stage in every pipeline, with filtering and refinement serving as quality-control steps downstream. This reflects the paper's bet that LLM-driven generation will be the dominant data source going forward, with human-curated data serving as seed material rather than the final product.

From monolithic scripts to composable operators. By defining operators with standardized key-based I/O bindings (input_* and output_* keys that map to named columns in a shared tabular storage), DataFlow enables operators developed for one domain to be reused in another with zero code changes—only the key bindings change. The Text-to-SQL case study (Section 6.1) illustrates this concretely: the same SQL Generator operator works across MySQL, SQLite, and PostgreSQL by simply swapping the prompt template, with no operator-level modifications.

From manual construction to agentic orchestration. DataFlow-Agent (Section 5) represents a bet that the abstractions themselves—standardized operators with explicit I/O contracts, prompt templates with parameterized slots, pipelines as DAGs—enable automated workflow construction from natural language that goes beyond simple parameter selection. The agent can synthesize entirely new operators when existing ones don't match the user's intent, using a retrieve-reuse-synthesize strategy with sandboxed debugging. This is distinguished from Data-Juicer's agent, which the paper characterizes as "largely constrained to parameterizing and sequencing a static library of pre-existing operators" (Section 5.3).

From closed development to ecosystem thinking. The extension mechanism (Section 4.4) is designed to create a Python-package-based ecosystem where domain experts can publish operators, prompt templates, and pipelines that others can import and compose. The CLI scaffolding that generates extension stubs lowers the barrier to contribution. This mirrors the role PyTorch's nn.Module played in standardizing model development—the paper explicitly draws this analogy: "much like how torch.nn.Module standardizes model composition in deep learning" (Section 1).

The Tension the Paper Acknowledges

Importantly, the paper is explicit about an inherent tension in its design: "arbitrarily many domains" produce "an open-ended set of domain-specific algorithms" that must be accommodated while maintaining "a stable and comprehensible operator space" (Section 4.3). The solution—a multi-dimensional categorization scheme (modality, core vs. domain-specific, functional)—is presented as a practical compromise rather than a theoretical solution. Core operators are "intentionally limited in number and relatively stable" and serve as the conceptual basis from which domain operators derive, but the domain operator space "expand[s] without bound as new domains, modalities, or tasks emerge."

This honesty about the tension is significant. It acknowledges that the framework cannot predefine all operators users will ever need—hence the importance of the extension mechanism and the agentic synthesis capability. The framework's value proposition is not completeness but structure: it provides the abstractions, interfaces, and composability guarantees within which an unbounded set of operators can coexist, be discovered, and be composed.

3. Technical Approach

3.1 Reader Orientation

DataFlow is a software framework that lets you build data preparation pipelines for large language models by composing reusable building blocks—operators, prompt templates, and storage backends—using a Python API that looks and feels like PyTorch. The problem it solves is the fragmentation and irreproducibility of LLM data preparation: instead of every team writing ad-hoc scripts that call LLM APIs, apply filters, and chain transformations in custom ways, DataFlow provides a standardized substrate where pipelines are expressed as explicit directed acyclic graphs (DAGs) of operators with defined input/output contracts, enabling composition, reuse, verification, and agentic construction from natural language.

3.2 Big-Picture Architecture (Diagram in Words)

The DataFlow architecture has six major components arranged in layers:

Bottom layer — Storage and Serving. A global DataFlowStorage abstraction maintains the canonical tabular representation of the dataset (rows = samples, columns = named fields) and exposes read()/write(data) operations. Decoupled from this is the LLM Serving API, which provides a unified generate_from_input(user_inputs, system_prompt, json_schema) entry point that abstracts over local inference engines (vLLM, SGLang) and online API services (ChatGPT, Gemini), handling batching, retries, and rate limiting transparently.

Middle layer — Operators and Prompt Templates. Operators are the fundamental transformation units. Each has an __init__() that configures static parameters and binds to serving/template objects, and a run(storage, input_*_key=..., output_*_key=...) method that reads named columns from storage, applies its transformation logic, and writes results to new named columns. LLM-driven operators additionally bind to a PromptTemplate object during initialization, whose build_prompt() method assembles task-relevant information into concrete prompts. Operators are organized across three orthogonal categorization dimensions: modality (text, visual, document), core vs. domain-specific, and functional (generate, evaluate, filter, refine).

Top layer — Pipelines. Pipelines are Python classes with __init__() (resource allocation, operator instantiation) and forward() (sequential or DAG-ordered execution of operators with explicit key bindings). They support compile() for static dependency analysis and validation, resume() for checkpointing, and stepwise execution. The key bindings implicitly define the dataflow topology—naming which input columns each operator reads and what output columns it writes.

Extension layer. DataFlow-Extensions are standalone Python packages containing additional operators, prompt templates, and pipelines. A CLI tool scaffolds new extension packages, and the ecosystem supports pip-installable distribution.

Agentic layer — DataFlow-Agent. Built on LangGraph, this orchestrates a multi-agent workflow that translates natural-language specifications into executable pipelines through four stages: intent decomposition (breaking high-level goals into sub-intents), operator synthesis (retrieving existing operators or generating new code via RAG-based few-shot learning with sandboxed debugging), pipeline assembly (constructing a DAG of operators), and verification (executing the pipeline in a sandbox, identifying runtime errors, and adjusting parameters or connections).

Output layer. Pipelines produce high-quality, task-aligned datasets consumed by downstream LLM training, evaluation, and retrieval applications.

Information flows through the system as follows: raw data enters via the storage layer → operators read fields, optionally invoke the LLM serving layer through prompt templates, and write results back to storage → the pipeline's forward() method sequences these operations according to explicit key dependencies → the compiled DAG validates structural correctness → the agentic layer (optionally) assembles this entire structure from natural language → final outputs are exported in common formats (JSON, JSONL, CSV, Parquet) for downstream consumption.

3.3 Roadmap for the Deep Dive

• First, the global storage abstraction (Section 4.1) because it is the backbone every operator reads from and writes to, and its design determines what composability guarantees are possible.
• Second, the LLM Serving API (Section 4.2.1) because LLM-driven operators—which constitute the majority of DataFlow's operator library—depend on it, and its unified abstraction is what enables backend-agnostic pipeline construction.
• Third, the operator programming interface (Section 4.2.2) because operators are the fundamental unit of computation, and their standardized key-based I/O contract is what enables composability, reuse, and agentic orchestration.
• Fourth, the prompt template interface (Section 4.2.3) because it explains how operators achieve domain flexibility without code changes—the mechanism that separates transformation logic from prompting strategy.
• Fifth, the pipeline composition interface (Section 4.2.4) because pipelines compose operators into end-to-end workflows, and the compile() procedure provides the structural guarantees that agentic verification depends on.
• Sixth, the operator categorization scheme (Section 4.3) because it reconciles the tension between unbounded domain requirements and the need for a comprehensible operator space—the design problem at the heart of any unified framework.
• Seventh, the DataFlow-Agent architecture (Section 5) because it represents the most ambitious claim in the paper: that the framework's abstractions are sufficiently well-structured to enable automated pipeline construction from natural language, including synthesizing entirely new operators.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and framework paper whose core idea is that LLM data preparation should be elevated from ad-hoc scripting to a principled, composable, and programmable dataflow abstraction—analogous to how PyTorch's nn.Module standardized neural network construction—and that providing such abstractions enables not only reproducibility and reuse but also agentic automation of pipeline construction.

The Global Storage Abstraction

The global storage abstraction is the foundation on which all operator composability rests. The paper defines it through a base class DataFlowStorage that exposes exactly two operations:

• read(): retrieve the current dataset (or specified fields) in a format required by the calling operator.
• write(data): update or append fields to the shared dataset representation.

This minimalist interface is deliberate. By constraining all data access to these two operations, the framework ensures that operators never communicate directly with each other or with the underlying filesystem. Instead, every operator's output becomes immediately available to every subsequent operator through the shared storage, and operators remain agnostic to whether the backing store is a Pandas DataFrame in memory, a Parquet file on disk, or a distributed database.

The default implementation uses Pandas as the execution substrate. Each sample is represented as a row in a DataFrame, and each field (instruction, response, chain-of-thought trace, score, metadata) is a named column. This tabular representation is described as "a suitable and expressive organizational format" for LLM-oriented data, which naturally decomposes into key–value pairs associated with each sample.

The design choice to use a tabular rather than a document-oriented or graph-based representation is significant. Tabular storage makes the key-binding mechanism (discussed below) straightforward: an input_*_key simply names a column to read, and an output_*_key names a column to write. This would be more complex in a nested document model where fields might exist at arbitrary depths. The tradeoff is that hierarchical or relational structure within samples must be flattened or encoded into column values, but for the text-centric data preparation tasks DataFlow targets, the paper argues this is a reasonable simplification.

An operator's standard execution pattern follows a read–transform–write paradigm, illustrated in Figure 2 of the paper:

def run(self, storage: DataFlowStorage, **kwargs):
    inputs = storage.read()                 # 1. Read input
    results = operator_transform(inputs, **kwargs)   # 2. Transform the data
    storage.write(results)                  # 3. Write output

The **kwargs capture the key bindings—input_question_key, output_score_key, etc.—that tell the operator which columns to process. This pattern means that an operator's transformation logic (step 2) can be written generically (e.g., "evaluate the text in column X and produce a score"), and the key bindings determine which concrete columns are read and written. The same operator can process question columns in one pipeline and prompt columns in another with zero code changes—only the key names change.

The storage.step() method (visible in Figure 3) creates a checkpointed view of storage at a particular pipeline step, enabling resumption from intermediate states. If a pipeline fails at operator 7, the user can resume from the checkpoint after operator 6 rather than re-executing the entire pipeline. This is critical for LLM-driven pipelines where operators may make expensive API calls.

The LLM Serving API

LLM-driven operators do not call model APIs directly. Instead, they invoke a unified serving abstraction that exposes a single high-level entry point:

generate_from_input(user_inputs, system_prompt, json_schema)

where user_inputs is a list of prompts (typically one per sample), system_prompt is an optional system-level instruction, and json_schema is an optional output schema for structured decoding. The method returns a list of model-generated outputs.

This abstraction shields operators from several backend-specific concerns:

• Batching: Local inference engines like vLLM and SGLang support continuous batching for high throughput. The serving layer can batch requests from multiple samples before dispatching to the engine.

• Retry strategies: API-based services (ChatGPT, Gemini) may return rate-limit errors or transient failures. The serving layer implements retry logic with appropriate backoff, so individual operators don't need to handle these failure modes.

• Request routing: Different operators or pipeline stages might use different models (e.g., a strong model for generation, a cheaper model for filtering). The serving layer can route requests to the appropriate backend based on operator configuration.

• Rate limiting: For API-based services with token-per-minute or request-per-minute limits, the serving layer can throttle requests to stay within quotas.

For local inference engines, DataFlow exploits backend-level parallelism. For online API services, the framework performs multi-threaded request dispatch to maximize throughput within rate limits.

The design choice to abstract serving behind a single function call rather than exposing engine-specific APIs is motivated by two goals. First, it enables backend substitution: a pipeline developed and tested with a local vLLM deployment can be switched to a cloud API by changing one configuration line, without modifying any operator code. This makes it easy to assess how different LLM choices influence data preparation quality—a question the paper explores extensively in its experiments. Second, it enables the serving layer to evolve independently of operators. Improvements to batching strategy, retry logic, or multi-engine load balancing can be implemented once in the serving layer and benefit all operators automatically.

The Operator Programming Interface

Operators are the fundamental unit of computation in DataFlow, analogous to layers in a neural network framework. The paper defines them through a two-phase interface that separates initialization from execution:

Phase 1: __init__(self, ...) — Static configuration. Receives hyperparameters, task-specific settings, and optionally binds to an LLM serving object and a prompt template object. Rule-based and lightweight-model operators omit the LLM bindings entirely. All external dependencies are resolved at initialization, so run() can focus exclusively on data transformation.

Phase 2: run(self, storage, **kwargs) — Execution. Accepts only a DataFlowStorage object and a set of input_* and output_* keys. The naming convention is: input_<field_name>_key indicates the storage column to read as the <field_name> input, and output_<field_name>_key indicates the column name to write for the <field_name> output. For example, an evaluation operator might be called as:

op.run(
    storage=storage.step(),
    input_question_key="prompt",
    input_answer_key="response",
    output_score_key="eval_score"
)

This tells the operator: read the question from the column named "prompt", read the answer from the column named "response", evaluate them, and write the resulting score to a new column named "eval_score".

The key-binding mechanism is the central design innovation for composability. It means that:

1. Operators are column-name-agnostic. The operator's internal logic references abstract field names (question, answer, score), and the actual column names are bound at call time. This allows the same operator class to work with datasets that use different naming conventions without any code modification.

2. Data dependencies are explicit. By examining the input_* and output_* keys used in a pipeline's forward() method, the framework can construct the dependency graph automatically. If operator B reads column "eval_score" and operator A writes column "eval_score", there is a data dependency from A to B.

3. Type and key validation is possible. The compile() procedure can check that every input_* key refers to a column that exists (either from the input dataset or from a previous operator's output) before execution begins, catching missing-field errors early.

Figure 3 of the paper illustrates how key bindings enable the same operator to process differently-named columns in different pipelines:

# Pipeline 1: columns are "question" and "Answer"
run(storage, input_question_key="question", input_answer_key="Answer", output_score_key="eval_score")

# Pipeline 2: columns are "prompt" and "response"
run(storage, input_question_key="prompt", input_answer_key="response", output_score_key="score")

The operator logic is identical; only the key names differ.

The paper distinguishes two phases because it mirrors the PyTorch pattern (configuration in __init__, computation in forward) that the deep learning community has found productive. Separating configuration from execution also makes operators easier to serialize, checkpoint, and reason about: all "decisions" are made at initialization time, and run() is a pure function of its inputs (the storage state and key bindings).

The Prompt Template Interface

Prompt templates address a specific reuse problem: operators that share the same high-level logic often differ only in prompt wording. The paper gives the example of Text-to-SQL generation: synthesizing queries for SQLite versus MySQL involves identical operator logic—generate SQL, execute it, filter invalid results—but the prompts differ in minor syntax details (e.g., quoting conventions, available functions).

Rather than creating separate operator classes for each variant, DataFlow decouples prompt construction from operator implementation through a dedicated PromptTemplate interface. The mechanism works as follows:

1. During initialization, each LLM-driven operator binds to a prompt template object:

   self.op = PromptedGenerator(
       llm_serving=self.llm_serving,
       prompt_template=MySQLPromptTemplate()
   )
   

2. During execution, the operator invokes the template's build_prompt() method, which assembles task-relevant information—input fields, schema hints, contextual metadata—into a concrete prompt string.

3. The operator passes the assembled prompt to the LLM serving layer, remaining agnostic to how the prompt was constructed.

To facilitate one-to-many mappings between operators and templates, LLM-driven operators expose a unified op.ALLOWED_PROMPTS interface that enumerates all compatible prompt templates. This means a user can discover which templates work with a given operator, and the framework can validate that a chosen template is compatible before execution.

The paper explicitly positions this as enabling operators to be "flexibly reused across domains or tasks by simply switching or tuning templates, without modifying operator logic." The declarative nature of the interface—templates specify what prompt structure to use, not how to render it—means that prompt engineering becomes a separable activity from pipeline construction. A domain expert can develop a new prompt template for medical Text-to-SQL without touching the SQL Generator operator code.

The Pipeline Composition Interface

Pipelines compose operators into multi-stage workflows. The paper models them explicitly after PyTorch's nn.Module: the __init__() method handles resource allocation (storage backends, LLM serving objects) and operator instantiation, while the forward() method encodes a single pass of execution as a sequence of operator run() calls with explicit key bindings.

Figure 4 provides a minimal example:

class TranslatePipeline(PipelineABC):
    def __init__(self):
        super().__init__()
        self.storage = FileStorage(entry_file="input_data.jsonl")
        self.llm_serving = APILLMServing(api_url="<api_url>", model_name="gpt-4o")
        self.op1 = PromptedGenerator(
            llm_serving=self.llm_serving,
            system_prompt="Translate the content to Chinese"
        )
        self.op2 = PromptedGenerator(
            llm_serving=self.llm_serving,
            system_prompt="Translate the content to English"
        )

    def forward(self):
        self.op1.run(self.storage.step(), input_key='raw_content', output_key='content_CN')
        self.op2.run(self.storage.step(), input_key='raw_content', output_key='content_EN')

This example reveals several design properties:

Explicit sequencing. The order of run() calls in forward() determines execution order. For simple linear pipelines, this is a straightforward sequence. For more complex pipelines, operators can be called conditionally or in loops—forward() is Python code, so any control flow is expressible.

Implicit topology from key bindings. The dependency graph is derived from key usage, not declared separately. If op1 writes content_CN and a later operator reads content_CN, the compiler infers the dependency. This reduces the declarative overhead while preserving analyzability.

Stepwise checkpointing. The storage.step() calls indicate checkpoint boundaries. After each step(), the storage state is versioned, enabling resumption from that point if a later operator fails.

The compile() procedure performs static analysis of the operator sequence prior to execution. Specifically, it:

1. Extracts all operator dependencies and parameters.
2. Constructs the corresponding DAG—nodes are operators, edges are data dependencies derived from key bindings.
3. Conducts key-level validation: detecting missing fields (an input_* key references a column no previous operator produces), type inconsistencies, and malformed dependency chains (e.g., circular dependencies).
4. Records all operator configurations and dependency information to produce a deferred execution plan.

The design choice to use deferred construction (separating compile() from forward()) follows the Factory Method pattern. Rather than executing operators immediately during pipeline construction, compile() produces an execution plan that is later realized by forward(). The paper gives two motivations for this:

First, the compiled execution graph provides complete structural information to the DataFlow-Agent, enabling it to "surface all key- and dependency-related errors in a single report" rather than requiring multiple debugging rounds. This reduces the agent's inference cost because it doesn't need to execute the pipeline to discover structural problems.

Second, the compiled graph defines a minimal execution plan supporting advanced runtime features like checkpointing and stepwise resumption. The paper claims this "improves iterative development and large-scale pipeline construction."

Pipeline resumption is invoked via:

pipeline.forward(resume_step=1)  # resume from operator index 1 (op2)

This would re-execute op2 and subsequent operators using the previously checkpointed storage state after op1. The resumption index is zero-based, matching Python list indexing conventions.

The Operator Categorization Scheme

The paper identifies a fundamental tension in designing a unified data preparation framework: "arbitrarily many domains" produce "an open-ended set of domain-specific algorithms" that the framework must accommodate, while simultaneously needing "a stable and comprehensible operator space" so that users can navigate, discover, and reason about available operators.

The solution is a multi-dimensional categorization scheme with three orthogonal dimensions. Categories are mutually exclusive within each dimension, while dimensions themselves are parallel. This means an operator belongs to exactly one category in each dimension, and the dimensions don't interact.

Dimension 1: Modality. Operators are separated by the type of data they process: text, visual content, or document-like inputs. The paper states that operators within the same modality "share compatible input–output semantics and can interoperate, whereas operators across different modalities often cannot be composed directly." For non-text modalities (images, PDFs), modality-specific operators parse or convert raw inputs into text before downstream transformations apply. The modality classification makes this conversion flow explicit and enables the pipeline compiler to validate that modality transitions are correctly specified—flagging, for example, an operator that expects text input but receives an image column.

Dimension 2: Core vs. Domain-Specific. Core operators "reflect the fundamental design philosophy of DataFlow and serve as the conceptual basis from which most other operators can be derived." They are intentionally limited in number and relatively stable, forming the recommended entry point for new users. Domain operators, by contrast, "expand without bound as new domains, modalities, or tasks emerge." Although in principle unbounded, the domain operators included in DataFlow are "limited to those required to support the best-performing pipelines across existing domains, ensuring practical conciseness and avoiding unnecessary proliferation."

The relationship between core and domain operators is that domain operators "may wrap or specialize core operators" and their semantics "can generally be expressed by instantiating the parameters of a corresponding core operator." This means the core operator set provides a completeness guarantee: any domain operator can be understood as a parameterization of some core operator, even if the domain operator provides a more convenient or discoverable interface.

Dimension 3: Functional. Operators fall into four categories, each with naming conventions that make their behavior predictable:

• Generate. Operators that add new textual fields to existing rows (suffix: Generator) or produce additional rows (suffix: RowGenerator). Example: generating answers to questions adds an answer field to each row; generating multiple paraphrases of a question creates new rows.

• Evaluate. Operators that compute scores or labels. SampleEvaluator attaches evaluation metadata to each row (e.g., a difficulty score for a math problem). DatasetEvaluator outputs dataset-level metrics (e.g., overall diversity, coverage).

• Filter. Operators that reduce the number of rows based on criteria derived from existing fields or evaluation results. Their semantics maintain row contents apart from newly added evaluation fields. Example: removing samples where the generated answer is incorrect.

• Refine. Operators that modify specific fields within existing rows without changing the number of samples. They apply lightweight transformations such as removing URLs, normalizing whitespace, or correcting common formatting errors. Suffix: Refiner.

This functional categorization maps onto what the paper calls the generate–evaluate–filter–refine paradigm that underlies most pipeline designs in DataFlow. Figure 5 illustrates this paradigm quantitatively: when a pipeline begins with 1,000 input samples, the number of data items typically increases during generation stages (as new rows or fields are added) and then contracts as evaluation, filtering, and refinement operators remove low-quality samples or condense information.

The paper argues this categorization scheme has been "validated across the diverse domains covered in this paper, including more than six state-of-the-art data preparation pipelines, demonstrating both its representational sufficiency and scalable generality."

The DataFlow-Agent Architecture

The DataFlow-Agent is the most architecturally ambitious component of the system. It translates natural-language specifications into executable, self-correcting data preparation pipelines by orchestrating a graph-based multi-agent workflow built on LangGraph. The paper frames it as achieving "a significantly higher degree of autonomy" compared to Data-Juicer's agent, which is "largely constrained to parameterizing and sequencing a static library of pre-existing operators."

The workflow proceeds through four stages, implemented by nine specialized agents:

Stage 1: Intent Decomposition. The Intent Analysis Agent receives the user's high-level natural language query and decomposes it into a structured sequence of actionable sub-intents. For example, "clean this dataset and generate SQL queries from the questions" might decompose into: (1) filter rows with empty questions, (2) normalize question text, (3) generate SQL for each question, (4) execute SQL to verify correctness, (5) filter failed executions. Concurrently, the Data Routing Agent analyzes the provided input data to determine the task category for routing downstream. If no dataset is provided, this agent generates synthetic data placeholders to enable dry-run execution—mock data with the expected schema that allows the pipeline to be tested without real inputs.

Stage 2: Operator Synthesis. This is where DataFlow-Agent distinguishes itself from configuration-based agents. Rather than assuming all needed operators exist in the library, it implements a retrieve-reuse-synthesize strategy:

• The Operator Retrieval Agent takes specific sub-intents and employs retrieval-augmented generation (RAG) to search the DataFlow operator library for the most relevant existing operators.

• The Operator Sequencing Agent evaluates candidate operators for I/O compatibility—matching output keys of one operator to input keys of the next. If no compatible sequence exists (a functional gap), it outputs detailed specifications for new operators.

• The Operator Reuse Agent first assesses whether the requirement can be met by reusing existing code via a prompt_template. Many "new" requirements are actually existing operators with different prompting strategies.

• Only when reuse is not feasible does the Operator Synthesis Agent generate new code. It uses RAG-based few-shot learning (retrieving examples of similar operators from the codebase) to generate context-aware operator code, then performs "automated unit-level debugging until the code is executable." The debugging loop presumably involves executing the generated operator against mock data, catching exceptions, and iteratively refining the code.

• After synthesis, the Operator Reuse Agent (invoked again) assesses the generated operator code for quality and creates a reusable prompt_template, ensuring the synthesized code can be reused in future pipelines without regeneration.

Stage 3: Pipeline Assembly. The Pipeline Construction Agent takes all validated operators (both pre-existing and newly synthesized) and assembles them into a coherent DAG. It reads the operators' I/O specifications (their declared input_* and output_* key patterns), determines the topological ordering that satisfies all data dependencies, and defines the initial connections so data can flow from the source (input dataset) to the sink (final output columns). The paper notes that the pipeline is "represented as a DAG," implying that the agent must handle branching and merging (e.g., generating multiple candidate answers per question, then filtering, then selecting the best), not just linear sequences.

Stage 4: Verification. The Pipeline Verification Agent executes the assembled pipeline within a sandboxed environment on a data sample. It identifies runtime errors—missing columns, type mismatches, LLM API failures, infinite loops—and "autonomously adjust[s] connections or parameters to output a validated, error-free pipeline." This is an integration testing loop: execute, observe failures, modify the pipeline definition, re-execute, repeat until the pipeline runs to completion without errors.

Finally, the Result Reporting Agent synthesizes the workflow details and execution results, generating a comprehensive report and an executable pipeline artifact as the final solution.

The paper's experimental evaluation of the agent (Section 7.8) quantifies its capability and limitations. Eighteen user queries were constructed across three difficulty levels (Easy: explicit operator specifications; Medium: coarse goals with constraints; Hard: only a high-level requirement). The LLM-Judge score—measuring consistency of operator coverage and execution order against reference implementations—drops sharply with difficulty: from 0.92 (Easy) to 0.60 (Hard) when evaluated against text specifications, and from 0.60 (Easy) to 0.23 (Hard) when evaluated against code ground truth. The paper attributes the code-mode degradation to "the stricter nature of code-level equivalence" and notes that "under-specified queries often lead to alternative yet plausible operator compositions that diverge from a single ground-truth program"—a candid assessment that the agent's autonomous synthesis capability, while novel, remains brittle on ambiguous specifications.

The Text-to-SQL Case Study: How Abstractions Enable Domain-Specific Pipelines

Section 6.1 provides a concrete illustration of how DataFlow's abstractions compose in practice. The Text-to-SQL pipeline is built from 10 operators, 2 pipelines, and 2 supporting modules:

Operators derived from the functional categories:

• SQL Generator (Generate): Produces SQL queries from scratch using database schema context. It randomly selects from four complexity levels (simple, moderate, complex, highly complex) and provides "clear definitions and few-shot examples" for each. The database schema—including CREATE TABLE statements and randomly sampled column values—provides context. Advanced SQL functions are "randomly supplied to increase realism." The operator constrains the number of returned columns to match typical query patterns.

• SQL Augmentor (Generate): Produces augmented SQL queries from seed SQL using six augmentation strategies: Data Value Transformation, Query Structure Modification, Business Logic Alteration, Complexity Enhancement, Introduction of Advanced SQL Features, and Performance and Optimization. Categories are "randomly selected and applied through few-shot prompting."

• Question Generator (Generate): Produces natural language questions from SQL queries, categorized into four stylistic types: Tone and Formality (formal vs. colloquial), Syntactic Structure and Intent (imperative, interrogative, declarative), Information Density and Clarity (concise, descriptive, ambiguous, metaphorical), and Interaction Mode (role-playing, procedural). A target style is randomly selected.

• Chain-of-Thought Generator (Generate): Produces step-by-step reasoning traces from the question, SQL, and database schema. A CoT is considered valid only if "the execution result of its generated SQL matches that of the reference SQL on the given database"—an execution-grounded verification step.

• Prompt Generator (Generate): Synthesizes final prompts containing the natural language question, database schema, and task instructions.

• Text2SQL Consistency Filter (Filter): An LLM-based filter that analyzes whether existing question-SQL pairs are consistent. This addresses the case where a seed dataset contains misaligned examples.

• SQL Execution Filter (Filter): Filters queries from two perspectives: whether the SQL executes successfully on the target database, and whether its runtime exceeds a preset threshold (slow queries are discarded "to ensure system responsiveness").

• SQL Component Classifier (Evaluate): Assigns difficulty levels following Spider's standards: simple, moderate, hard, extra hard—based on syntactic components (column selections, aggregate functions, GROUP BY, ORDER BY, INTERSECT, nested subqueries).

• SQL Execution Classifier (Evaluate): Instructs the LLM to generate SQL $k$ times on the same input prompt and counts successful executions $n$. Difficulty is classified based on the ratio $\frac{n}{k}$. This is explicitly model-dependent: "more capable LLMs achieve higher success rates on the same task and thus are considered to have lower execution difficulty."

Supporting modules:

• Database Manager Module: Encapsulates low-level database interaction through an abstract base class DatabaseConnector with three standardized interfaces: connect_db (establishing a database connection), execute_sql (executing SQL statements and returning results), and get_schema (retrieving complete schema metadata). For each database system (MySQL, SQLite, PostgreSQL), developers subclass DatabaseConnector and implement system-specific driver invocation and error-handling logic. The module "improves processing throughput under high-concurrency workloads and abstracts schema metadata retrieval, thereby reducing the upper layers' dependency on the underlying database structure."

• Prompt Template Module: Enables the SQL Generator operator to be reused across different database systems or difficulty specifications by substituting the prompt class. The operator logic is unchanged; only the build_prompt method in the template class varies.

Two pipelines compose these operators:

SQL Generation Pipeline (generating from scratch): SQL Generator → SQL Execution Filter → Question Generator → Chain-of-Thought Generator → Prompt Generator → SQL Component Classifier → SQL Execution Classifier. This is a linear sequence producing fully labeled Text-to-SQL examples with difficulty annotations.

SQL Refinement Pipeline (augmenting from seed data): SQL Execution Filter → Text2SQL Consistency Filter → SQL Augmentor → SQL Execution Filter → Question Generator → Chain-of-Thought Generator → Prompt Generator → SQL Component Classifier → SQL Execution Classifier. This pipeline starts with seed SQL, filters low-quality and inconsistent pairs, augments the survivors, and then follows the same labeling sequence as the generation pipeline.

The case study demonstrates the paper's composability claims in action: operators developed for one pipeline (e.g., Question Generator) are reused in the other with different upstream inputs but identical invocation patterns. The key-binding mechanism ensures that the Question Generator reads whichever column contains SQL queries—whether generated from scratch or augmented from seeds—without modification.

4. Key Insights and Innovations

Innovation 1: Elevating LLM-Driven Synthesis to a First-Class Dataflow Abstraction

The paper's most fundamental conceptual move is redefining what it means to build a data preparation system for the LLM era. Prior to DataFlow, the dominant mental model—embodied in systems like NeMo Curator and Data-Juicer—treated data preparation as a curation problem: you start with raw data (web crawls, existing datasets), apply filters and heuristics to remove low-quality content, and produce a cleaned corpus. LLM-driven generation, when supported at all, was an auxiliary feature bolted onto a filtering-oriented architecture, not a first-class design principle.

DataFlow inverts this. The generate–evaluate–filter–refine paradigm (Section 4.3) places synthesis at the front of every pipeline, with filtering and refinement serving downstream as quality-control steps. This isn't a superficial reordering—it reflects a fundamentally different thesis about where data comes from. In the curation model, the LLM is a quality inspector. In DataFlow's model, the LLM is a data factory, and the framework's job is to provide the production line: prompt templates for specification, serving abstractions for execution, evaluation operators for quality control, and refinement operators for polishing.

What makes this a genuine innovation rather than an obvious extension is the architectural commitment it demands. Supporting LLM-driven synthesis as a first-class paradigm requires infrastructure that curation-oriented systems never needed: unified serving abstractions that span local engines and cloud APIs, prompt template interfaces that decouple generation logic from prompting strategy, operators that can both read and produce new rows (RowGenerator semantics), and verification workflows that execute generated outputs against ground-truth environments (like the SQL Execution Filter running queries against actual databases). The paper's nearly 200-operator library—with the majority being LLM-driven—reflects this commitment concretely.

The significance extends beyond DataFlow itself. By demonstrating that synthesis-first pipelines consistently match or exceed curation-based approaches across six domains—and that a unified 10K-sample synthesis corpus can outperform 1M samples of filtered instruction data (Table 10)—the paper makes an empirical case that the future of LLM data preparation is generative, not extractive. This shifts the research agenda from "how do we filter the web better?" toward "how do we design better synthesis operators, prompt templates, and verification strategies?"—a fundamentally different set of questions.

Innovation 2: Standardized Key-Binding Contracts Enable Operator Composability Without Configuration Overhead

The dominant approach to composability in prior LLM data preparation systems—Data-Juicer's YAML-config-based recipes, NeMo Curator's component-based pipelines—required users to explicitly declare dataflow through configuration files. This works for pre-built operators in known sequences but creates friction whenever a pipeline needs custom logic: new operators must be registered in the configuration schema, data dependencies must be manually specified, and validation of those dependencies happens at runtime (often after expensive LLM calls).

DataFlow's key-binding mechanism (Section 4.2.2) solves a subtle but pervasive problem: how do you make operators composable without requiring users to write configuration files that redundantly specify what the code already expresses? The insight is deceptively simple: by adopting Python's keyword-argument convention (input_question_key="prompt", output_score_key="eval_score"), data dependencies become implicit in the call site rather than declared separately. The compile() procedure extracts the dependency graph from these key bindings through static analysis—no separate configuration file, no manual DAG specification.

This is impactful because it bridges two worlds that prior systems kept separate. On one side, you have the flexibility of code: forward() methods can contain arbitrary Python logic, including conditionals and loops, making DataFlow pipelines expressive enough to capture the ad-hoc workflows that previously forced practitioners away from standardized frameworks. On the other side, you get the analyzability of declarative specifications: compile() produces a formal DAG with dependency validation, checkpointing, and resumption support—features that typically require config-based systems.

The comparison to PyTorch's nn.Module is more than marketing. In deep learning, nn.Module succeeded because it didn't force users to choose between flexibility (raw Tensor operations) and structure (declared layer graphs)—it provided both through a single Python-class interface. DataFlow attempts the same unification for data preparation. The __init__/forward pattern gives you the flexibility of arbitrary Python in execution (forward can branch, loop, or call external services) while compile() gives you the structural guarantees of a declarative system. Prior frameworks forced a choice: use a config-based tool for standard pipelines, or write ad-hoc scripts for custom logic. DataFlow's claim is that you can have both in one system.

The evidence that this design works at scale comes from the breadth of pipelines the paper implements without task-specific glue code—six pipelines across text, math, code, SQL, RAG, and knowledge extraction, all expressed within the same __init__/forward pattern with key-binding conventions. The Text-to-SQL case study (Section 6.1) is particularly illustrative: operators developed for the generation pipeline are reused in the refinement pipeline with different upstream inputs but identical invocation patterns, and the only change is which columns' keys are passed.

Innovation 3: Verifier-Based Difficulty Estimation for Model-Dependent Data Quality Assessment

Buried in Section 6.1.1's description of the SQL Execution Classifier operator is a conceptual move with implications beyond Text-to-SQL: using the model itself as a measurement instrument to estimate instance difficulty in a model-dependent way. The operator doesn't classify SQL queries by static syntactic features (that's the Component Classifier's job). Instead, it repeatedly queries the LLM $k$ times on the same input and counts successful executions $n$, classifying difficulty based on the ratio $\frac{n}{k}$.

This is intellectually distinctive because it inverts the standard relationship between difficulty and capability. Traditional dataset difficulty labels (like MATH's 1–5 levels or Spider's component-based classification) are model-independent—they describe properties of the question, not properties of the model-question interaction. But for data preparation purposes—deciding which examples need more synthesis effort, which need refinement, which can be used as-is—model-independent difficulty is the wrong metric. An easy question for GPT-4 might be hard for a smaller model, and the data preparation strategy should adapt accordingly.

The SQL Execution Classifier operationalizes a model-relative difficulty concept: difficulty is not a property of the SQL query but of the model's ability to generate a correct execution for it. The paper explicitly notes this: "execution difficulty is model-dependent: more capable LLMs achieve higher success rates on the same task and thus are considered to have lower execution difficulty." This reframing connects data preparation to the test-time compute scaling literature, where prompt difficulty—defined as pass@1 under the base model—is similarly model-relative and is the key variable for allocating inference budgets.

The broader significance is that this operator pattern—repeated model invocation to estimate capability boundaries—can be applied to any generation task with a verifiable correctness criterion. Math problems (execute and check the answer), code generation (run unit tests), and agentic tasks (check whether the goal was achieved) all admit similar execution-based difficulty estimation. The paper doesn't develop this into a general theory, but the operator's existence in the library provides a template for doing so.

Innovation 4: The Agent Can Synthesize Novel Operators, Not Just Configure Existing Ones

The field's prior approach to "agentic" data preparation—represented by Data-Juicer's recommendation agent—was fundamentally a configuration selection problem: given a library of pre-built operators, select the right ones and parameterize them correctly. The paper characterizes this as "largely constrained to parameterizing and sequencing a static library of pre-existing operators" (Section 5.3).

DataFlow-Agent makes a qualitatively different claim: it can synthesize entirely new operators when existing ones don't match the user's intent, using a retrieve-reuse-synthesize strategy with sandboxed debugging. This is not an incremental improvement in agent capability—it changes the agent's relationship to the operator library from consumer to producer. The agent doesn't fail when a required operator is missing; it writes one, tests it, and packages it with a reusable prompt template for future use.

The conceptual move is recognizing that the framework's abstractions—standardized operator interfaces with explicit I/O contracts, prompt templates with parameterized slots, key-binding conventions—create a sufficiently constrained synthesis target that code generation becomes feasible. The agent doesn't need to generate arbitrary Python; it needs to generate a class with an __init__ and run method that respects the key-binding convention and optionally invokes self.llm_serving through a prompt template. The structure of the target is known, which makes few-shot generation from existing operator examples tractable.

The significance of this innovation is tempered by the experimental results (Section 7.8), which show sharp degradation on under-specified queries: LLM-Judge scores drop from 0.92 to 0.60 (text-spec evaluation) and 0.60 to 0.23 (code-GT evaluation) as descriptions become less explicit. The paper is candid about this limitation, noting that "under-specified queries often lead to alternative yet plausible operator compositions that diverge from a single ground-truth program." This is less a failure of the agent and more a fundamental challenge in specification: if the user can't articulate what they want precisely, no agent can read their mind. The Hard-tier results (0.23 on code-GT) suggest the agent's synthesis capability is currently limited to cases where the user provides enough detail to constrain the operator's behavior—a significant caveat to the "autonomous pipeline construction" narrative.

Nevertheless, even the Easy and Medium-tier results demonstrate a capability—novel operator synthesis with automated debugging—that prior systems didn't attempt. This establishes a research direction rather than a solved problem: as LLM code generation improves, the agent's synthesis capability should improve without changes to the DataFlow framework, since the synthesis target (standardized operator interfaces) remains constant.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across six distinct domains, each using different datasets. For text data preparation, pretraining experiments use a 100B-token subset of SlimPajama-627B (Section 7.1.1). SFT experiments use WizardLM and Alpaca datasets, plus DataFlow-synthesized DataFlow-SFT-15K and DataFlow-Chat-15K, with evaluation on Math (MATH, GSM8K, AIME24, Minerva, Olympiad), Code (HumanEval, MBPP), and Knowledge (MMLU, C-EVAL) benchmarks. Conversation evaluation adds TopDial, Light, AlpacaEval, and Arena-Hard. For math reasoning (Section 7.2), training uses 10K subsets from NuminaMath-derived synthesis evaluated on GSM8K, MATH, AMC23, Olympiad, Gaokao24-Mix, Minerva, and AIME 2024/2025. For code (Section 7.3), training uses subsets of Ling-Coder-SFT evaluated on BigCodeBench, LiveCodeBench (v6), CruxEval (Input and Output), and HumanEval+. For Text-to-SQL (Section 7.4), training uses Spider-train, BIRD-train, and EHRSQL-train evaluated on Spider (dev and test), BIRD dev, EHRSQL, Spider-DK, Spider-Syn, and Spider-Realistic. For agentic RAG (Section 7.5), training uses DataFlow-synthesized multi-hop questions evaluated on HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle. For knowledge extraction (Section 7.6), training uses 140M tokens of raw medical data from MedQA Books, StatPearls, and clinical guidelines, evaluated on PubMedQA, Covert, and PubHealth. For unified multi-domain (Section 7.7), DataFlow-Instruct-10K combines 3K math, 2K code, and 5K text samples, with baselines constructed from Infinity-Instruct.

• Base model(s). The paper uses a heterogeneous set of base models across experiments, chosen to demonstrate DataFlow's generality across model families and scales. Text data preparation: Qwen2.5-0.5B trained from scratch for pretraining, Qwen2.5-7B-Base for SFT and conversation experiments (Section 7.1.1). Math reasoning: Qwen2.5-32B-Instruct fine-tuned for 1–2 epochs (Section 7.2.1). Code: Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct (Section 7.3.1). Text-to-SQL: Meta-Llama-3.1-8B-Instruct and Qwen2.5-Coder-7B-Instruct, with additional zero-shot evaluation of GPT-4o-mini, GPT-4-Turbo, GPT-4o, DeepSeek-Coder-7B-Instruct, Qwen2.5-7B-Instruct, OpenCoder-8B-Instruct, and Granite variants (Section 7.4.1). Agentic RAG: Qwen2.5-7B-Instruct trained with GRPO reinforcement learning via the ReCall framework (Section 7.5.1). Knowledge extraction: Qwen2.5-7B-Instruct fine-tuned via SFT for 37,500 steps over five epochs (Section 7.6.1). Unified multi-domain: Qwen2-7B-Base and Qwen2.5-7B-Base, with comparisons to their Instruct counterparts (Section 7.7.1). This diversity is a deliberate design choice: the paper aims to show that DataFlow-generated data improves performance across model scales (0.5B to 32B), model families (Qwen2, Qwen2.5, Llama-3.1), and model types (base, instruct, code-specialized).

• Metrics. The paper uses domain-specific accuracy metrics. For math reasoning, it reports Exact Match (%) on each benchmark individually and aggregates into a Math-Avg. For code, it reports pass@1 (%) on each benchmark and averages into Code-Avg. For Text-to-SQL, it reports execution accuracy (Ex) under two decoding strategies: greedy (Gre, temperature 0) and majority voting (Maj, 8 samples at temperature 0.8, selecting the most frequent execution result). Benchmarks are reported individually with a final Average across all test sets. For agentic RAG, it reports Exact Match (%) on each benchmark and computes out-of-distribution (OOD) averages excluding each training dataset's in-domain test set. For knowledge extraction, it reports accuracy (%) on each medical QA benchmark. For unified multi-domain, it averages over Math, Code, and Knowledge benchmarks separately (Table 10 and Table 11). For pretraining, it reports accuracy across ARC-C, ARC-E, MMLU, HellaSwag, WinoGrande, and Gaokao-MathQA, with an overall average (Table 2). For conversation, it reports both domain-specific metrics (TopDial, Light, averaged) and general benchmarks (MMLU, AlpacaEval, Arena-Hard, averaged) in Table 4.

• Baselines. The paper employs a multi-tiered baseline strategy specific to each domain. For text pretraining (Table 2): Random-30B (random 30B-token subset), FineWeb-Edu-30B (educational filtering from FineWeb-Edu [50]), Qurating-30B (Qurating filters [64] with thresholds: educational_value >= 7.5, facts_and_trivia >= 4.0, required_expertise >= 5.0, writing_style >= 1.0). For text SFT (Table 3): Alpaca(random) and Alpaca(filtered), WizardLM(random) and WizardLM(filtered), each with 5K samples, plus DataFlow-SFT-15K in both random and filtered variants. For conversation (Table 4): ShareGPT-15K, UltraChat-15K, and the base Qwen2.5-7B without fine-tuning. For math reasoning (Table 5): the base Qwen2.5-32B-Instruct, SYNTHETIC-1-10K [43], and Open-R1-10K [28]. For code (Table 6): the base instruct models, Code Alpaca-1K [5], and Self-OSS-Instruct-SC2-Exec-Filter-50K(1K) [63]. For Text-to-SQL (Table 7): multiple zero-shot LLMs (GPT-4o-mini, GPT-4-Turbo, GPT-4o, DeepSeek-Coder-7B-Instruct, Qwen2.5-Coder-7B-Instruct, Qwen2.5-7B-Instruct, OpenCoder-8B-Instruct, Meta-Llama-3.1-8B-Instruct, Granite-8B-Code-Instruct, Granite-3.1-8B-Instruct), plus models fine-tuned on SynSQL at three scales (50K, 90K, 2.5M) [37] and on Spider+BIRD+DataFlow-Text2SQL-90K (a hybrid baseline combining human datasets with synthetic data). For agentic RAG (Table 8): the base Qwen2.5-7B-Instruct, plus models trained on HotpotQA-10K [68] (1–3 epochs), Musique-20K [58] (1 epoch), and 2Wiki-30K [25] (2 epochs). For knowledge extraction (Table 9): CoT (zero-shot Chain-of-Thought prompting), RAG (retrieval-augmented generation with top-k=10 using medcpt-query-encoder and medcpt-article-encoder). For unified multi-domain (Tables 10 and 11): the base models without fine-tuning, Inf-10K (random 10K subset of Infinity-Instruct [39]), and Inf-1M (random 1M subset), plus the corresponding Instruct models as upper-bound references.

• Generation budget / compute accounting. The paper does not standardize on a single compute metric across experiments because the relevant comparison varies by domain. For pretraining (Section 7.1.1), budgets are measured in tokens: each method uses 30B tokens, and a Qwen2.5-0.5B model is trained from scratch. For SFT (Sections 7.1–7.7), budgets are measured in number of training samples: 5K, 10K, 15K, 20K, or 90K depending on the experiment. All competing methods use the same sample count within each experiment. For Text-to-SQL ablation on scale (Table 7), DataFlow-Text2SQL-50K and DataFlow-Text2SQL-90K are compared against SynSQL at matching sizes (SynSQL-50K, SynSQL-90K) and against SynSQL-2.5M to test data efficiency. For unified multi-domain (Section 7.7), DataFlow-Instruct-10K is compared against Inf-10K (matched 10K) and Inf-1M (100× larger) to test whether quality can compensate for quantity. For agentic RAG (Table 8), effective training scales vary by dataset (HotpotQA-10K, Musique-20K, 2Wiki-30K, DF-AgenticRAG-10K), and multi-epoch training is used to match effective scale where possible (e.g., 2 epochs of 10K = 20K effective samples). For code (Table 6), comparisons are at identical sample counts (1K, 5K, 10K). The paper's approach to compute accounting is therefore sample-matched rather than FLOPs-matched—it asks "given the same number of training examples, does DataFlow-generated data produce better models?" rather than "given the same generation budget, does DataFlow produce better data?" The cost of producing the DataFlow datasets (LLM API calls for generation, evaluation, and refinement) is not accounted for in any comparison, which is a significant omission for claims about practical efficiency.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests for any of its main results. For agentic orchestration (Section 7.8), difficulty-level bins (Easy, Medium, Hard) are used to stratify evaluation, and an external LLM-as-judge provides continuous scores in [0, 1], but no statistical protocol is described. For Text-to-SQL (Section 7.4), two decoding strategies (greedy and majority voting) provide some measure of robustness, but results are reported as point estimates without error bars. For math reasoning (Section 7.2), results are reported for both 1-epoch and 2-epoch training, serving as informal robustness checks, but no claim is made about statistical significance of differences. The evaluation protocol is comprehensive in breadth (many benchmarks) but lacks statistical rigor: small test sets (e.g., AIME24 with 30 questions) can produce large variance in point estimates, and several reported "improvements" of 1–3 percentage points may not be statistically distinguishable from noise given the sample sizes. This is a notable limitation for a paper making specific quantitative claims about superiority.

Main Quantitative Results

Text Data Preparation (Section 7.1)

Pretraining filtering (Table 2). DataFlow's multi-filter intersection (DataFlow-30B) achieves the highest average score of 35.69 across six general benchmarks, compared to Random-30B (35.26), FineWeb-Edu-30B (35.57), and Qurating-30B (35.02). The absolute margin over the strongest baseline (FineWeb-Edu) is only 0.12 points—effectively within noise. Individual benchmark results show mixed patterns: DataFlow-30B achieves the best scores on ARC-E (45.58 vs. 45.41 for FineWeb-Edu-30B) and Gaokao-MathQA (27.35, tied with Random), but underperforms FineWeb-Edu-30B on ARC-C (25.51 vs. 26.45) and HellaSwag (37.58 vs. 38.06). The improvements are small and inconsistent, suggesting that at the 30B-token scale, the choice of filtering strategy has limited impact on a 0.5B-parameter model's performance—a finding the paper does not discuss critically.

SFT data filtering (Table 3). DataFlow's filtering pipeline consistently improves performance over random sampling across Alpaca and WizardLM datasets, though with varying magnitude. For WizardLM, filtering raises the Math average from 39.9 to 44.8 (+4.9 points), Code average from 78.8 to 78.9 (+0.1), and Knowledge average from 75.5 to 75.8 (+0.3). For Alpaca, the gains are smaller: Math from 37.3 to 39.8 (+2.5), Code from 73.6 to 74.8 (+1.2), Knowledge unchanged at 75.9. The more striking result is that DataFlow-SFT-15K (random) already achieves a Math average of 49.3, substantially outperforming filtered WizardLM (44.8) and Alpaca (39.8), with the filtered version adding only 0.4 points (to 49.7). This suggests DataFlow's synthesis pipeline produces inherently higher-quality data than human-constructed instruction datasets, and that aggressive filtering provides diminishing returns when the base data quality is already high—an important insight for practitioners deciding whether to invest in better generation or better filtering.

Conversation synthesis (Table 4). DataFlow-Chat-15K achieves the highest scores on both domain-specific benchmarks (TopDial: 7.98, Light: 8.10, Avg: 8.04) and general benchmarks (MMLU: 73.41, AlpacaEval: 10.11, Arena-Hard: 1.10, Avg: 28.21). The AlpacaEval improvement is particularly large: 10.11 vs. 7.05 for the base Qwen2.5-7B model—a 43% relative improvement. ShareGPT-15K and UltraChat-15K both degrade general benchmark performance relative to the base model (Avg dropping from 26.36 to 26.03 and 25.91 respectively), while DataFlow-Chat-15K improves it to 28.21. This asymmetric result—synthetic DataFlow data improves performance while human-collected conversation data hurts it—is the strongest evidence in the paper that DataFlow's synthesis-first approach produces higher-quality supervision than commonly used human datasets.

Math Reasoning Data Preparation (Section 7.2)

Table 5, 2-epoch results. DataFlow-Reasoning-10K achieves the highest overall average of 55.7 after 2 epochs of fine-tuning, compared to Open-R1-10K (54.2) and SYNTHETIC-1-10K (54.0), with the base Qwen2.5-32B-Instruct at 46.95. The absolute margin over Open-R1 is 1.5 points. The pattern across benchmarks is revealing: DataFlow-Reasoning-10K excels on Gaokao24-Mix (42.9 vs. 20.9 for Open-R1 and 24.2 for SYNTHETIC-1—a massive 22-point gap on this benchmark) and Olympiad (45.2 vs. 44.1 and 45.0), but underperforms Open-R1-10K on AMC23 (75.0 vs. 80.0) and AIME benchmarks (AIME24@32: 45.4 vs. 51.0; AIME25@32: 40.0 vs. 40.7). The 1-epoch results show an even starker pattern: DataFlow-Reasoning-10K achieves 51.6 average vs. 48.7 for Open-R1 and 46.6 for SYNTHETIC-1 at 1 epoch, meaning the gap narrows with additional training—from +2.9 at 1 epoch to +1.5 at 2 epochs. This suggests DataFlow's data provides stronger initial learning signal but the advantage partially diminishes as models overfit to the 10K-sample scale.

The Gaokao24-Mix result deserves scrutiny: a +22.0 point gap over Open-R1 is implausibly large for a 10K-sample dataset difference and may indicate that DataFlow's synthesis pipeline inadvertently includes Gaokao-like problems (through the NuminaMath seed expansion) while Open-R1's data distribution does not. The paper does not investigate data contamination or distribution overlap as potential explanations.

Code Data Preparation (Section 7.3)

Table 6, Qwen2.5-14B-Instruct results. DataFlow-Code-10K achieves the highest overall average of 51.0, compared to Code Alpaca-1K (47.3), Self-OSS (46.0), and the base model (48.4). The average improvement over the base model is +2.6 points, driven primarily by gains on BigCodeBench (37.5 → 41.9, +4.4) and HumanEval+ (74.4 → 76.2, +1.8). LiveCodeBench and CruxEval show more modest improvements. Notably, DataFlow-Code-1K already achieves a 50.9 average at only 1K samples—essentially matching the 10K result (51.0)—suggesting that even a small amount of DataFlow-synthesized code data is sufficient, and that the scaling curve from 1K to 10K is very flat. This has practical implications: users may not need to generate large code datasets to capture most of the benefit.

For the 7B model, the scaling pattern is clearer: DataFlow-Code-1K (45.4), -5K (45.9), and -10K (46.2) show monotonic but diminishing improvement. Code Alpaca-1K (42.1) and Self-OSS (43.2) both underperform the base model (44.0), meaning these commonly used code instruction datasets actually harm performance. This is consistent with the conversation-domain finding—human-collected or web-scraped instruction data can be detrimental, while DataFlow's pipeline-refined synthetic data provides clean enough supervision to improve even a strong instruct model.

Text-to-SQL Data Preparation (Section 7.4)

Table 7, Qwen2.5-Coder-7B-Instruct results under greedy decoding. DataFlow-Text2SQL-90K achieves an average execution accuracy of 71.6, compared to SynSQL-90K (65.5), SynSQL-2.5M (70.0), and the base model (61.2). The margin over SynSQL-90K at matched scale is +6.1 points. Remarkably, DataFlow-Text2SQL-90K outperforms SynSQL-2.5M (71.6 vs. 70.0) while using only 3.6% as many training examples (90K vs. 2.5M). The largest absolute gains appear on EHRSQL (24.3 → 56.1, +31.8) and BIRD-dev (50.9 → 59.2, +8.3), while Spider-test improves from 77.1 to 85.0 (+7.9).

The majority voting results (Maj columns) tell a more nuanced story. Under majority voting, DataFlow-Text2SQL-90K achieves 74.0 average vs. SynSQL-2.5M at 71.6—a gap of +2.4 that narrows from the +6.2 gap under greedy decoding. This suggests DataFlow's data provides more benefit for deterministic decoding (which relies on the model learning precise SQL generation strategies) than for ensembled decoding (where sampling diversity matters more). Additionally, the Spider+BIRD+DataFlow-Text2SQL-90K hybrid baseline (69.6 greedy, 73.3 majority) underperforms pure DataFlow-Text2SQL-90K (71.6 greedy, 74.0 majority), indicating that mixing human-curated data with synthetic data can be counterproductive—a finding with implications for practitioners who might assume "more data is always better."

For Meta-Llama-3.1-8B-Instruct, the same pattern holds: DataFlow-Text2SQL-90K achieves 65.3 greedy (68.2 majority) vs. SynSQL-90K at 59.2 (64.0) and SynSQL-2.5M at 63.4 (66.1). The cross-model consistency strengthens the claim that DataFlow's synthesis quality, not model-specific factors, drives the improvement.

Agentic RAG Data Preparation (Section 7.5)

Table 8. This experiment is structured differently from the others: rather than comparing synthetic-vs-synthetic at matched scale, it compares DataFlow-AgenticRAG-10K against established human-constructed multi-hop datasets, using OOD averages that exclude each training dataset's in-domain test set for fair comparison. The key results:

• vs. HotpotQA-10K (3 epochs): DF-AgenticRAG achieves OOD average of 37.4 vs. HotpotQA's 36.4, a +1.0 point advantage despite being entirely synthetic.
• vs. Musique-20K (1 epoch): DF-AgenticRAG (2 epochs, effective 20K) achieves OOD of 43.6 vs. Musique's 42.4, a +1.2 point advantage.
• vs. 2Wiki-30K (2 epochs): DF-AgenticRAG (3 epochs, effective 30K) achieves OOD of 36.4 vs. 2Wiki's 33.8, a +2.6 point advantage.

The in-domain results (including the training dataset's own test set) tell an additional story: DF-AgenticRAG-10K (3 epochs) achieves HotpotQA: 42.6, 2Wiki: 45.5, Musique: 20.2, Bamboogle: 46.4, with an overall average of 38.7. This is competitive with or exceeds HotpotQA-10K (3 epochs, Avg: 38.6) and Musique-20K (Avg: 36.6), but trails 2Wiki-30K (Avg: 39.1). The fact that DF-AgenticRAG-10K achieves this at only 10K samples (vs. 20K–30K for the baselines, before multi-epoch adjustment) demonstrates strong data efficiency.

Notably, Bamboogle—a dataset specifically designed to test compositional generalization—shows the largest gains: 46.4 for DF-AgenticRAG (3 epochs) vs. 40.8 for HotpotQA (3 epochs) and 42.4 for 2Wiki (2 epochs). This suggests DataFlow's synthetic multi-hop questions produce better generalization to out-of-distribution compositional structures than human-authored questions, possibly because the pipeline's random document selection and variety of question styles produce more diverse reasoning patterns.

Knowledge Extraction (Section 7.6)

Table 9. The SFT model trained on DataFlow-Knowledge synthetic data achieves PubMedQA: 53.40%, Covert: 68.33%, PubHealth: 40.86%. The margins over baselines are dramatic: CoT achieves only 36.40%/48.33%/29.00%, and RAG achieves 43.33%/17.55%/19.60%. The RAG baseline's poor performance on Covert (17.55% vs. 48.33% for CoT) is striking—retrieval actually harms performance on clinical knowledge questions compared to zero-shot reasoning—while DataFlow's SFT model more than triples the RAG score (17.55% → 68.33%). This demonstrates that structured, pipeline-verified synthetic QA data provides supervision that neither zero-shot prompting nor retrieval-augmented generation can approximate.

However, the baseline comparison is weak in one important respect: the paper does not compare against SFT on existing medical QA datasets (e.g., MedQA, MedMCQA), which would establish whether the DataFlow pipeline produces better training data than simply using available human-annotated medical corpora. The comparison against CoT and RAG establishes that DataFlow-Knowledge is better than no fine-tuning, but does not establish superiority over alternative fine-tuning data sources.

Unified Multi-Domain Data Preparation (Section 7.7)

Tables 10 and 11. This is the paper's most ambitious experiment: can a single 10K-sample dataset spanning math, code, and text, produced entirely through DataFlow pipelines, compete with much larger generic instruction datasets?

Math (Table 10). For Qwen2.5-7B-Base, DataFlow-Instruct-10K achieves a Math-Avg of 46.7, compared to Inf-10K (22.6) and Inf-1M (33.3). The 10K DataFlow dataset outperforms the 1M Infinity-Instruct dataset by 13.4 points on math average. The gap to Qwen2.5-7B-Instruct is only 3.1 points (46.7 vs. 49.8), meaning DataFlow's 10K synthetic data recovers most of the benefit of full instruction tuning on math. For the weaker Qwen2-7B-Base, DataFlow-Instruct-10K achieves 32.4 vs. Inf-1M at 27.9 (+4.5) and approaches Qwen2-7B-Instruct at 34.0 (-1.6). The GSM8K result for Qwen2.5-7B is particularly revealing: DataFlow-Instruct-10K reaches 88.2 vs. the base model's 67.1 (+21.1), substantially outperforming Inf-1M (82.0) and approaching the Instruct model (92.4). This suggests DataFlow's math synthesis produces supervision specifically effective for grade-school-style word problems.

A concerning anomaly: Qwen2.5-7B-Base + Inf-10K achieves only 22.6 Math-Avg, far below the base model's 37.1—meaning training on 10K Infinity-Instruct samples catastrophically degrades math performance. Even Inf-1M (33.3) doesn't recover to base-model levels. This implies Infinity-Instruct's math distribution is either low-quality or actively harmful when mixed with the model's pretrained knowledge. The paper does not investigate this degradation.

Code (Table 11). Results are more compressed. For Qwen2.5-7B-Base, DataFlow-Instruct-10K achieves Code-Avg of 78.6 vs. Inf-10K (77.6) and Inf-1M (78.0)—margins of only +1.0 and +0.6 points respectively. The base model already achieves 76.5, so improvements are modest across all methods. This suggests code capability is largely determined by pretraining and is relatively insensitive to small-scale SFT data, regardless of quality. For Qwen2-7B-Base, the pattern reverses: DataFlow-Instruct-10K (66.2) slightly underperforms Inf-10K (67.8) and Inf-1M (68.2), suggesting the DataFlow code pipeline may be better tuned for Qwen2.5's pretraining distribution than Qwen2's.

Knowledge (Table 11). All methods cluster tightly. For Qwen2.5-7B-Base, DataFlow-Instruct-10K achieves Knowledge-Avg of 76.2 vs. Inf-10K (75.8), Inf-1M (75.8), and the base model (76.0). The maximum difference between any method is 0.4 points. This strongly suggests that MMLU and C-Eval performance is dominated by the base model's pretrained knowledge and is minimally affected by 10K-scale SFT data—a null result that the paper does not highlight but that is informative for practitioners.

Overall assessment of Tables 10–11. The unified multi-domain experiment convincingly demonstrates that DataFlow's domain-specialized synthesis produces dramatically better math supervision than generic large-scale instruction data (Infinity-Instruct), but the advantage largely vanishes for code and knowledge. The paper's abstract claims that "a unified 10K-sample dataset produced by DataFlow enables base models to surpass counterparts trained on 1M Infinity-Instruct data" is technically true for math but misleading when generalized—the overall average across all three domains would show a much smaller gap, driven almost entirely by math. Reporting domain-aggregated results would provide a more honest picture.

Agentic Orchestration (Section 7.8)

Table 12. This experiment evaluates DataFlow-Agent's pipeline construction capability rather than downstream model performance. Under text-spec evaluation (comparing generated pipeline structure against task specifications), the LLM-Judge scores are: Easy: 0.92, Medium: 0.86, Hard: 0.60, Overall: 0.80. Under code-GT evaluation (comparing generated pipeline implementations against reference code), scores drop to: Easy: 0.60, Medium: 0.59, Hard: 0.23, Overall: 0.49.

The Easy-tier text-spec score of 0.92 indicates near-perfect pipeline construction when the user provides explicit operator specifications and processing steps. The Medium-tier score of 0.86 suggests the agent handles coarse goals reasonably well. The Hard-tier drop to 0.60—and the code-GT collapse to 0.23—reveals the current system's fundamental limitation: when the specification is ambiguous, the agent can produce "alternative yet plausible operator compositions that diverge from a single ground-truth program." The paper's candid acknowledgment of this limitation is a strength of the evaluation design.

The gap between text-spec and code-GT evaluation (0.80 vs. 0.49 overall) reflects that matching a natural-language description is substantially easier than matching a specific implementation. A pipeline might satisfy the stated requirements while using different operators or different ordering than the reference implementation. Whether this matters depends on the use case: if the goal is functional correctness (does the pipeline produce the right output?), text-spec alignment may be sufficient; if the goal is exact reproducibility of a known-good pipeline, code-GT alignment is needed.

The experiment uses only 6 pipelines × 3 difficulty levels = 18 queries—a small sample that makes the reported scores sensitive to the specific pipelines and descriptions chosen. No information is provided about which pipelines were selected or how descriptions were constructed, limiting reproducibility.

Ablation Studies and Robustness Checks

Data scale ablation for Text-to-SQL (Table 7). DataFlow-Text2SQL-50K vs. -90K: for Qwen2.5-Coder-7B-Instruct under greedy decoding, 50K achieves 67.0 average and 90K achieves 71.6—a substantial +4.6 point gain from 40K additional samples, suggesting the Text-to-SQL pipeline benefits meaningfully from data scaling. Under majority voting, the gap is smaller: 71.4 vs. 74.0 (+2.6). For Meta-Llama-3.1-8B-Instruct, the scaling is less pronounced: 60.2 → 65.3 (+5.1 greedy), 65.7 → 68.2 (+2.5 majority). This indicates that the optimal data scale depends on both the base model and the decoding strategy, and that DataFlow's Text-to-SQL pipeline has not saturated at 90K samples.

Filtering ablation for SFT data (Table 3). Comparing random vs. filtered sampling for Alpaca and WizardLM at 5K scale: filtering provides substantial gains for WizardLM on Math (39.9 → 44.8, +4.9) but minimal gains for Alpaca on Math (37.3 → 39.8, +2.5) and negligible gains on Code and Knowledge for both datasets. This suggests that some instruction datasets (WizardLM) contain a meaningful fraction of low-quality math examples that filtering removes, while others (Alpaca) are more uniformly mediocre. The DataFlow-SFT-15K results show minimal filtering benefit (49.3 → 49.7, +0.4), suggesting DataFlow's synthesis pipeline produces data that is already clean.

Model scale ablation for code (Table 6). Comparing Qwen2.5-7B-Instruct vs. -14B-Instruct: DataFlow-Code-1K improves the 7B model from 44.0 to 45.4 (+1.4) and the 14B model from 48.4 to 50.9 (+2.5). The larger absolute gain for the 14B model suggests that higher-capacity models extract more value from the same synthetic data, possibly because they can better leverage the structured reasoning cues in DataFlow's code instructions.

Training epochs ablation for math reasoning (Table 5). DataFlow-Reasoning-10K improves from 51.6 (1 epoch) to 55.7 (2 epochs, +4.1), compared to Open-R1-10K from 48.7 to 54.2 (+5.5) and SYNTHETIC-1-10K from 46.6 to 54.0 (+7.4). DataFlow's data provides the strongest 1-epoch performance but the smallest incremental gain from a second epoch, suggesting it extracts more learning per sample in the first pass. The different slopes also imply that DataFlow's data might saturate earlier, and that at higher epoch counts, Open-R1 or SYNTHETIC-1 might close or reverse the gap—a possibility not explored.

Decoding strategy ablation for Text-to-SQL (Table 7, Gre vs. Maj columns). Majority voting (8 samples, temperature 0.8) consistently improves over greedy decoding across all models and training configurations, but the margin varies. For Qwen2.5-Coder-7B-Instruct + DataFlow-Text2SQL-90K, majority voting provides +2.4 points (71.6 → 74.0). For SynSQL-2.5M, the gain is +1.6 (70.0 → 71.6). DataFlow-trained models benefit more from ensembling, possibly because the synthetic data encourages learning diverse SQL generation strategies that complement each other under sampling.

Conversation dataset ablation (Table 4). DataFlow-Chat-15K is compared against ShareGPT-15K and UltraChat-15K, all at 15K samples. ShareGPT-15K degrades general benchmark performance from 26.36 (base model) to 26.03, and UltraChat-15K degrades it to 25.91. DataFlow-Chat-15K improves it to 28.21. This is not strictly an ablation (the datasets differ in provenance, not a single controlled variable), but it demonstrates that DataFlow's conversation synthesis produces qualitatively different—and more beneficial—training data than commonly used human conversation datasets.

Synthesis pipeline ablation for unified multi-domain (Tables 10–11, comparing DataFlow-Instruct-10K vs. Inf-10K vs. Inf-1M). This compares domain-specialized synthesis (DataFlow) against generic instruction data (Infinity-Instruct) at matched 10K scale and at 100× larger scale. The finding that DataFlow-Instruct-10K dramatically outperforms Inf-10K on math but is comparable on code and knowledge suggests that domain-specialized synthesis is most valuable for tasks requiring structured reasoning (math, SQL, multi-hop QA) and less critical for tasks where the base model's pretrained knowledge dominates (MMLU, C-EVAL). This is a non-obvious interaction between data synthesis strategy and task type that the paper doesn't explore in depth.

Negative results. Several findings qualify as informative negative results: (1) Code Alpaca-1K and Self-OSS degrade code performance relative to the base instruct model (Table 6), demonstrating that not all instruction data is beneficial. (2) Inf-10K catastrophically degrades Qwen2.5-7B-Base math performance from 37.1 to 22.6 (Table 10), showing that mixing low-quality math data with a strong base model can be actively harmful. (3) For knowledge benchmarks, all 10K-scale fine-tuning methods produce near-identical results to the base model (Table 11), a null result indicating that MMLU/C-EVAL are insensitive to small-scale SFT. (4) The DataFlow-Agent's Hard-tier code-GT score of 0.23 (Table 12) demonstrates that autonomous operator synthesis from ambiguous specifications remains largely unsolved.

Critical Assessment

Claim: "DataFlow consistently improves downstream LLM performance." The evidence supports this claim broadly across six domains, but the magnitude and reliability of improvement vary considerably by domain and metric. The strongest results are in Text-to-SQL (DataFlow-Text2SQL-90K outperforms SynSQL-2.5M by +1.6 points greedy on Qwen2.5-Coder-7B-Instruct while using 3.6% of the data—Table 7) and math reasoning (DataFlow-Reasoning-10K achieves 55.7 vs. 54.2 for Open-R1 at 2 epochs—Table 5). The weakest results are in pretraining text filtering (Table 2: DataFlow-30B leads by only 0.12 points over FineWeb-Edu-30B, an insignificant margin) and knowledge benchmarks (Table 11: all methods within 0.4 points). The paper's abstract emphasizes the strong results without contextualizing the weak ones, creating an impression of uniform superiority that the detailed results don't fully support. A fairer characterization: DataFlow-generated data substantially outperforms baselines on tasks requiring structured reasoning (math, SQL, multi-hop QA), moderately outperforms on code generation, and is roughly equivalent on pretraining filtering and knowledge recall.

Claim: "Our math, code, and text pipelines outperform curated human datasets and specialized synthetic baselines." This claim is partially supported but requires qualification. For math reasoning, DataFlow-Reasoning-10K does outperform Open-R1-10K (+1.5 at 2 epochs) and SYNTHETIC-1-10K (+1.7), but the margin is modest relative to the base model's variance, and the claim of "1–3 point gains on MATH, GSM8K, and AIME" is imprecise: Table 5 shows DataFlow-Reasoning-10K actually underperforms Open-R1 on AIME24@32 (45.4 vs. 51.0) and roughly ties on AIME25@32 (40.0 vs. 40.7), with gains concentrated on GSM8K (94.4 vs. 93.9) and MATH (76.6 vs. 77.2—actually worse). For code, "+7% average improvements" (from the abstract) translates to aggregate averages in Table 6: Qwen2.5-14B + DataFlow-Code-10K achieves 51.0 vs. base 48.4, which is +2.6 absolute points, not +7% (unless the paper is computing relative improvement: 2.6/48.4 ≈ 5.4%, which rounds to neither 7% nor matches the "over 7%" phrasing). The Code Alpaca and Self-OSS baselines underperform the base model, so the improvement over those baselines is larger, but the paper's claim is ambiguous about which baseline the percentages reference.

Claim: "A unified 10K-sample dataset produced by DataFlow enables base models to surpass counterparts trained on 1M Infinity-Instruct data." This claim is true for math (46.7 vs. 33.3 for Qwen2.5-7B-Base, Table 10) but false or unverified for other domains. On code, DataFlow-Instruct-10K (78.6) and Inf-1M (78.0) are within 0.6 points (Table 11). On knowledge, they are within 0.4 points. The paper doesn't report an overall aggregate, but computing a simple average across Math-Avg, Code-Avg, and Knowledge-Avg for Qwen2.5-7B gives: DataFlow-Instruct-10K = (46.7 + 78.6 + 76.2) / 3 ≈ 67.2; Inf-1M = (33.3 + 78.0 + 75.8) / 3 ≈ 62.4. The 4.8-point overall advantage is driven almost entirely by the math domain. Claiming that the 10K dataset "surpasses" the 1M dataset without specifying that the advantage is domain-specific is misleading. A domain-disaggregated claim would be more accurate and more useful to practitioners.

Missing experiments that would strengthen the paper:

1. Statistical significance testing. None of the experiments report confidence intervals, standard deviations, or p-values. Many reported "improvements" are small (1–3 points on benchmarks with 500–5000 test examples), and without error bars, it's impossible to distinguish signal from noise. This is particularly problematic for claims about AIME performance (30 questions per test, where a single question represents >3 percentage points).

2. Ablation of DataFlow pipeline components. The paper never isolates which specific operators contribute most to data quality. For example, does the MathQ-Verify filtering step (Section 7.2.1) account for most of the math reasoning improvement, or does the CoT generation with DeepSeek-R1 matter more? Without component-level ablations, users can't prioritize which parts of the pipeline to adopt.

3. Comparison against simple baselines for synthesis. Several domains compare against other synthetic datasets (Open-R1, SYNTHETIC-1, SynSQL) but not against trivial synthesis baselines. For math, how much of the improvement comes from simply prompting DeepSeek-R1 to generate CoT solutions without the full DataFlow pipeline? For code, what if you just use the seed Ling-Coder-SFT data directly without DataFlow processing? Without these baselines, it's unclear whether the pipeline's complexity is necessary.

4. Cost accounting for data production. The paper makes efficiency claims (90K vs. 2.5M samples for Text-to-SQL) but never accounts for the LLM inference cost of producing DataFlow datasets. If generating 90K DataFlow Text-to-SQL examples costs 10× more in API calls than sampling 2.5M SynSQL examples (due to multi-stage generation, verification, and refinement), the efficiency claim becomes less compelling. A total-cost comparison would be more honest.

5. Data contamination analysis. The Gaokao24-Mix result for math reasoning (42.9 for DataFlow vs. 20.9 for Open-R1—a 22-point gap, Table 5) is so large it raises questions about whether DataFlow's NuminaMath seed expansion inadvertently includes Gaokao test-set problems. The paper should investigate and report n-gram overlap or embedding similarity between generated training data and test benchmarks.

6. Cross-model generalization for agentic RAG. The agentic RAG experiments train and evaluate exclusively on Qwen2.5-7B-Instruct. Whether DataFlow's multi-hop question synthesis benefits other model families (Llama, Mistral) is untested, limiting the generality of the claim that synthetic data "consistently exceed the robustness of existing human-annotated multi-hop datasets" (Section 7.5.2).

7. Longer training for unified multi-domain. The unified experiment uses only 10K samples and presumably a small number of epochs. It's possible that Inf-1M would outperform DataFlow-Instruct-10K with more training (the 1M dataset may have a slower but higher-asymptote learning curve). Running both to convergence rather than fixed-epoch comparisons would provide a fairer evaluation.

Genuine weaknesses in experimental design:

• Single-run results with no error estimation make quantitative comparisons unreliable at the reported precision. The paper reports results to 1–2 decimal places, implying precision that the experimental design doesn't support.

• Weak pretraining baselines. The pretraining experiment (Table 2) shows a 0.12-point advantage for DataFlow-30B over FineWeb-Edu-30B, which is likely within noise for a 0.5B model evaluated on single-run benchmarks. The paper shouldn't claim this as a meaningful improvement.

• No comparison against simply using more seed data. For math reasoning, Open-R1 and SYNTHETIC-1 are themselves synthetic datasets. A missing baseline is "train on 10K randomly sampled NuminaMath problems directly without any DataFlow processing"—this would isolate whether the pipeline adds value over the seed data.

• The conversation experiment has contradictory baselines. ShareGPT and UltraChat degrade performance (Table 4), which establishes that DataFlow-Chat-15K is better than these specific datasets, but doesn't establish it's better than all human conversation data. A comparison against more carefully curated conversation datasets (e.g., OpenAssistant, Dolly) would strengthen the claim.

• Unclear data leakage paths. The DataFlow pipelines use strong LLMs (GPT-4o, DeepSeek-R1, o4-mini) for synthesis and verification. If these models were trained on test-set-like data, their generated outputs might inadvertently encode test-set information, inflating downstream performance. No decontamination analysis is reported.

Where the claims hold conditionally:

The claim that DataFlow-generated data outperforms baselines holds most strongly when: (1) the task requires structured, multi-step reasoning (math, SQL, multi-hop QA) rather than factual recall (MMLU, C-EVAL); (2) the baseline is another synthetic dataset at matched scale rather than a larger-scale or human-curated dataset; (3) the base model has sufficient capacity to absorb the synthetic data's structure (14B > 7B for code, 32B for math reasoning); (4) the evaluation uses greedy decoding rather than ensembling (the DataFlow advantage narrows under majority voting for Text-to-SQL). The claim weakens substantially when: (1) the task is dominated by pretrained knowledge; (2) the data scale is very small (5K SFT results show filtering benefits are dataset-dependent); (3) the evaluation metric is coarse (pretraining benchmark averages compress differences to sub-1-point margins).

6. Limitations and Trade-offs

Limitation 1: The Cost of Producing DataFlow Datasets Is Unaccounted For, Making Efficiency Claims Incomplete

The assumption or constraint. The paper makes a central argument about data efficiency: DataFlow-synthesized training data achieves superior downstream performance at smaller sample sizes than baselines. For Text-to-SQL, DataFlow-Text2SQL-90K outperforms SynSQL-2.5M (Table 7). For unified multi-domain, DataFlow-Instruct-10K outperforms Inf-1M on math (Table 10). These comparisons are fundamentally sample-matched: they ask "given the same number of training examples, which produces a better model?"

The paper never accounts for the cost of producing the DataFlow datasets. Every DataFlow pipeline involves multi-stage LLM-driven synthesis, evaluation, and refinement. The Text-to-SQL pipeline (Section 6.1.2) invokes up to 7 LLM-driven operators per pipeline (SQL Generator, Question Generator, CoT Generator, Prompt Generator, Text2SQL Consistency Filter, and two Classifiers), plus SQL Execution Filter which runs queries against actual databases. The math reasoning pipeline (Section 7.2.1) uses o4-mini for problem synthesis, MathQ-Verify for quality verification, and DeepSeek-R1 for CoT generation. The code pipeline (Section 7.3.1) processes seed data through a multi-stage generation and filtering workflow. The unified 10K dataset (Section 7.7.1) composites outputs from three separate synthesis pipelines.

The consequence. The paper's headline claims about efficiency are asymmetric. "3.6% as many training examples" (Section 5) for Text-to-SQL is a valid statement about downstream training data volume, but it is not a statement about total resource consumption. If producing 90K DataFlow Text-to-SQL examples requires 10× more LLM API calls than producing 2.5M SynSQL examples (because SynSQL is a simpler single-stage synthesis, while DataFlow runs multi-stage generate–evaluate–filter–refine with execution-grounded verification), then DataFlow is less efficient in total compute, even if it is more efficient in downstream training samples.

The same issue applies to the unified multi-domain claim. DataFlow-Instruct-10K requires running three separate domain-specialized synthesis pipelines, each using strong LLMs (DeepSeek-R1 for math CoT, the code pipeline's LLM-driven generation and filtering, and the text pipeline's Condor Generator + Refiner + SFT filtering stack). Inf-1M, by contrast, is sampled from a pre-existing large-scale instruction dataset whose synthesis cost is already amortized. Comparing the downstream training efficiency without accounting for the upstream data production cost is like claiming a hand-crafted meal is "more efficient" than fast food because you eat fewer calories, while ignoring the hours spent cooking.

What evidence exists in the paper. The paper provides no accounting whatsoever of LLM inference costs for data production. No API call counts, token counts, estimated FLOPs, or dollar costs are reported for any pipeline. The cost of the LLMs used for synthesis (GPT-4o, DeepSeek-R1, o4-mini) is not discussed. Section 7's experimental methodology sections describe what data was produced and at what scale, but never at what production cost. The paper's comparison framework is exclusively downstream-training-sample-matched.

The paper does not acknowledge this as a limitation. The closest statement is an implicit recognition in Section 3.2 that DataFlow pipelines involve LLM inference at scale, but this is framed as a feature ("LLM-driven operators invoke local inference engines or online API-based services via the unified serving abstraction") rather than as a cost that should be accounted for in efficiency comparisons.

Mitigation status. The paper does not attempt to address this limitation and does not flag it as an area for future work. A fair comparison would report total cost, including both data production and downstream training, perhaps in a format like: "DataFlow-Text2SQL-90K (X API calls for production + Y GPU-hours for training) vs. SynSQL-90K (Z API calls + Y GPU-hours) vs. SynSQL-2.5M (Z' API calls + Y' GPU-hours)." Without this, practitioners cannot make informed decisions about whether DataFlow's data quality advantage justifies its production cost.

Limitation 2: Hard Problems Remain Essentially Unsolved — DataFlow-Generated Data Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. DataFlow's synthesis paradigm assumes that an LLM can generate useful training data for a given task. This assumption breaks down when the task is outside the capability range of the models used for synthesis. The paper relies on strong LLMs (GPT-4o, DeepSeek-R1, o4-mini) as synthesis engines — these models must themselves be capable enough to generate correct, high-quality examples for the target domain.

The consequence. For tasks where even strong LLMs struggle, DataFlow pipelines will produce low-quality or incorrect training data, which can actively harm downstream models. The paper provides concrete evidence of this in two places:

First, the agentic RAG results (Table 8) show that DataFlow-AgenticRAG-10K achieves only 20.2 on Musique after 3 epochs — barely above the base model's 9.9. Musique requires composing multiple reasoning hops across disparate documents, and the paper's o4-mini-based synthesis pipeline (Section 7.5.1) appears unable to reliably generate training examples that teach this skill. The verification module "eliminat[es] samples with problems such as intermediate question leakage, logical errors, and excessively high or low difficulty," but this filtering cannot create quality where the generator produces none.

Second, the DataFlow-Agent's Hard-tier code-GT score of 0.23 (Table 12) demonstrates that when the user's specification is ambiguous, the agent cannot reliably synthesize correct operators — it produces "alternative yet plausible operator compositions that diverge from a single ground-truth program." This is not a failure of the agent's architecture but of the underlying synthesis capability: the LLM driving the Operator Synthesis Agent (Section 5.2) does not understand the user's intent well enough to produce correct code.

More broadly, this limitation means DataFlow's value proposition is conditional on the synthesis LLM's capabilities. If you want to prepare data for a task where even GPT-4o-level models perform poorly (novel scientific reasoning, complex multi-step planning, tasks requiring specialized domain knowledge the LLM lacks), DataFlow provides no advantage over alternative approaches — you need human annotators, not LLM-driven synthesis. The paper's experiments all operate in domains (math, code, SQL, general text) where strong LLMs are known to be competent. The framework's effectiveness on genuinely hard or novel tasks is unproven.

What evidence exists in the paper. The Musique result (20.2 after 3 epochs, Table 8) and the Hard-tier agent score (0.23, Table 12) are direct evidence of capability boundaries. The paper does not discuss this limitation in general terms but provides the data that reveals it. Additionally, the knowledge extraction results (Table 9) show that DataFlow-Knowledge achieves only 40.86% on PubHealth — a domain where even strong LLMs may lack sufficient medical knowledge for reliable QA synthesis. The paper doesn't investigate whether the remaining errors stem from synthesis failures (the LLM generated incorrect QA pairs that passed verification) or from the downstream model's limitations.

Mitigation status. The paper does not address this limitation. There is no discussion of how to determine whether a target task is within the synthesis LLM's capability range, no proposal for hybrid human-LLM synthesis pipelines for hard tasks, and no acknowledgment that DataFlow's effectiveness is bounded by LLM capability. This is a significant omission for practitioners considering DataFlow for domains where LLM performance is uncertain.

Limitation 3: Single-Run Evaluations Without Statistical Rigor Make Quantitative Comparisons Unreliable at Reported Precision

The assumption or constraint. The paper reports all experimental results as point estimates — single numbers to one or two decimal places — without confidence intervals, standard deviations, error bars, or statistical significance tests. This implies a level of precision and reliability that the experimental design does not support.

The consequence. Many of the paper's specific quantitative claims are not statistically distinguishable from noise given the test set sizes and the magnitude of reported differences:

• Pretraining filtering (Table 2): DataFlow-30B achieves 35.69 average vs. FineWeb-Edu-30B at 35.57 — a difference of 0.12 points. With a 0.5B model evaluated on benchmarks like ARC-C (which has ~1,200 test questions in the Challenge set), a 0.12-point difference is almost certainly within sampling noise. The paper's claim that "DataFlow method achieves the highest average score" (Section 7.1.2) overstates the reliability of this result.

• SFT filtering (Table 3): Alpaca(filtered) improves Math average by 2.5 points over Alpaca(random) (37.3 → 39.8). With the Math average aggregating over benchmarks like MATH (5,000 test questions) and GSM8K (~1,300 test questions), a 2.5-point swing could be statistically significant — but without variance estimates, we cannot know.

• AIME benchmarks (Table 5): AIME24 has only 30 questions. A single-question difference represents 3.3 percentage points. DataFlow-Reasoning-10K achieves 45.4 on AIME24@32 vs. Open-R1-10K at 51.0 — a 5.6-point gap that could reflect as few as 2 questions' difference. The paper's abstract claims "1–3 point gains on MATH, GSM8K, and AIME" without acknowledging that AIME gains are measured on a 30-question test where a single question is worth 3.3 points.

• Code benchmarks (Table 6): HumanEval+ has 164 problems, meaning each problem is worth ~0.6 points. DataFlow-Code-10K (14B) achieves 76.2 vs. 73.8 for DataFlow-Code-1K (14B) — a 2.4-point difference that could arise from ~4 additional correct solutions. Without confidence intervals, we cannot assess whether scaling from 1K to 10K code samples produces a reliable improvement.

The problem compounds when the paper reports aggregated averages across multiple benchmarks. The Math-Avg in Table 10 averages over 7 benchmarks with different test set sizes and variances. Aggregating point estimates without propagating uncertainty produces a number (46.7) that looks precise but whose reliability is unknown.

What evidence exists in the paper. The paper provides no statistical protocol anywhere. Section 7's experimental methodology describes datasets, models, metrics, and baselines, but never mentions standard deviations, confidence intervals, significance tests, or the number of evaluation runs. The agentic orchestration experiment (Section 7.8) uses an LLM-Judge score that is presumably continuous in [0, 1], but reports only averages across 6 pipelines × 3 difficulty levels without any measure of variance across pipelines or queries. The Text-to-SQL experiments report two decoding strategies (greedy and majority voting) which provide some robustness check, but still as point estimates without variance.

Mitigation status. The paper does not acknowledge the absence of statistical rigor as a limitation. The reporting conventions (results to 1–2 decimal places, claims of "improvement" for sub-1-point differences) treat the numbers as if they are precise measurements. For a paper making specific quantitative claims about superiority, this is a significant methodological weakness. Future work should report confidence intervals (bootstrapped over test examples or across multiple training runs), and claims about improvement should be qualified by whether the difference exceeds expected variance.

Limitation 4: Difficulty Estimation for Adaptive Data Preparation Is Not Developed, Despite Being a Natural Extension of the Framework

The assumption or constraint. DataFlow treats all examples within a pipeline uniformly — the same generation, evaluation, filtering, and refinement operators are applied to every sample, regardless of how easy or hard that sample is for the target model. The framework provides classification operators (SQL Component Classifier, SQL Execution Classifier in Section 6.1.1) that measure difficulty, but these measurements are used only for labeling the output data, not for adapting the pipeline's behavior based on difficulty.

The consequence. This is a missed opportunity that limits DataFlow's data efficiency. The test-time compute scaling literature (referenced in the prior sections' analysis) has established that adaptive allocation based on estimated difficulty can yield 4× efficiency improvements over uniform strategies. The same principle should apply to data synthesis: easy examples (where the LLM reliably produces correct outputs on the first attempt) need less verification and refinement than hard examples (where the LLM struggles). A difficulty-adaptive DataFlow pipeline could:

• Allocate more generation budget (more candidate samples, higher temperature, multiple synthesis strategies) to hard examples.
• Apply heavier verification (multiple verifiers, execution-based checks, human-in-the-loop review) only to examples where the generator's confidence is low.
• Route easy examples through a fast path (generate once, light verification) and hard examples through an intensive path (generate multiple candidates, verify with execution, refine, regenerate).
• Use the SQL Execution Classifier's model-dependent difficulty estimates not just as output labels but as routing signals within the pipeline itself.

The paper provides the building blocks for this — difficulty classifiers exist in the Text-to-SQL pipeline, the operator abstraction supports conditional execution in forward(), and the key-binding mechanism would naturally support branching based on computed fields — but never explores adaptive allocation.

The consequence for practitioners is that DataFlow pipelines are wasteful on easy examples (spending verification and refinement compute where it isn't needed) and potentially insufficient on hard examples (applying the same fixed budget where more effort would help). In the Text-to-SQL pipeline, every generated SQL goes through the same Execution Filter, Question Generator, CoT Generator, Prompt Generator, and two Classifiers — even if the SQL is trivially simple and obviously correct.

What evidence exists in the paper. The Text-to-SQL pipeline's SQL Execution Classifier (Section 6.1.1) explicitly implements a model-dependent difficulty measure — it queries the LLM k times and classifies difficulty based on the success ratio n/k. The paper notes that "execution difficulty is model-dependent: more capable LLMs achieve higher success rates on the same task and thus are considered to have lower execution difficulty." This is exactly the kind of difficulty estimate that could drive adaptive allocation, but the classifier is used only as a final labeling step after all other processing is complete (Figure 7: it appears at the end of both pipelines). The difficulty label is attached to the output data for downstream consumers but never feeds back into the pipeline's own behavior.

Mitigation status. The paper does not discuss adaptive or difficulty-conditioned pipeline behavior as a design goal or future direction. This is a pragmatic limitation — DataFlow as a v1.0 system prioritizes uniform, reproducible pipelines over adaptive ones — but it represents a clear path for future improvement that would strengthen the framework's efficiency claims and connect it to the broader literature on compute-adaptive inference and data generation.

Limitation 5: Single Benchmark and Single Model Family Per Domain — Cross-Domain and Cross-Model Generalization Is Unverified

The assumption or constraint. The paper evaluates DataFlow across six domains, but each domain is tested on a narrow slice of benchmarks and model families:

• Text pretraining: One model (Qwen2.5-0.5B), one corpus (SlimPajama-627B subset), evaluated on 6 general benchmarks.
• Math reasoning: One model family (Qwen2.5-32B-Instruct), compared against two synthetic baselines (Open-R1, SYNTHETIC-1), evaluated on 8 math benchmarks.
• Code: Two model sizes from one family (Qwen2.5-7B/14B-Instruct), compared against two baselines, evaluated on 4 code benchmarks.
• Text-to-SQL: Two base models from two families (Llama-3.1-8B, Qwen2.5-Coder-7B), evaluated on 6 SQL benchmarks — the strongest cross-model evidence in the paper.
• Agentic RAG: One model (Qwen2.5-7B-Instruct), trained with one RL framework (ReCall/GRPO), evaluated on 4 multi-hop benchmarks.
• Knowledge extraction: One model (Qwen2.5-7B-Instruct), one domain (medical), evaluated on 3 medical QA benchmarks.
• Unified multi-domain: Two model sizes from two generations (Qwen2-7B, Qwen2.5-7B), compared against Infinity-Instruct.

The consequence. The paper demonstrates that DataFlow works well for Qwen-family models on standard benchmarks, but provides limited evidence for generalization to other model families (Llama, Mistral, Gemma, DeepSeek), other domains (legal, financial, scientific beyond medical), or other data modalities (multimodal, multilingual). This matters because the interaction between synthetic data quality and model architecture/pretraining is not well understood:

• Model family dependence: Qwen models may have particular inductive biases or pretraining data distributions that make them more or less receptive to DataFlow's synthetic data. The Text-to-SQL experiment shows DataFlow benefits both Llama-3.1-8B and Qwen2.5-Coder-7B (Table 7), which is encouraging but limited to one domain. The unified multi-domain experiment (Tables 10–11) shows qualitatively different patterns for Qwen2-7B vs. Qwen2.5-7B — Inf-1M degrades Qwen2.5-7B math dramatically (37.1 → 33.3) but degrades Qwen2-7B math only slightly (20.1 → 27.9 is actually an improvement) — suggesting strong model-generation interactions in how synthetic data is absorbed.

• Benchmark specificity: The paper evaluates on standard benchmarks that are widely used and potentially saturated. DataFlow's advantages might be smaller or absent on truly out-of-distribution tasks that better test generalization. The agentic RAG experiment's OOD evaluation (Table 8) partially addresses this, showing DataFlow's synthetic data improves cross-benchmark generalization, but this is only one domain.

• Domain specificity: The knowledge extraction pipeline is evaluated only on medical QA. Whether the same pipeline design (MinerU normalization, segmentation, filtering, QA generation, verification) transfers to legal or financial domains is unverified. Domain-specific terminology, document structures, and reasoning patterns may require pipeline modifications that the paper's "no task-specific glue code" claim doesn't anticipate.

What evidence exists in the paper. The Text-to-SQL experiment provides the best cross-model evidence (Table 7), showing DataFlow benefits both Llama-3.1-8B and Qwen2.5-Coder-7B with similar patterns. The code experiment tests two model sizes but only one model family. The unified experiment tests two model generations but again only Qwen. The agentic RAG, math reasoning, conversation, and knowledge extraction experiments each use a single model. The paper does not discuss model-family generalization as a concern.

Mitigation status. The paper implicitly acknowledges scope limitations by testing multiple model families only in Text-to-SQL, but does not frame this as a limitation requiring future work. A systematic cross-model-family evaluation (e.g., testing the math pipeline on Llama-3, Mistral, and DeepSeek base models) would substantially strengthen the claim that DataFlow's data quality is model-agnostic rather than Qwen-specific.

Limitation 6: The Revision/Refinement Paradigm Is Underexplored — Refine Operators Apply Only Lightweight Transformations, Not Iterative LLM-Driven Improvement

The assumption or constraint. DataFlow defines four functional operator categories: generate, evaluate, filter, and refine (Section 4.3). The Refine category is described as operators that "modify specific fields within existing rows without changing the number of samples" and "often apply lightweight transformations such as removing URLs or emojis from text."

This is a notably narrow definition of refinement compared to what "refinement" means in the broader LLM data preparation literature, where it typically refers to iterative LLM-driven improvement: generating a candidate, having an LLM critique it, and having the same or another LLM revise it based on the critique — potentially through multiple rounds. The paper's Condor Refiner pipeline (mentioned in Section 7.1.1) and the SQL Refinement Pipeline (Section 6.1.2) use the term "refinement" but what they actually do is augment (SQL Augmentor generates variant queries from seeds) or filter and re-generate (the Condor Refiner generates new responses, it doesn't iteratively improve existing ones).

The consequence. DataFlow misses a major category of LLM-driven data improvement. The revision model literature (Qu et al., 2024; and the test-time compute scaling paper analyzed in the reference example) has established that iterative self-refinement — where a model conditions on its own previous incorrect outputs to produce improved outputs — can substantially boost data quality, particularly on reasoning tasks. This is fundamentally different from DataFlow's current refinement model:

• DataFlow's refinement: Remove URLs, normalize whitespace, fix formatting. These are rule-based or lightweight-LLM operations that don't change the semantic content of the data.
• Iterative LLM-driven refinement: "This solution has a logical error in step 3. Regenerate step 3 and all subsequent steps." This requires the LLM to understand why the current output is wrong and how to fix it — a much more complex operation that can transform incorrect data into correct data.

The absence of true iterative refinement means DataFlow pipelines cannot rescue near-misses. If the SQL Generator produces a query that is almost correct but has a minor syntax error, the SQL Execution Filter removes it entirely. An iterative refinement operator could instead: (1) execute the query, (2) observe the error message, (3) prompt the LLM to fix the specific error, (4) re-execute, (5) keep the fixed version. This would increase yield from the generation stage and reduce waste. The paper's generate–evaluate–filter–refine paradigm treats generation and filtering as a one-pass pipeline: generate candidates, filter bad ones, keep the rest. Iterative refinement would enable a generate–evaluate–refine–regenerate loop that is more sample-efficient.

What evidence exists in the paper. The paper's own data shows that DataFlow pipelines are low-yield. Figure 5 shows the evolution of sample counts across pipeline stages. All pipelines start with 1,000 input samples and end with some final count. The filtering stages reduce sample counts — sometimes dramatically, though exact numbers aren't provided. If a pipeline starts with 1,000 seeds and produces only 200 high-quality outputs after filtering, 800 seeds' worth of generation compute was wasted. Iterative refinement could recover some fraction of those 800 by fixing correctable errors. The paper never quantifies this waste or explores refinement as a mitigation.

The Text-to-SQL pipeline's SQL Augmentor (Section 6.1.1) is the closest thing to semantic refinement, but it generates new variants from seeds rather than improving existing outputs. The Condor Refiner (Section 7.1.1) is mentioned by name but never described in detail — from context, it appears to be a re-generation operator rather than a critique-and-revise loop.

Mitigation status. The paper does not acknowledge the gap between its definition of "refinement" and the broader literature's understanding of iterative LLM-driven improvement. The Refine category, as implemented, is essentially a catch-all for lightweight post-processing. This is a design choice, not an oversight — the paper's philosophy emphasizes generation quality and filtering rigor over iterative improvement — but it represents a clear direction for future work that would strengthen DataFlow's data efficiency and connect it to the self-improvement literature. The operator abstraction (standardized key-based I/O, LLM serving integration) provides all the infrastructure needed for iterative refinement operators; the framework just doesn't include them.

7. Implications and Future Directions

How This Work Changes the Landscape

DataFlow does not introduce a new algorithm for data synthesis, a new training objective, or a new model architecture. Its contribution is architectural and methodological: it proposes that LLM data preparation should be treated as a programmable, composable, and verifiable dataflow problem rather than as ad-hoc scripting or configuration-based curation. This is a reframing, not a paradigm shift—the individual techniques (LLM-driven generation, execution-based filtering, prompt templates) all exist in prior work—but the reframing has practical consequences that change which problems the field can systematically attack.

From curation to synthesis as the default mental model. The paper's generate–evaluate–filter–refine paradigm places synthesis at the front of every pipeline, treating LLMs as data factories rather than data inspectors. This inverts the relationship that systems like NeMo Curator and Data-Juicer established, where generation was an auxiliary feature bolted onto a filtering-oriented architecture. The inversion matters because it changes the design pressure on data preparation systems: if synthesis is the primary operation, the system must optimize for prompt management, multi-model orchestration, execution-grounded verification, and generation budget allocation—concerns that are secondary or absent in curation-oriented frameworks. The paper's empirical results support this inversion: across six domains, DataFlow-synthesized data matches or exceeds both human-curated datasets (Tables 3, 4, 8) and large-scale synthetic corpora (Tables 5, 7, 10) at substantially smaller downstream training scales.

Reconciling contradictory findings about synthetic data quality. The paper provides evidence that resolves an apparent contradiction in the literature. On one side, works like Self-Instruct, Alpaca, and WizardLM showed that LLM-generated instruction data could meaningfully improve models. On the other side, practitioners frequently observed that training on web-scraped or casually generated synthetic data degraded performance—the paper itself demonstrates this: Code Alpaca-1K and Self-OSS harm Qwen2.5-7B-Instruct's code performance (Table 6), and Inf-10K catastrophically degrades Qwen2.5-7B-Base's math from 37.1 to 22.6 (Table 10). The reconciliation is that synthetic data quality is extremely sensitive to the generation pipeline's design, and that unverified, unfiltered, single-pass generation produces data that is often worse than no fine-tuning at all. DataFlow's multi-stage pipelines—generation followed by execution-grounded verification (SQL Execution Filter, MathQ-Verify), consistency checking (Text2SQL Consistency Filter), and targeted refinement—consistently avoid the degradation that simpler synthesis approaches suffer. This converts the question from "does synthetic data help?" to "what pipeline design choices make synthetic data helpful?"—a more productive framing.

Making data preparation a domain for systematic optimization. Before DataFlow, comparing data preparation strategies required comparing entire pipelines implemented as ad-hoc scripts, where differences could arise from any of dozens of undocumented implementation choices. DataFlow's standardized operator interface with explicit key-based I/O contracts makes it possible to run controlled ablation experiments on data preparation: swap one filter for another, insert a verification step, change the prompt template—all while keeping the rest of the pipeline identical. This transforms data preparation from an artisanal craft into an engineering discipline where improvements can be isolated, measured, and accumulated. The paper itself doesn't fully exploit this capability (it reports whole-pipeline comparisons rather than component-level ablations), but the infrastructure enables such studies.

Redirecting research attention from scale to structure. The unified multi-domain experiment (Tables 10–11) provides the paper's most important reframing: DataFlow-Instruct-10K (10K samples, domain-specialized synthesis) dramatically outperforms Inf-1M (1M samples, generic instruction data) on math (46.7 vs. 33.3) and is competitive on code and knowledge. This suggests that for tasks requiring structured reasoning, pipeline design matters more than data scale—a 100× increase in data quantity cannot compensate for poor synthesis strategy. This redirects research attention from "how do we generate more data?" toward "how do we design better generation, verification, and refinement operators?"—a shift that makes data preparation a systems design problem rather than a scaling problem.

Establishing verifier over-optimization as a relevant concern for data synthesis. The paper's generate–evaluate–filter paradigm implicitly relies on verifiers (execution-based checks, LLM-based consistency filters, MathQ-Verify) to distinguish good from bad synthetic outputs. But the paper also provides evidence that verifiers are imperfect: the SQL Execution Filter removes only queries that fail to execute or exceed a runtime threshold—it cannot detect semantically incorrect queries that happen to execute successfully. The Text2SQL Consistency Filter is itself an LLM that may make errors. This creates a verifier quality bottleneck analogous to the reward model over-optimization problem in RLHF: as synthesis pipelines become more sophisticated at optimizing against verifier signals, they may produce data that scores well under the verifier but is actually low-quality. The paper doesn't explore this failure mode, but its architecture makes it visible and studyable in a way that ad-hoc scripts do not.

Follow-Up Research This Work Enables

Component-level ablation of DataFlow pipelines to identify which operators drive quality improvements. The paper reports whole-pipeline comparisons (DataFlow vs. Open-R1, DataFlow vs. SynSQL) but never isolates individual operators' contributions. A follow-up study would take the math reasoning pipeline (Section 7.2.1) and train Qwen2.5-32B-Instruct on ablated variants: (a) NuminaMath seeds with no processing, (b) seeds + problem synthesis only, (c) seeds + synthesis + MathQ-Verify filtering, (d) full pipeline including CoT generation with DeepSeek-R1. Comparing downstream MATH/GSM8K/AIME performance across these ablations would reveal whether the 1.5-point advantage over Open-R1 (Table 5) comes primarily from seed quality, problem verification, or CoT generation strategy. The operator abstraction's standardized I/O contracts make this experiment straightforward: each ablation is just a different forward() method composing a subset of operators with the same key bindings. Without this component-level understanding, practitioners cannot prioritize which parts of the pipeline to adopt—they must either use the entire framework or guess at which pieces matter.

Difficulty-adaptive budget allocation within DataFlow pipelines using the execution classifier pattern. The Text-to-SQL pipeline's SQL Execution Classifier (Section 6.1.1) already computes a model-dependent difficulty estimate by querying the LLM k times and measuring the execution success ratio n/k. But this estimate is used only as an output label—it never feeds back into the pipeline's behavior. A natural extension would use this difficulty signal to adapt the generation budget per example: easy queries (high n/k) follow a fast path (generate once, light verification), while hard queries (low n/k) receive more intensive treatment (generate multiple candidates with diverse strategies, verify with execution, apply the SQL Augmentor to explore variants, and select the best). The key question is whether difficulty-adaptive allocation improves yield—the fraction of seed examples that produce usable training data—without increasing total compute. The experiment would compare a fixed-budget DataFlow pipeline against an adaptive variant that allocates total generation budget proportional to estimated difficulty, measuring both downstream model performance and total API cost. This connects DataFlow directly to the test-time compute scaling literature, where difficulty-conditioned allocation yields 4× efficiency gains, and would test whether the same principle applies to data synthesis.

Cross-model-family stress-testing of DataFlow's synthesis quality claims. The paper's strongest cross-model evidence is in Text-to-SQL (Table 7), where DataFlow benefits both Llama-3.1-8B and Qwen2.5-Coder-7B. But the math, code, and unified experiments use only Qwen-family models. A systematic cross-family evaluation would fine-tune Llama-3.1-8B, Mistral-7B, and DeepSeek-Coder-7B on DataFlow's math, code, and unified datasets, comparing against the same baselines used in the paper (Open-R1, SYNTHETIC-1, Code Alpaca, Inf-1M). The critical measurement is the interaction between model family and synthesis strategy: does DataFlow-Reasoning-10K improve Llama-3.1-8B's MATH by a similar margin as Qwen2.5-32B's (Table 5), or does the benefit depend on architectural or pretraining-data similarities between the synthesis LLM (DeepSeek-R1) and the target model? A negative result—DataFlow's advantage is Qwen-specific—would substantially narrow the framework's applicability claim and suggest that synthetic data quality is not model-agnostic.

Iterative refinement operators that rescue near-miss generation outputs. DataFlow's current Refine category (Section 4.3) applies only lightweight transformations (URL removal, whitespace normalization). A genuine iterative refinement operator for the Text-to-SQL pipeline would: (1) execute a generated SQL query, (2) if execution fails, feed the error message + the failed query back to the LLM with a prompt like "Fix the following SQL query. Error: [error]. Query: [query].", (3) execute the revised query, (4) repeat until execution succeeds or a maximum number of attempts is reached, (5) keep the fixed version if successful, discard if not. This is a generate–evaluate–refine–regenerate loop rather than the current generate–evaluate–filter–discard pipeline. The experiment would compare the yield (fraction of seed queries that produce usable training examples) and downstream model performance of the pipeline with and without the iterative refinement operator, at matched total generation budget. The hypothesis is that iterative refinement increases yield by recovering queries with minor errors, reducing the number of seeds needed to produce a target dataset size. The operator abstraction already provides the infrastructure for this (the operator would read the SQL column, call the LLM serving layer with an error-aware prompt, write the fixed SQL back, and the SQL Execution Filter would re-execute downstream)—the experiment tests whether the approach works in practice.

Failure mode analysis: when does DataFlow-generated data harm downstream models? The paper provides two instances of synthetic data degradation—Code Alpaca harming code performance (Table 6) and Inf-10K catastrophically degrading math (Table 10)—but doesn't systematically characterize when and why synthetic data is harmful. A follow-up study would deliberately construct low-quality DataFlow pipelines by degrading specific stages: (a) removing the SQL Execution Filter from the Text-to-SQL pipeline (so invalid SQL enters training), (b) removing MathQ-Verify from the math pipeline (so logically inconsistent problems enter training), (c) replacing DeepSeek-R1 with a weaker model for CoT generation, (d) removing the Text2SQL Consistency Filter (so misaligned question-SQL pairs enter training). For each degraded pipeline, measure the downstream model's performance relative to the full pipeline and to the no-fine-tuning baseline. This would produce a failure taxonomy that tells practitioners which pipeline stages are essential (degradation when removed) vs. nice-to-have (minimal impact when removed). The taxonomy would also help the agentic orchestration layer prioritize verification and filtering when constructing pipelines automatically.

Combining DataFlow's synthesis pipelines with the revision model training paradigm from Qu et al. (2024). The test-time compute scaling paper analyzed in the reference example demonstrated that fine-tuning a model to revise its own incorrect outputs (conditioning on previous wrong answers to produce improved answers) boosted pass@1 on math reasoning. DataFlow's operator abstraction could implement revision-model training data generation as a pipeline: (1) use the math reasoning pipeline to generate candidate solutions, (2) use an evaluation operator to identify incorrect solutions that are "close" to correct (e.g., by final-answer edit distance), (3) construct multi-turn training sequences of incorrect-followed-by-correct answers, (4) output a revision-training dataset. The experiment would compare a model fine-tuned on standard DataFlow math data vs. the same model fine-tuned on revision-format DataFlow data, measuring both standard accuracy and the model's ability to improve its own answers through iterative revision at inference time. This connects DataFlow's synthesis capability to the self-improvement literature and tests whether DataFlow-generated data can teach models not just to answer correctly but to recover from their own errors.

Practical Applications and Downstream Use Cases

Cost-efficient multi-domain instruction tuning for small-to-medium models. The unified multi-domain experiment (Section 7.7) provides the economic justification: DataFlow-Instruct-10K, a 10K-sample dataset spanning math, code, and text, enables Qwen2.5-7B-Base to achieve a Math-Avg of 46.7—within 3.1 points of the full Qwen2.5-7B-Instruct model (Table 10) and 13.4 points ahead of training on 1M Infinity-Instruct samples. For organizations fine-tuning 7B-class models on internal tasks, this translates to a concrete recipe: generate ~10K domain-specialized synthetic examples using DataFlow's pipelines (or custom pipelines built on DataFlow's operator library) rather than sampling 1M examples from generic instruction datasets. The downstream training cost is reduced 100× in data volume, and the synthesis cost—while not free—involves a one-time investment to generate the dataset rather than ongoing costs for larger-scale training. The paper's open-source release of DataFlow-Instruct-10K makes this immediately actionable: practitioners can fine-tune on the released dataset directly, or use it as a template for constructing domain-specific variants.

High-quality Text-to-SQL data generation at 3.6% of prior data scale requirements. The Text-to-SQL results (Table 7) demonstrate that DataFlow-Text2SQL-90K outperforms SynSQL-2.5M on Qwen2.5-Coder-7B-Instruct under both greedy (71.6 vs. 70.0) and majority voting (74.0 vs. 71.6) while using only 90K training examples—3.6% of the SynSQL scale. For organizations building Text-to-SQL systems, this means that generating a small, high-quality dataset using DataFlow's SQL Generation and SQL Refinement pipelines (Section 6.1.2) is more effective than procuring or generating much larger quantities of lower-quality synthetic data. The pipeline is directly reusable: the paper provides operators for MySQL, SQLite, and PostgreSQL (via the DatabaseConnector abstraction), and the prompt template mechanism allows adaptation to new database systems by swapping templates without modifying operator code. An organization with a proprietary database schema can: (1) implement a DatabaseConnector for their system, (2) run the SQL Generation Pipeline to produce seed queries from their schema, (3) optionally apply the SQL Refinement Pipeline if they have existing query logs to augment, (4) fine-tune their model on the resulting dataset. The paper's reported +31.8 point improvement on EHRSQL (24.3 → 56.1, Table 7) is particularly relevant for domain-specific database applications where off-the-shelf models perform poorly.

Multi-hop question generation for RAG system evaluation and training. The agentic RAG results (Table 8) demonstrate that DataFlow-AgenticRAG-10K produces multi-hop questions with superior cross-dataset generalization compared to human-authored datasets: OOD averages of 37.4 (vs. HotpotQA's 36.4), 43.6 (vs. Musique's 42.4), and 36.4 (vs. 2Wiki's 33.8). For teams developing RAG systems, this provides a method for generating evaluation benchmarks that test compositional reasoning without the risk of benchmark contamination (since the questions are synthetically generated from Wikipedia documents, they are guaranteed to be absent from public training sets). The pipeline is straightforward to adopt: randomly sample documents from a Wikipedia dump, use o4-mini (or a comparably strong model) with DataFlow's generation and verification operators to produce questions, and validate with the verification module. The paper's exclusion of documents appearing in test benchmarks (Section 7.5.1) provides a template for constructing clean evaluation sets. Beyond evaluation, the synthetic questions can serve as training data for RAG-capable models, with the OOD generalization results suggesting they teach more robust reasoning patterns than human-authored questions.

Domain-specific knowledge extraction from unstructured corpora at scale. The knowledge extraction pipeline (Section 7.6) converts raw documents (PDFs, textbooks, clinical guidelines) into structured, verified QA pairs, achieving PubMedQA: 53.40%, Covert: 68.33%, PubHealth: 40.86% when used to fine-tune Qwen2.5-7B-Instruct—dramatically outperforming both zero-shot CoT and RAG baselines (Table 9). For organizations with large proprietary document collections (legal firms with case law databases, financial institutions with regulatory filings, healthcare systems with clinical guidelines), this pipeline offers a semi-automated path to convert unstructured text into training data for domain-specialized models. The pipeline's stages—MinerU normalization, document segmentation, quality filtering, QA generation, automated verification—are implemented as DataFlow operators that can be adapted to new domains by swapping domain-specific components (e.g., replacing medical terminology filters with legal terminology filters) while keeping the overall pipeline structure. The 140M-token input scale demonstrates the pipeline can process substantial corpora, and the 37,500-step SFT training (five epochs) provides a concrete training recipe.