DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

ArXiv: 2510.19336

🎯 Pitch

A simple MLP trained on only 250 data-mixture samples can predict downstream multitask fine-tuning performance with R²=0.81, eliminating the costly trial-and-error of mixing 12 heterogeneous phone-agent datasets. When used to optimize for function-calling alone, DaMo boosts BFCL-v3 metrics by 12.47% over competing mixture methods, and the learned optimal ratios transfer across model families with just 20 calibration points.

1. Executive Summary

This paper proposes DaMo (Data Mixing Optimizer), a method that trains a neural network to predict downstream task performance for any given data mixture in multitask supervised fine-tuning of multimodal LLMs, then extrapolates the optimal mixing ratios without brute-force search. Using InternVL2.5-4B as the base model and a suite of 12 training datasets, DaMo fits an MLP on only 250 sampled mixture configurations (1,000 checkpoints total) and achieves R² = 0.81 on held-out data, enabling efficient identification of high-performing mixtures. Evaluated on the authors' newly introduced PhoneAgentBench — a benchmark of 1,235 QA pairs spanning task planning, tool usage, multimodal memory, and screen context understanding for mobile phone agents — DaMo yields a 3.38% average performance gain over the state-of-the-art DML method and improves general-benchmark scores by 2.57% on average (with a striking 12.47% improvement on BFCL-v3 when optimizing for that task alone). The method transfers robustly across model families — maintaining Pearson correlations of 0.75–0.95 when extended from InternVL2.5-4B to Qwen2.5VL-3B, Qwen2.5VL-7B, and InternVL3-14B with only 20 calibration samples — establishing that optimal data mixtures learned on a smaller model remain near-optimal for larger or architecturally different models after a lightweight linear calibration step.

2. Context and Motivation

The Core Problem: We Don't Know How to Mix Fine-Tuning Datasets

The fundamental question this paper tackles is deceptively simple: if you have 12 heterogeneous training datasets for fine-tuning a multimodal LLM, how should you combine them to maximize performance on your downstream tasks? This matters because, unlike pretraining—where scaling laws provide principled guidance on how to allocate compute between model size and data quantity—there is essentially no rigorous framework for determining optimal data mixtures during supervised fine-tuning (SFT). Prior to this work, practitioners relied on heuristics, costly trial-and-error, or methods designed for pretraining loss prediction that fail to capture the complex, non-monotonic dynamics of SFT.

This gap is significant for several reasons the authors highlight in Section 1:

Multitask capability integration is essential for real-world agents. Mobile phone agents must simultaneously master environment perception, task planning, multimodal reasoning, function calling, and personalized memory. Training a single model to handle all these capabilities requires combining multiple specialized datasets, and the proportions profoundly affect which capabilities emerge and which suffer from interference.
Industrial practice relies on expensive manual iteration. The paper explicitly notes that state-of-the-art models like LLaMA3 and Tulu3 determined their SFT data mixtures through "costly manual iteration" (Section 2, Data Mixing paragraph). This is not scalable—each trial requires training a full model from scratch and evaluating across multiple benchmarks. A principled, automated method for determining optimal mixtures would dramatically reduce development time and compute expenditure.
The mixing space is combinatorially vast, making brute-force search impossible. With 12 training datasets and a batch size of 16, the space of fixed data mixtures alone contains $C_{12+16-1}^{12-1} \approx 1.3 \times 10^7$ distinct configurations (Section 4.1). Training and evaluating even a tiny fraction of these would consume thousands of GPU-hours. Any practical solution must extrapolate from sparse samples.

Why Prior Approaches Fall Short

The paper identifies two distinct research communities that have approached data mixing, each with fundamental limitations when applied to multitask SFT of MLLMs.

Pretraining Data Mixing Methods: Optimizing the Wrong Objective

A significant body of work has tackled data mixture optimization for pretraining, where the goal is to minimize language model perplexity (PPL) on validation data. The paper surveys several representative approaches (Section 2):

DoReMi (Xie et al., 2023a) uses Group Distributionally Robust Optimization to learn domain-specific weights that minimize worst-case excess loss across domains.
ODM (Albalak et al., 2023) frames data selection as a multi-armed bandit problem, dynamically adjusting sampling probabilities based on online loss signals.
BiMix (Ge et al., 2024) jointly models domain proportions and data quantity scaling using bivariate power law functions to predict pretraining loss.

These methods share a common architecture: they fit some parametric function (typically exponential or power-law) to predict validation loss as a function of data mixture proportions, then select the mixture that minimizes predicted loss. The critical problem, as the paper argues in Section 4.2, is that validation loss and downstream task performance are fundamentally misaligned in fine-tuning settings.

The paper provides direct empirical evidence for this misalignment in Figure 3(a). When training a model on a single dataset (MMU) and evaluating on PhoneAgentBench subtasks, four qualitatively different relationships emerge between training steps (a proxy for data volume) and downstream performance:

Enhancement: ACU performance improves steadily with more MMU training.
Conflict: APP-Rec performance actually degrades as MMU training continues—more data from one source actively harms performance on another task.
Neutrality: MM-NER performance is essentially uncorrelated with MMU training progress.
Overfitting: MT-Plan initially improves but then sharply declines with continued training, exhibiting harmful overfitting beyond an optimal data volume.

None of these patterns—especially the non-monotonic, conflicting, and U-shaped trajectories—can be captured by the smooth exponential or power-law functions used in pretraining data mixing methods. These functions assume training loss decreases monotonically toward some asymptotic floor, an assumption that fundamentally breaks in multitask fine-tuning where datasets can interact antagonistically. The paper makes this explicit:

"These nonlinearities fundamentally prevent analytical solutions... and invalidate conventional exponential and power functions." (Section 4.2)

Fine-Tuning Data Mixing Methods: Limited Scope and Missing Generalization

Research specifically on SFT data mixing remains sparse and fragmented. The paper characterizes the landscape (Section 2):

Manual heuristics dominate industrial practice. Uniform sampling (all datasets weighted equally) and "natural" sampling (proportional to dataset size) are common defaults, but as Figure 4(a) demonstrates, these perform no better than random mixtures—the vertical dashed lines representing baseline methods fall squarely within the central mass of the random mixture score distribution. The paper states bluntly: "baseline methods show no discernible advantage, demonstrating the inefficiency of heuristic approaches."
DML (Data Mixing Laws, Ye et al., 2024) is identified as the most directly comparable prior work. DML extends exponential loss prediction from pretraining to the fine-tuning setting, fitting a parametric function to predict downstream loss from data mixture proportions. However, like its pretraining counterparts, it relies on the assumption that loss decreases smoothly with data scale—an assumption that Figure 3 directly contradicts for the interactive, antagonistic dynamics of multitask SFT. DaMo's consistent 2.57–3.38% improvement over DML across benchmarks validates the paper's argument that exponential functions are the wrong inductive bias for this problem.
SFTMix (Xiao et al., 2024) optimizes intra-dataset ratios using mixup-style interpolation but cannot handle multi-source data mixing—it operates within a single dataset, not across heterogeneous datasets. Similarly, MoE-based approaches (Zhu et al., 2024) adjust data weights dynamically but do so through heuristic routing mechanisms without a principled optimization criterion.

The gap the paper identifies is thus twofold: no existing method directly models the mapping from data mixture proportions to downstream task performance (rather than to loss), and no existing method can handle the complex, nonlinear, and non-monotonic interactions that arise when fine-tuning on heterogeneous datasets targeting diverse capabilities.

How This Paper Positions Itself

The paper frames its contribution through what it calls the Downstream Task Performance Prediction (DaPP) paradigm. Rather than predicting intermediate metrics like training loss and hoping they correlate with task performance, DaPP directly trains a neural network to map from (data mixture proportions, training steps) to actual benchmark scores. This is a crucial conceptual shift with two immediate consequences:

First, it eliminates the loss-metric mismatch. The paper explicitly connects this to broader observations about the disconnect between training objectives and downstream performance (citing Huang et al., 2019 and Isik et al., 2024). By making the neural network's output directly be the quantity of interest (PhoneAgentBench scores, or BFCL-v3 accuracy, or average benchmark performance), DaMo aligns optimization with evaluation.

Second, it leverages the universal approximation capacity of neural networks to model the complex interaction patterns visible in Figure 3. The paper's central hypothesis is that an MLP with sufficient capacity can learn to represent antagonistic dataset interactions, overfitting regimes, and non-monotonic scaling behavior from sparse samples—patterns that parametric exponential functions structurally cannot capture. The empirical validation of this hypothesis is the R² = 0.81 achieved with only 250 sampled mixtures (Table 2), which the paper notes "indicates that the performance of MLLM on downstream tasks has an inherent connection with the characteristics and mixing patterns of training data, and DaMo learns this mapping via neural networks."

The paper also positions itself at the intersection of two orthogonal concerns that are typically treated separately: (a) domain-specific phone agent optimization (via PhoneAgentBench), and (b) preserving general capabilities (via BFCL-v3, MME, OCRBench). Table 3 demonstrates that DaMo can optimize for PhoneAgentBench without catastrophically degrading general benchmarks—in fact, the optimal mixture for PhoneAgentBench also improves general benchmark scores by 13.73% over the base model, suggesting that the diverse training datasets collectively build transferable capabilities.

The Benchmark Gap

A secondary but important motivation for this work is the absence of appropriate evaluation infrastructure. The authors survey existing agent benchmarks and identify systematic limitations (Section 2, Agent Benchmark subsection and Section 3):

Single-dimensional benchmarks: PlanBench (planning only), ToolBench/BFCL (tool use only), ReflectionBench (self-reflection only), LTM Benchmark (memory only)—each evaluates one capability in isolation, but real phone agents must orchestrate all of them.
Unimodal benchmarks: AgentBench and KAgentBench evaluate LLMs as text agents but ignore the multimodal reality of phone interaction where the model must process screenshots, icons, and visual layouts.
GUI-only mobile benchmarks: ScreenSpot-Pro, MobileViews, and MMBench-GUI evaluate visual grounding in mobile interfaces—critical, but they miss planning, tool orchestration, memory, and multimodal dialogue capabilities that define a complete phone agent.
End-to-end benchmarks without granularity: GAIA evaluates general assistant capabilities holistically but provides a single aggregate score that obscures which specific capabilities need improvement.

PhoneAgentBench is thus positioned to fill what the paper calls "a critical gap: the absence of a comprehensive benchmark supporting multimodal interaction while systematically evaluating mobile phone agents across planning, tool usage, memory, and other dimensions." The four evaluation dimensions (MT-Plan, Mobile-FC, ACU, etc.) are designed to map directly onto specific training datasets, creating a closed-loop optimization problem: which training mixture maximizes the weighted combination of these evaluation scores?

The Scalability Question: Do Mixtures Transfer?

A final motivating question concerns transferability. Prior work on pretraining data mixing (DoReMi, BiMix) assumes that optimal mixtures found on a smaller proxy model can be applied directly to the target large model. The paper questions whether this assumption holds for SFT, where model-specific capabilities and failure modes may cause the optimal mixture to shift. Section 5.4 is designed to test this explicitly, but the motivation is set up in Section 4: DaMo's neural network predictor is trained on one specific model (InternVL2.5-4B). Whether and how its predictions transfer to architecturally different models (Qwen2.5VL series, InternVL3) determines the practical utility of the approach, since retraining DaMo from scratch for every new model would eliminate the efficiency advantages. The paper's finding that a linear calibration with only 20 samples suffices to achieve Pearson correlations above 0.9 (Figure 5, bottom row) validates the core assumption that the shape of the mixture-performance landscape is largely model-agnostic, with only a model-specific offset that can be corrected cheaply.

Summary of the Motivation

The paper addresses a concrete bottleneck in the MLLM development pipeline: given a fixed collection of training datasets and downstream evaluation tasks, how do you determine the best data mixture without exhaustive search? Existing methods either optimize the wrong objective (pretraining loss prediction), assume oversimplified parametric forms (exponential decay), or rely on manual heuristics. DaMo proposes to replace these with a directly learned neural surrogate of the mixture-to-performance mapping, trained on sparse samples and extrapolating across the full mixture space. The introduction of PhoneAgentBench simultaneously creates the evaluation infrastructure needed to validate such methods in the mobile agent domain, which has been underserved by existing benchmarks that either miss multimodal interaction or lack multi-dimensional capability coverage. The scalability experiments address the practical concern of amortizing the cost of DaMo training across multiple model deployments.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The paper builds a neural network-based surrogate model—called DaMo (Data Mixing Optimizer)—that learns to predict how well a fine-tuned multimodal LLM will perform on downstream tasks given any combination of training dataset proportions, without actually training the LLM on that combination. The problem it solves is combinatorial: with 12 training datasets, there are over 13 million possible fixed-mix ratios, and exhaustive search through training-and-evaluating each one is computationally impossible. The "shape" of the solution is a two-phase process—first, train a small number of LLM checkpoints under randomly sampled data mixtures to collect training data for the neural network; second, use the trained network to score every possible mixture in the full space and select the highest-scoring one for final LLM training.

3.2 Big-picture architecture (diagram in words)

The system has five major components:

Training datasets — 12 heterogeneous open-source and self-built datasets (220K total instructions) covering function calling, task planning, OCR, NER, multimodal understanding, app recognition, and general instruction following.
Sample generation pipeline — randomly selects data mixture ratios from the fixed-mix space, trains the base MLLM (InternVL2.5-4B) for 1440 steps saving checkpoints every 360 steps, and evaluates each checkpoint on all 10 downstream tasks to produce (mixture, step, score) tuples.
MLP predictor (DaMo core) — a two-layer multi-layer perceptron that maps from a 13-dimensional input (12 dataset proportions plus training steps) to a 10-dimensional output (predicted scores on each downstream benchmark). Trained on the sampled tuples.
Extrapolation engine — iterates over all $C_{12+16-1}^{12-1}$ possible mixtures in the fixed-mix space, uses the MLP to predict performance for each, sorts by predicted score, and selects the top-k mixtures.
Final MLLM training — trains the base model on the selected optimal mixture and evaluates on all benchmarks.

Information flows as follows: random mixtures are sampled → the base MLLM is trained on each mixture, with checkpointed evaluations → the MLP is trained on these (mixture, step, score) pairs → the MLP predicts scores for all possible mixtures → the top-scoring mixture is used to train the final model → this final model is evaluated.

3.3 Roadmap for the deep dive

First, the formal problem formulation (Equation 1 and the combinatorial complexity of the mixture space), because understanding the intractability of brute-force search motivates everything that follows.
Second, the fixed-mix-space pruning and the size calculation, since this defines the search space that DaMo operates over and explains why the approach is feasible.
Third, the performance prediction framework (DaPP) and the neural network architecture, covering how the MLP is trained to map from mixtures to downstream scores, what inputs and outputs it uses, and how the training data is collected.
Fourth, the extrapolation procedure—how DaMo uses the trained MLP to search the full space efficiently and select the optimal mixture, including the algorithm pseudocode.
Fifth, key design decisions and their justifications: why an MLP rather than a parametric function, why the fixed-mix assumption, why these specific training hyperparameters, and how the approach connects to the empirical patterns in Figure 3.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems and methodology paper whose core idea is that a neural network can learn the complex, non-monotonic mapping from data mixture proportions to downstream task performance from sparse samples, enabling efficient optimization of mixture ratios without exhaustive experimentation.

Problem Formulation: The Combinatorial Challenge

The paper formalizes the multitask fine-tuning optimization problem in Section 4.1. The setup involves fine-tuning a multimodal LLM using a mixture of $m$ heterogeneous training datasets, denoted as:

$\mathcal{D} = \bigcup_{i=1}^{m} \mathcal{D}_i$

where $\mathcal{D}$ represents the combined training corpus, formed by taking the union of all $m$ individual datasets $\mathcal{D}_i$ . Each dataset $\mathcal{D}_i$ contains $n_i$ labeled samples, and the total number of samples across all datasets is $N = \sum_{i=1}^{m} n_i$ .

What this represents operationally: You have 12 separate data collections (function calling data, OCR data, task planning data, etc.), each with its own size and content characteristics. The combined training set is simply all these datasets merged together, but during training, the batch composition—how many examples from each dataset appear in each batch—is controlled by a mixing strategy.

The model starts from pretrained parameters $\theta_0$ (InternVL2.5-4B in the primary experiments) and is fine-tuned for a maximum of $T = \lceil N / b \rceil$ training steps, where $b$ is the batch size.

The data mixture proportion is defined as:

$\mathbf{p} = [p_1, p_2, \ldots, p_m]$

where $\mathbf{p}$ is a vector of $m$ non-negative real numbers representing the sampling probabilities for each dataset, $p_i$ is the proportion of training samples drawn from dataset $\mathcal{D}_i$ , and the constraint $\sum_{i=1}^{m} p_i = 1$ ensures the proportions form a valid probability distribution.

What this vector represents concretely: If $\mathbf{p} = [0.3, 0.1, 0.05, \ldots]$ , then for every batch of 16 examples, on average 4.8 examples come from dataset 1, 1.6 from dataset 2, 0.8 from dataset 3, and so on. The actual sampling is stochastic, but the expected proportions match $p_i$ .

The evaluation setup involves $k$ downstream test datasets:

$\mathcal{D}^{test} = \bigcup_{j=1}^{k} \mathcal{D}_j^{test}$

In the paper's primary experiments, $k = 10$ evaluation tasks: the 6 subtasks of PhoneAgentBench (MT-Plan, MM-RR, MM-NER, Mobile-FC, ACU, APP-Rec) plus 4 open-source benchmarks (BFCL-v3, MME-perception, MME-reasoning, OCRBench). Each produces a score $s_j \in [0, 1]$ (all metrics are normalized to percentages 0–100%, with higher values indicating better performance).

The overall average score is defined as:

$S_{\theta} = \frac{1}{k} \sum_{j=1}^{k} s_j$

where $S_{\theta}$ is the arithmetic mean of the $k$ individual task scores, $\theta$ represents the model parameters after fine-tuning, and $s_j$ is the score on the $j$ -th downstream task.

What this computes: A single scalar summary of model quality across all 10 evaluation tasks, equally weighted. This is the objective function that data mixture optimization aims to maximize.

The optimization objective is:

$\mathbf{p}^* = \mathop{\arg\max}_{\mathbf{p} \in \mathcal{P}, t \leq T} \mathbb{E}_{\theta \sim \mathcal{A}(\mathbf{p}, t, \theta_0)} S_{\theta}$

where $\mathbf{p}^*$ is the optimal data mixture we are searching for, $\mathcal{P}$ denotes the complete data mixing space (all possible ways to arrange samples from different datasets into batches), $\mathcal{A}(\mathbf{p}, t, \theta_0)$ represents the fine-tuning process that produces model parameters $\theta$ from initial parameters $\theta_0$ using mixture $\mathbf{p}$ for $t$ training steps, and the expectation $\mathbb{E}$ accounts for stochasticity in training (random seed, data shuffling, dropout).

What this equation asks operationally: Over all possible ways to compose training batches from the 12 datasets, and over all possible training durations up to $T$ steps, which specific batch-composition strategy and stopping point yields the highest expected average score across the 10 downstream tasks?

Why this is computationally intractable in the full space: Without constraints, the data mixing space $\mathcal{P}$ represents all batch-wise permutations—every possible ordering of the $N$ individual training samples into batches of size $b$ . The size of this space is:

$|\mathcal{P}| = \frac{N!}{(b!)^T}$

Plugging in the paper's numbers ( $N = 220,000$ total samples, $b = 16$ batch size, $T = 14,400$ training steps from the hyperparameter table—though the paper states $T = 1440$ steps as the maximum in their actual experiments), this is an astronomically large number. Even enumerating it is impossible, let alone training a model for each configuration.

The paper says explicitly: "Without any constraints, the size of the set $\mathcal{P}$ that represents batch-wise permutations is given by $|\mathcal{P}| = \frac{N!}{(b!)^T}$ , which is computationally intractable."

The Fixed-Mix Space Pruning

To make the problem tractable, the paper introduces two simplifying assumptions that dramatically reduce the search space:

Assumption 1: Disregard the order of samples within the same dataset. In other words, all examples from dataset $\mathcal{D}_1$ are treated as interchangeable—whether example A appears before example B has no effect on model performance. This eliminates factorial terms associated with within-dataset permutations.

Assumption 2: Keep the data mixture fixed throughout the entire training process. Rather than allowing the mixture to change from step to step (e.g., more function-calling data early, more general data late), the proportions $\mathbf{p}$ remain constant across all $T$ training steps.

Under these two assumptions, the space collapses from batch-wise permutations to a much smaller fixed data mixing space denoted $\mathcal{P}_{fix}$ . The only thing that matters now is how many examples from each dataset appear in each batch, not which specific examples or in what order.

The size of $\mathcal{P}_{fix}$ is computed using the stars-and-bars formula from combinatorics (the principle of combination with repetition):

$|\mathcal{P}_{fix}| = C_{m + b - 1}^{m - 1} = \binom{m + b - 1}{m - 1}$

where $m$ is the number of datasets (12), $b$ is the batch size (16), and the expression represents the number of ways to partition $b$ indistinguishable "batch slots" into $m$ distinguishable "dataset buckets."

Plugging in the numbers: $C_{12 + 16 - 1}^{12 - 1} = C_{27}^{11} = \binom{27}{11} \approx 1.3 \times 10^7$ , or about 13 million possible mixtures.

What this number represents concretely: For each of the 16 slots in a training batch, you must assign one of 12 dataset labels. The batch composition is fully determined by the count vector—how many examples from dataset 1, how many from dataset 2, etc. The number of distinct count vectors summing to 16 with 12 categories is $\binom{27}{11}$ , which equals 13,037,895. Each of these count vectors corresponds to a specific data mixture $\mathbf{p}$ (by normalizing counts to proportions).

Why this pruning is essential: The paper reduces an astronomically infinite space to a discrete, enumerable space of 13 million configurations. While 13 million is still too large for brute-force search (each would require training a full MLLM for 1440 steps), it is small enough that a learned surrogate model can be used to predict performance for all configurations and select the best one—which is exactly what DaMo does.

The paper is explicit about the theoretical grounding: "According to the principle of combination with repetition, the size of this fixed data mixing space $\mathcal{P}_{fix}$ is given by $|\mathcal{P}_{fix}| = C_{m+b-1}^{m-1}$ ."

A subtle but important point about what IS and IS NOT included in $\mathcal{P}_{fix}$ : this space captures all possible proportions but does NOT capture all possible total data volumes. The proportions $p_i$ determine what fraction of each batch comes from each dataset, but the total number of training steps $T$ is a separate variable. The mixture $\mathbf{p}$ can be held fixed for $t \leq T$ steps. This means the actual search space for DaMo is $\mathcal{P}_{fix} \times \{1, 2, \ldots, T\}$ —the Cartesian product of all possible mixtures with all possible training durations. The paper accounts for this by including the training step $t$ as an input to the MLP, making the effective search space $(m+1)$ -dimensional (12 mixture dimensions plus step dimension).

The Core Insight: Why Exponential/Power-Law Functions Fail

Before introducing the neural network solution, the paper systematically demonstrates why conventional parametric functions—specifically the exponential and power-law families used in pretraining data mixing—cannot capture the dynamics of multitask fine-tuning. This analysis in Section 4.2 is essential for understanding the design choice to use an MLP.

The paper conducts two diagnostic experiments:

Single-dataset training analysis (Figure 3a). A model is trained exclusively on the Multimodal Understanding (MMU) dataset and evaluated on multiple PhoneAgentBench subtasks at different training checkpoints. Four qualitative patterns emerge:

Enhancement: ACU (Agent Context Understanding) performance monotonically improves with more MMU training. This is the "well-behaved" case that pretraining loss functions CAN model—positive correlation between data volume and downstream performance.
Conflict: APP-Rec (APP Recognition) performance monotonically decreases with more MMU training. Training on MMU data actively degrades the model's ability to recognize mobile apps—a case of negative transfer where one dataset hurts performance on a task associated with a different dataset.
Neutrality: MM-NER (Multimodal NER) performance is essentially flat and uncorrelated with MMU training steps. The MMU dataset contains no relevant signal for named entity recognition.
Overfitting: MT-Plan (Multimodal Task Planning) initially improves with MMU training but then sharply declines beyond some optimal checkpoint—a classic U-shaped or inverted-U curve indicating that MMU data provides useful transfer up to a point, after which harmful overfitting dominates.

Why this breaks parametric functions: Exponential functions of the form $L(t) = L_0 \cdot e^{-\alpha t} + L_{\infty}$ (or power-law variants) are monotonic—they assume performance always improves (or at worst plateaus) with more data. They cannot represent antagonistic interactions where more data from one source actively harms performance on certain tasks. They also cannot capture non-monotonic patterns like the inverted-U for MT-Plan.

Dual-dataset mixture analysis (Figure 3b). The paper trains models on mixtures of two datasets—APP-Rec and MMU—at varying proportions and training steps, then evaluates on the APP-Rec benchmark. The resulting 3D surface (X-axis: training steps, Y-axis: APP-Rec dataset ratio in the mixture, Z-axis: APP-Rec benchmark score) exhibits what the paper calls "non-convex topology with non-monotonic fluctuations along both axes."

What this surface shows concretely: The APP-Rec score varies non-monotonically with both the proportion of APP-Rec data in the mixture AND the training duration. There is no simple "more APP-Rec data = better APP-Rec performance" relationship, because MMU data interacts with APP-Rec learning in complex ways. Some intermediate mixture ratios outperform both extremes, and the optimal ratio shifts depending on training duration.

The paper states: "This nonlinearity fundamentally prevents analytical solutions for Eq. [1] and invalidates conventional exponential and power functions."

The implication for design: Any function that tries to model the mixture-to-performance mapping MUST have sufficient capacity to represent non-monotonic, interactive, and multi-modal surfaces. Neural networks—specifically MLPs with non-linear activations—are the natural choice because they are universal function approximators with no built-in monotonicity assumption.

The MLP Predictor: Architecture and Training

The paper implements the downstream task performance prediction function $f$ as a multi-layer perceptron:

$\hat{\mathbf{s}} = f_{MLP}(\mathbf{p}, t; \theta_0)$

where $\hat{\mathbf{s}} \in [0, 1]^{k}$ is the predicted vector of $k$ downstream task scores, $\mathbf{p} \in [0, 1]^{m}$ is the data mixture proportion vector ( $m = 12$ ), $t$ is the training step (represented as a scalar), and $\theta_0$ implicitly conditions the prediction on the base model's initial capabilities (since all training starts from the same pretrained weights).

Architecture details (from Table 6, Appendix A.1):

Input layer dimension: 13 (12 dataset proportions + 1 training step value)
Hidden layer 1: 100 neurons, ReLU activation
Hidden layer 2: 100 neurons, ReLU activation
Output layer dimension: 10 (predicted scores on the 10 downstream tasks)
Optimizer: Adam
Learning rate: $1 \times 10^{-6}$
Training steps: 1500
Implementation: scikit-learn's MLPRegressor

What this MLP does operationally: It takes a 13-dimensional vector as input—the proportions of the 12 datasets and the training step number—and outputs 10 predicted performance scores. The ReLU activations introduce non-linearity that allows the network to model the complex surface shapes observed in Figure 3. The first hidden layer can learn to detect specific interaction patterns between subsets of datasets (e.g., "when dataset 3 has proportion > 0.2 AND dataset 7 has proportion < 0.1, ACU scores tend to be high"). The second hidden layer can compose these learned features into more complex decision boundaries.

Why a two-layer MLP rather than a deeper network: The paper doesn't explicitly justify the depth choice, but several factors are implicit. With only 250 mixtures × 4 checkpoints = 1000 training samples, deeper networks with more parameters would likely overfit. Two layers of 100 neurons each provide roughly $13 \times 100 + 100 \times 100 + 100 \times 10 = 12,300$ parameters (plus biases), which is reasonable for 1000 training samples. The ReLU activation is chosen over sigmoid/tanh because it avoids vanishing gradients in deeper optimization, though with only two layers this is less critical.

Why 100 neurons in each hidden layer: This is a hyperparameter choice that the paper does not ablate. Given the small sample size (1000 points), 100 neurons per layer gives the model enough capacity to capture complex interactions while leaving enough degrees of freedom in the data to avoid severe overfitting—the R² = 0.81 in 10-fold cross-validation (Table 2) confirms this balance is achieved.

A critical detail about the training data format: The MLP is NOT trained on individual mixture evaluations. Rather, each training sample is a tuple $(\mathbf{p}, t, \mathbf{s})$ where $\mathbf{p}$ is the mixture ratio vector, $t$ is the checkpoint step, and $\mathbf{s}$ is the vector of 10 downstream task scores evaluated at that checkpoint. The 250 randomly sampled mixtures, each evaluated at 4 checkpoints (every 360 steps up to 1440), produce $250 \times 4 = 1000$ training samples for the MLP (Table 2 and Appendix A.1 confirm this).

Why include training step as an input rather than training separate models per step: Including $t$ as an input dimension makes the MLP learn a continuous function over both mixture space and training duration, enabling it to interpolate to unseen step values and predict the optimal stopping point. If the MLP only took $\mathbf{p}$ as input, it would need separate models for each training duration, which would be less data-efficient. The paper's approach treats $t$ as just another feature, leveraging the MLP's ability to model smooth functions over this dimension as well.

The coefficient of determination (R²) as the fitting metric:

The paper evaluates the MLP's predictive quality using R², defined implicitly as:

$R^2 = 1 - \frac{\sum_{i} (s_i - \hat{s}_i)^2}{\sum_{i} (s_i - \bar{s})^2}$

where $s_i$ is the actual score, $\hat{s}_i$ is the predicted score, and $\bar{s}$ is the mean of actual scores.

What R² measures: The proportion of variance in actual downstream scores that is explained by the MLP's predictions. R² = 1 means perfect prediction; R² = 0 means the predictions are no better than always predicting the mean; negative values mean the predictions are worse than the mean.

Why 10-fold cross-validation: The paper uses cross-validation rather than a single train-test split to get a more robust estimate of the MLP's generalization ability given the small sample size (1000 points). The data is partitioned into 10 folds; the MLP is trained on 9 folds and evaluated on the held-out fold, repeated 10 times with different held-out folds; the reported R² is averaged across all 10 folds. This provides a more reliable signal than a single split, especially important since the paper's entire claim rests on the MLP's ability to generalize to unseen mixtures.

The Sample Generation Pipeline

The training data for the MLP is generated through what the paper calls a pilot experiment—small-scale training runs specifically designed to produce (mixture, score) mappings efficiently. Section 5.2 and Appendix A.1 detail this process:

Step 1: Random mixture sampling. 250 data mixtures $\mathbf{p}$ are randomly selected from the $\mathcal{P}_{fix}$ space of $\approx 1.3 \times 10^7$ possible configurations. Since these are discrete mixtures (integer counts per dataset per batch), the sampling procedure must respect the constraint that all proportions correspond to valid count vectors in $\mathcal{P}_{fix}$ .

Step 2: MLLM training on each mixture. For each of the 250 sampled mixtures, the base model InternVL2.5-4B is fine-tuned using the mixture. The training hyperparameters are (from Table 6):

Optimizer: AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , $\epsilon = 1 \times 10^{-6}$
Maximum sequence length: 16384 tokens
Batch size: 16
Gradient accumulation steps: 8 (effective batch size $16 \times 8 = 128$ )
Training steps: 1440 total
Warmup steps: 144 (10% of total steps, linear warmup)
Peak learning rate: $1 \times 10^{-5}$
Weight decay: 0.1
Gradient clipping: 1.0

Why these specific values: The peak learning rate of $1 \times 10^{-5}$ is relatively standard for fine-tuning large models—high enough to make meaningful parameter updates but low enough to avoid catastrophic forgetting of pretrained knowledge. The 10% warmup ratio (144/1440 steps) is a common practice to stabilize early training when gradients can be noisy. Weight decay of 0.1 provides regularization. Gradient clipping at 1.0 prevents exploding gradients, which can occur when mixing datasets of very different characteristics.

Step 3: Checkpointing and evaluation. During each 1440-step training run, checkpoints are saved at every 360 steps—producing checkpoints at steps 360, 720, 1080, and 1440. The paper specifies $\tau = 360$ as the checkpoint interval (implicitly from the statement "saving checkpoints at every $\tau$ steps"). Each of these 4 checkpoints per mixture is evaluated on all 10 downstream tasks, producing the score vector $\mathbf{s}$ .

Step 4: Dataset assembly. The result is $250 \text{ mixtures} \times 4 \text{ checkpoints} = 1000$ data points, each in the format $(\mathbf{p}, t, \mathbf{s})$ —a 13-dimensional input (12 mixture proportions + training step) and a 10-dimensional output (task scores).

Computational cost accounting (Table 2): The paper reports training costs in H20 GPU-hours (NVIDIA H20 GPUs, a datacenter GPU):

50 mixtures: 872 H20-hours
100 mixtures: 1817 H20-hours
150 mixtures: 2581 H20-hours
200 mixtures: 3521 H20-hours
250 mixtures: 4225 H20-hours

The cost scales roughly linearly with the number of mixtures (each mixture requires a full training run of 1440 steps on 8 GPUs). The paper settles on 250 mixtures because "when the number reaches 250, the fitting score of the MLP gradually converged. Considering the training cost, we stopped further experiments."

The efficiency calculation: At the convergence point, 250 mixtures × 4 checkpoints each = 1000 training points for the MLP. These 1000 points represent only $1000 / 1.3 \times 10^7 \approx 0.0077\%$ of the full $\mathcal{P}_{fix}$ space, yet achieve R² = 0.81. This is the key efficiency claim: the MLP learns the structure of the performance landscape from a tiny fraction of the space, suggesting the underlying mapping has relatively low intrinsic dimensionality.

Why Fit Score Converges: The Latent Structure Hypothesis

Table 2 shows a clear pattern in how R² evolves with more training samples:

Number of fitting samples	R²
50 (200 checkpoints)	0.58
100 (400 checkpoints)	0.57
150 (600 checkpoints)	0.74
200 (800 checkpoints)	0.78
250 (1000 checkpoints)	0.81

Key observation: The jump from 100 to 150 samples produces the largest improvement (0.57 → 0.74), and gains diminish beyond 200. This suggests that roughly 150–200 sampled mixtures (600–800 checkpoints) are sufficient to capture the major modes of variation in the performance landscape, with additional samples providing diminishing refinement.

What this implies about the performance landscape: The fact that an MLP can achieve R² = 0.81 from only 0.0077% of the space implies that the mapping from data mixture to downstream performance is not arbitrary or chaotic—it has substantial smoothness and low-dimensional structure. Datasets fall into functional categories where proportions within categories have similar effects on specific downstream tasks. The MLP learns these categories and their interaction patterns.

The paper explicitly interprets this: "This indicates that the performance of MLLM on downstream tasks has an inherent connection with the characteristics and mixing patterns of training data, and DaMo learns this mapping via neural networks."

Extrapolation: Finding the Optimal Mixture

Once the MLP $f_{MLP}$ is trained, the optimization problem simplifies dramatically. The reformulated objective from Equation 1 becomes (Equation 4 in the paper):

$\mathbf{p}^*_{fix} = \mathop{\arg\max}_{\mathbf{p} \in \mathcal{P}_{fix}, t \leq T} \frac{1}{k} \sum_{j=1}^{k} f_{MLP}^j(\mathbf{p}, t; \theta_0)$

where $\mathbf{p}^*_{fix}$ is the optimal mixture within the fixed-mix space, $f_{MLP}^j$ is the MLP's predicted score for the $j$ -th downstream task, and the overall objective is the average of the $k$ predicted scores across all tasks.

What this computes operationally: For every possible combination of (mixture $\mathbf{p}$ , training step $t$ ) in the discretized space $\mathcal{P}_{fix} \times \{1, \ldots, T\}$ , the MLP predicts the downstream performance vector, computes the average score, and identifies the combination that maximizes this average. The MLP's inference cost is negligible—a forward pass through a small neural network with only ~12,300 parameters takes microseconds—so scoring all $1.3 \times 10^7$ possible mixtures is computationally trivial, requiring only a few seconds on a CPU.

The algorithm (from Appendix B):

Initialize the training set $\mathcal{M} = \emptyset$ for the MLP.
Randomly sample a small subset $\mathcal{P}_{mlp} \subset \mathcal{P}_{fix}$ of mixtures.
For each mixture $\mathbf{p}^i$ $p^{i}$ in $\mathcal{P}_{mlp}$ $P_{m l p}$ :
- Train the MLLM with mixture $\mathbf{p}^i$ for $T$ steps, saving checkpoints at intervals.
- Evaluate each checkpoint on downstream tasks to obtain score vectors $\mathbf{s}$ .
- Add all $(\mathbf{p}^i, t, \mathbf{s})$ tuples to $\mathcal{M}$ .
Fit the MLP: $f_{MLP} \leftarrow fit(\mathcal{M})$ .
Find the optimal mixture and step: $\mathbf{p}^*, t^* \leftarrow \arg\max_{\mathbf{p} \in \mathcal{P}_{fix}} f_{MLP}(\mathbf{p}, t)$ .
Train the final MLLM: $\theta^* \leftarrow \text{Trainer}(\mathcal{D}, \mathbf{p}^*, t^*, \theta_0)$ .
Evaluate: $\hat{\mathbf{s}} \leftarrow \text{Evaluator}(\theta^*, \mathcal{D}^{test})$ .
Return $\theta^*$ and $\hat{\mathbf{s}}$ .

Top-k selection in practice: The paper doesn't just pick the single top-scoring mixture. Section 5.3 describes selecting "the top 50 data mixtures with the best predicted performance" and training all 50, then reporting the best one. Figure 4(b) shows the distribution of actual scores for these top-50 mixtures, demonstrating that DaMo's predictions are well-calibrated enough that most of the highly-ranked mixtures genuinely outperform the baselines.

Why top-50 rather than just top-1: This provides robustness against MLP prediction errors. Even if the absolute #1 predicted mixture turns out to be suboptimal due to prediction noise, the true optimal mixture is likely within the top-50. Training on 50 mixtures and selecting the best actual performer costs more than training on just the #1 predicted mixture, but the paper's cost accounting suggests this is worthwhile as a validation step. The key point is that 50 is vastly smaller than 13 million.

Handling the training step dimension: The MLP predicts performance at any training step $t$ , not just the specific checkpoint steps (360, 720, etc.) seen during training. This means the extrapolation procedure can identify not just the optimal mixture but also the optimal stopping point—the mixture might produce peak performance at step 842 rather than a round checkpoint value. The paper selects the checkpoint with the best predicted score, which in practice is one of the discrete checkpoint steps since those are what can be evaluated.

Scalability Extension: Transferring DaMo Across Models

Section 5.4 and Appendix C describe how DaMo, originally trained on InternVL2.5-4B, is adapted to other model families with minimal additional cost. This is a two-phase process:

Phase 1: Direct transfer assessment. DaMo's predictions from the original InternVL2.5-4B model are evaluated on target models (Qwen2.5VL-3B-Instruct, Qwen2.5VL-7B-Instruct, InternVL3-14B) by training a few random mixtures on the target model and computing the Pearson correlation $r$ between DaMo's predicted scores and the target model's actual scores:

$r = \frac{\sum_i (\hat{s}_i - \bar{\hat{s}})(s_i - \bar{s})}{\sqrt{\sum_i (\hat{s}_i - \bar{\hat{s}})^2} \sqrt{\sum_i (s_i - \bar{s})^2}}$

where $\hat{s}_i$ is DaMo's predicted score for mixture $i$ , $s_i$ is the target model's actual score, and bars denote means.

What Pearson correlation measures in this context: Whether mixtures that DaMo predicts to be good (high predicted score) actually ARE good (high actual score) on the target model. $r = 1$ would mean perfect rank-ordering—the best mixture for InternVL is also the best for Qwen. $r = 0$ means no relationship—optimal mixtures do not transfer at all.

The paper reports correlations of 0.75–0.95 across model transfers (Figure 5, top row), which is strong evidence that the shape of the performance landscape is largely model-agnostic. The paper states: "This suggests that optimal mixtures identified for the base model likely remain near-optimal for the target models."

Phase 2: Linear calibration. Since different models have different absolute performance levels (Qwen2.5VL-7B might achieve higher absolute scores than InternVL2.5-4B on the same mixture), a simple linear transformation corrects for model-specific offsets:

$g = f(\cdot) \mathbf{W} + b$

where $g$ is the calibrated predictor for the target model, $f(\cdot)$ is the original DaMo's predicted score vector, $\mathbf{W}$ is a learned weight matrix, and $b$ is a learned bias vector.

What this linear calibration does: It learns an affine transformation of DaMo's predictions that accounts for the target model's different baseline capabilities. If Qwen2.5VL-7B consistently scores 10 points higher than InternVL2.5-4B across all mixtures, the bias term captures that. If certain types of mixtures disproportionately benefit Qwen, the weight matrix captures that pattern.

The calibration requires only 20 training samples—20 randomly selected mixtures are trained on the target model, and the resulting actual scores are used to fit the linear transformation parameters. After calibration, Pearson correlations improve to above 0.90 across all model transfers (Figure 5, bottom row).

Why linear calibration is sufficient rather than retraining DaMo: The underlying data mixture effects—which datasets enhance which capabilities, which create conflict, optimal ratios—appear to be fundamental properties of the tasks and datasets, not the specific model architecture. The model-specific differences are largely in overall capability level and perhaps some relative strengths/weaknesses, which can be captured by a simple linear shift and scaling of the predicted scores. That this works with only 20 calibration samples (20 × 1440 steps of training per target model = a few hours of GPU time) makes DaMo practically transferable at very low cost.

Cost comparison: Training DaMo from scratch on a new model would require the full 250 mixtures × 1440 steps = 4225 H20-hours. The calibration approach requires 20 mixtures × 1440 steps ≈ 338 H20-hours—a ~12.5× reduction. The paper's reported improvement from pre-mapping correlations (0.75–0.85) to post-mapping correlations (0.90+) demonstrates that this small investment provides substantial alignment benefits.

Design Choice: Why MLP is Better Than Exponential/Power-Law Parametric Functions

The paper's central design choice—using a neural network rather than a parametric function—is justified through the empirical analysis in Section 4.2 but deserves explicit consolidation here.

What parametric functions would predict: Exponential functions of the form $L(\mathbf{p}, t) = A(\mathbf{p}) \cdot e^{-B(\mathbf{p}) \cdot t} + C(\mathbf{p})$ or power-law variants predict that performance monotonically improves with training steps and varies smoothly with data proportions. The parameters $A$ , $B$ , $C$ are themselves functions of the mixture $\mathbf{p}$ , often modeled as linear combinations: $A(\mathbf{p}) = \sum_i a_i p_i$ .

Why this fails for multitask SFT:

Monotonicity violation: APP-Rec performance decreases with MMU training (Figure 3a, Conflict pattern)—an exponential function with positive $B$ (decay) would predict improvement, and with negative $B$ (growth) would predict worsening, but neither can capture improvement-then-decline (MT-Plan overfitting).
Interaction terms: Parametric models that express mixture effects as linear combinations of per-dataset contributions ( $L(\mathbf{p}) = \sum_i w_i p_i + \text{const}$ ) cannot capture antagonistic interactions where the effect of dataset A depends on the presence of dataset B. Figure 3b explicitly shows such interactions for APP-Rec + MMU.
Multi-modality: The performance surface in Figure 3b is non-convex, meaning there can be multiple local optima—a parametric function with convex form (as most exponential/power-law models are) would have a single global optimum and miss alternative high-performing regions.

What the MLP gains: ReLU activations create piecewise-linear functions that can approximate arbitrary non-linear, non-monotonic, multi-modal surfaces. The universal approximation theorem guarantees that with sufficient neurons, an MLP can represent any continuous function on a compact domain—which the mixture-proportion simplex certainly is. The question is whether 1000 training samples suffice; the R² = 0.81 answer is that they do, for the purpose of identifying high-performing mixtures.

A limitation the paper acknowledges: The MLP provides no interpretable relationship between mixture proportions and performance. Unlike a parametric model where coefficients have semantic meaning (e.g., "dataset 3 contributes positively to ACU performance"), the MLP is a black box. The paper does not analyze feature importance or attempt to extract interpretable rules from the trained network—it treats DaMo purely as a prediction-and-optimization tool.

Design Choice: Fixed-Mix vs. Dynamic/Curriculum Mixing

The fixed-mixture assumption ( $\mathcal{P}_{fix}$ rather than the full $\mathcal{P}$ ) is a deliberate simplification that the paper acknowledges as a limitation in Appendix D. The justification is pragmatic rather than principled:

What dynamic mixing would require: Allowing the mixture to change during training (e.g., $\mathbf{p}(t) = f(t)$ for some scheduling function) would expand the search space from discrete to continuous in the time dimension—rather than a single vector $\mathbf{p}$ , you would need to optimize a function $\mathbf{p}: [0, T] \rightarrow \Delta^{m-1}$ (a path through the probability simplex). This is a functional optimization problem that is much harder than the vector optimization DaMo solves.

Why fixed mixing is still valuable: The paper's results (Tables 3–5) demonstrate that fixed optimal mixtures discovered by DaMo substantially outperform heuristic baselines and DML. This suggests that for the datasets and tasks studied, the benefits of dynamic mixing over optimal static mixing may be second-order. The paper explicitly flags dynamic mixing as future work in Appendix D, proposing to "integrate Monte Carlo Tree Search (MCTS) with reinforcement learning to iteratively determine stage-specific data mixtures," but notes that "preliminary attempts to relax these assumptions... remain exploratory."

The implicit design philosophy: Fixed mixing captures the first-order effect (what datasets to include and in what proportions), while dynamic scheduling captures second-order effects (when to emphasize which datasets during training). Optimizing the first-order effect with a principled method already yields substantial gains; the second-order effect is left for future work. This is a defensible research strategy—solve the simpler problem thoroughly before tackling the harder one.

Design Choice: Training Steps as an MLP Input Rather Than a Separate Optimization Variable

The paper includes training step $t$ as an input dimension to the MLP rather than training separate MLPs for each checkpoint or treating step selection as a post-hoc choice. This has several advantages:

Unified prediction surface: The MLP learns a continuous function $f(\mathbf{p}, t)$ that can interpolate to unseen step values. If the optimal stopping point for a mixture is at step 842 rather than a round checkpoint, the MLP can predict that. In practice, the paper selects from discrete checkpoints (every 360 steps), so this interpolation capability is not fully exploited, but the architecture supports it.

Data efficiency: All 1000 training points (250 mixtures × 4 checkpoints) contribute to learning the MLP parameters. If separate MLPs were trained for each checkpoint, each would have only 250 points—likely insufficient for the 12-dimensional input space.

Computational efficiency at extrapolation time: A single forward pass of the MLP predicts scores for all 10 tasks at any step $t$ , rather than requiring separate model evaluations per step. This matters because the extrapolation procedure must evaluate $1.3 \times 10^7$ mixtures × multiple possible training steps, and even microsecond-level differences multiply to seconds or minutes of total runtime.

The tradeoff: The MLP must learn to disentangle the mixture effect from the training-step effect. If the relationship between training step and performance varies qualitatively across different mixtures (which Figure 3b suggests it does), the MLP needs enough capacity to represent these interaction terms. The 100-neuron hidden layers provide this capacity, as evidenced by the R² = 0.81.

4. Key Insights and Innovations

Innovation 1: Abandoning the Loss Prediction Paradigm for Direct Performance Modeling

The fundamental conceptual move in this paper is rejecting the intermediate proxy of training or validation loss as a guide for data mixture optimization and instead building a model that directly predicts downstream task scores. This is more than a methodological substitution—it's a diagnosis of why an entire research thrust had reached its ceiling.

Prior work on data mixing—whether for pretraining (DoReMi, BiMix, ODM) or fine-tuning (DML)—operates within a common framework: fit a parametric function (typically exponential or power-law) to predict some form of loss as a function of data mixture proportions, then optimize the mixture to minimize that predicted loss. The implicit assumption is that lower loss translates to better downstream performance. This assumption is reasonable during pretraining, where the training objective (next-token prediction) aligns closely with what the loss measures, and where scaling laws have established monotonic relationships between loss reduction and capability improvement.

The paper's critical insight is that this assumption fundamentally breaks in multitask SFT. Figure 3(a) is the diagnostic that justifies the entire paradigm shift: four downstream tasks evaluated on the same model trained with the same single dataset exhibit qualitatively different relationships with training progress. One improves monotonically (Enhancement), one degrades monotonically (Conflict), one is uncorrelated (Neutrality), and one exhibits inverted-U overfitting (MT-Plan). These patterns cannot be captured by any monotonic loss function, no matter how well-parameterized. More importantly, they demonstrate that the relationship between training data and downstream performance is mediated by task interactions—transfer, interference, and overfitting dynamics—that are invisible to loss but dominate the outcomes practitioners actually care about.

The field's prior response to the loss-performance mismatch had been largely ad hoc: post-hoc checkpoint selection (training long and picking the best checkpoint based on downstream evaluation), manual mixture tuning through expensive iteration (as documented for LLaMA3 and Tulu3), or simply ignoring the mismatch and hoping for the best. DaMo's contribution is to make the mismatch the central object of study and to propose a general-purpose solution: learn the mapping directly from mixture to performance using a flexible function approximator, bypassing loss entirely. This is a fundamental shift rather than an incremental refinement because it changes what the optimizer optimizes—the objective function itself is different, not just the optimization algorithm.

The significance extends beyond the immediate performance gains (3.38% over DML). By demonstrating that performance can be predicted from mixture proportions alone with R² = 0.81 from sparse samples (Table 2), the paper establishes that there exists a learnable, relatively low-dimensional structure relating training data composition to downstream capabilities—a finding that has implications for how we think about transfer learning, catastrophic interference, and the design of training curricula. This is a diagnostic contribution as much as a methodological one: it says that the shape of the mixture-performance landscape is not arbitrary noise but a coherent object that can be modeled, studied, and eventually understood mechanistically.

Innovation 2: The MLP as a Universal Surrogate for the Mixture-Performance Landscape

The second innovation is the choice of a neural network as the surrogate model, which is both a technical decision and a conceptual statement about the problem structure. Where prior work insisted on interpretable parametric forms (exponential, power-law) with explicit assumptions about monotonicity and separability, DaMo's MLP makes no structural assumptions about how datasets interact—it lets the data dictate the functional form.

This matters because the paper provides explicit evidence that the "well-behaved" assumptions of parametric models are violated in practice. Figure 3(b) shows the performance surface for APP-Rec as a function of APP-Rec + MMU mixture proportions, revealing what the paper calls "non-convex topology with non-monotonic fluctuations along both axes." A parametric model with a convex form would miss local optima. A model with monotonicity baked in (as exponential decay does) would miss the inverted-U pattern. A model assuming additive separability (each dataset contributes independently) would miss the interaction effects visible in the surface's curvature.

The MLP's universality is not just about capacity—it's about inductive bias. By choosing a model with essentially no built-in assumptions about the shape of the performance landscape, DaMo implicitly claims that we don't know enough about multitask fine-tuning dynamics to impose structural constraints a priori. This is a substantive epistemic position: the paper argues that the field's understanding of how datasets interact during SFT is so limited that flexible black-box models outperform carefully designed parametric ones. The fact that R² reaches 0.81 with only 1000 training samples (Table 2) validates this position—if the true underlying function were well-approximated by exponentials, the MLP would either replicate that form or overfit; the fact that it achieves good generalization suggests it's learning genuine structure that parametric models miss.

There's a deeper implication here about when neural networks are appropriate for scientific modeling. In domains where the underlying physics or dynamics are well-understood (e.g., scaling laws for pretraining loss), parametric models with strong inductive biases are preferable because they generalize reliably with few data points and provide interpretable parameters. In domains where the underlying dynamics are poorly understood and likely involve complex interactions (multitask SFT), flexible function approximators may be more scientific because they don't encode false assumptions. DaMo's success is an argument that data mixture optimization for SFT belongs in the latter category—at least at the current state of knowledge.

This is an incremental advance in methodology (neural networks as surrogates are common in many fields) but a fundamental reframing for this specific problem, since it rejects the dominant paradigm of loss-based parametric fitting that had structured essentially all prior work on data mixing.

Innovation 3: Empirical Proof That Optimal Mixtures Transfer Across Model Families

The third innovation is the discovery that the optimal data mixture landscape is largely model-agnostic—a finding with significant practical and theoretical implications that goes beyond the specific DaMo implementation.

The paper's transfer experiments (Section 5.4, Figure 5) show that DaMo trained on a 4B-parameter InternVL model achieves Pearson correlations of 0.75–0.95 when predicting the relative performance of different mixtures on architecturally different models (Qwen2.5VL-3B, Qwen2.5VL-7B, InternVL3-14B). With only 20 calibration samples to fit a linear correction, correlations improve to above 0.90, meaning the rank-ordering of mixtures is nearly preserved across model families, scales, and training recipes.

Prior work on data mixing for pretraining (DoReMi) had assumed this transfer property—running mixture optimization on a small proxy model and applying the result to the target large model—but had not validated it for SFT, where model-specific capabilities and failure modes could plausibly cause the optimal mixture to shift substantially. A 7B model might benefit more from complex task-planning data than a 3B model because it has greater capacity to learn compositional reasoning; conversely, a 3B model might need more function-calling data because it struggles to generalize tool-use patterns from limited examples. The paper's finding that neither of these intuitions dominates—that the underlying data mixture effects are sufficiently robust to survive model changes—is a non-obvious empirical result, not a foregone conclusion.

The practical significance is clear: it means DaMo can be trained once on a small, cheap model and deployed across a family of larger or different-architecture models with minimal recalibration. The computational cost of data mixture optimization is thus amortized over multiple downstream deployments.

The theoretical significance is more subtle. The transfer result suggests that what determines a good data mixture is fundamentally about the tasks and datasets themselves—their content, difficulty, and mutual relationships—rather than about the specific model processing them. The mixture-performance landscape is shaped primarily by what the data teaches and how different teaching signals interact, and only secondarily by how well a particular model can absorb those signals. If confirmed more broadly, this would shift the field's focus from model-specific mixture tuning toward dataset characterization—understanding the "curriculum content" of training data independent of the learner.

This is a fundamental empirical finding (not an incremental refinement) because it establishes a transfer property that was previously assumed but unverified for the SFT setting, and because it constrains theories about where the difficulty in multitask fine-tuning originates.

Innovation 4: Closed-Loop Optimization Through PhoneAgentBench

The fourth contribution is the creation and use of PhoneAgentBench as a closed-loop optimization target—a benchmark explicitly designed so that its evaluation dimensions map onto specific training datasets, enabling systematic mixture optimization in a way that general-purpose benchmarks do not support.

This is methodologically significant because it addresses a structural problem in MLLM development: most benchmarks are designed for evaluation, not optimization. They measure whether a model is good, but they don't tell you why it's good or how to make it better. PhoneAgentBench's four capability dimensions (multimodal task planning, device-native tool usage, multimodal memory, screen context understanding) are each tied to identifiable training data types, creating what amounts to a differentiable objective for data mixture optimization. When DaMo predicts that increasing the proportion of Task-Planning data from 5% to 15% will improve MT-Plan scores by X points, that prediction is directly testable because the training dataset and evaluation task are aligned.

Existing benchmarks lack this property. BFCL-v3 measures function calling, but its relationship to specific training datasets is diffuse—many different data types might improve function calling in non-obvious ways. MME-reasoning measures general reasoning, but there's no single training dataset that directly teaches that capability. By constructing PhoneAgentBench dimensions that mirror the training data taxonomy, the paper creates a system where optimization and evaluation speak the same language—a necessary condition for the kind of systematic mixture optimization that DaMo enables.

The construction methodology itself embodies a design principle: benchmarks should be built not just to measure but to guide improvement. The paper's explicit documentation of data construction procedures (Appendix A.2) and the linking of each evaluation task to specific training data categories make PhoneAgentBench more of an optimization scaffold than a traditional static benchmark. This is an incremental conceptual contribution to benchmark design—the idea of alignment between training taxonomy and evaluation dimensions is not entirely new, but its systematic application to mobile phone agents and its integration with an automated optimization method represents a concrete advance in closing the loop between measurement and improvement.

Innovation 5: The Negative Result That Dynamic/Curriculum Mixing Is Not (Yet) Necessary

An understated but important contribution is the implicit negative result that fixed data mixtures, when properly optimized, achieve substantial gains—challenging the intuitive assumption that dynamic or curriculum-based mixing strategies are required for strong multitask performance.

The paper acknowledges in Appendix D that dynamic mixture adjustment (changing proportions during training) is a natural extension, and that "preliminary attempts to relax these assumptions... remain exploratory." The fact that fixed optimal mixtures discovered by DaMo achieve 23% absolute improvement over the base model on PhoneAgentBench (44.83% → 68.18%, Table 3) and outperform all baselines on general benchmarks suggests that the first-order effect in data mixing is simply getting the proportions right—and that curriculum scheduling, while potentially beneficial, is a second-order refinement.

This is a practically significant finding because dynamic mixing adds substantial complexity: it expands the optimization space from a single vector to a time-varying function, requires designing scheduling policies, and introduces sensitivity to training dynamics that are hard to model. If fixed mixing with DaMo already captures most of the achievable gain, the cost-benefit calculus for dynamic mixing shifts considerably—it may not be worth the engineering investment for many applications.

The result also connects to broader debates about curriculum learning in deep learning. While curriculum strategies have shown benefits in some settings (particularly when training data has clear difficulty gradients), their advantages over well-tuned uniform sampling have been inconsistent. DaMo's success with fixed mixing provides another data point suggesting that dataset selection and proportioning dominate scheduling effects in the multitask fine-tuning regime, at least for the scale and diversity of datasets studied.

This is neither a theoretical advance nor a major empirical breakthrough—it's a pragmatic negative result that constrains where future effort should be directed and validates the fixed-mixing assumption that makes DaMo's optimization problem tractable. Its value lies in establishing that the simpler approach is not just a stepping stone but a genuinely strong baseline in its own right.

5. Experimental Analysis

Evaluation Methodology

Training Datasets. The training corpus comprises 12 datasets—a mix of open-source and internally constructed collections—totaling approximately 220K instructions in both Chinese and English (Appendix A.3, Table 8). The open-source component includes ShareGPT4, NER (aggregated from Chinese-NER-SFT, Sentiment-Analysis, and Few-Shot-NER-SFT), Infinity-MM, OCR (aggregated from Vision-OCR-Financial-Reports-10K, Arxiv-OCR-v0.1-SFT, and Invoices-and-Receipts-OCR-v1), and SuperCLUE-Agent. The self-built component includes Multimodal Instruction Evolution (MMIE), APP Recognition (APP-Rec), Reference Resolution (RR), Multimodal Understanding (MMU), Function Calling (FC), Task Planning (TP), and Image-Text Relevance (ITR). These span mobile-specific capabilities (function calling, task planning, app recognition) and general multimodal skills (OCR, NER, visual understanding, instruction following), creating the heterogeneous mixture landscape that motivates the optimization problem.
Base Model. The primary experiments use InternVL2.5-4B (Chen et al., 2024b) as the base multimodal LLM. The paper does not provide an explicit justification for this specific model choice beyond its role as the foundation for the DaMo fitting pipeline, but the scalability experiments (Section 5.4) extend to Qwen2.5VL-3B-Instruct, Qwen2.5VL-7B-Instruct (Bai et al., 2025), and InternVL3-14B (Zhu et al., 2025), establishing that DaMo is not tied to a single architecture. The 4B scale is chosen as the "pilot" model for training DaMo because it is large enough to exhibit meaningful multitask learning dynamics but small enough that training 250 mixtures × 4 checkpoints (4225 H20-hours, Table 2) remains feasible.
Metrics. All evaluation metrics are normalized to percentages in the range 0–100%, with higher values indicating better performance. The specific metric varies by task: MT-Plan uses the T-Eval planning evaluator (longest ordered action sequence from similarity-matched pairs, producing a plan quality score); MM-NER uses entity F1-score; Mobile-FC uses exact-match accuracy (1 point for perfect function name + parameter match, 0 otherwise); ACU uses BLEU score of the de-anaphorized output against the reference; APP-Rec uses exact-match accuracy of the predicted app name; open-source benchmarks (BFCL-v3, MME-perception, MME-reasoning, OCRBench) use their standard evaluation protocols. The primary aggregate metric—"overall average score"—is the unweighted arithmetic mean across all evaluated tasks: $S_\theta = \frac{1}{k} \sum_{j=1}^{k} s_j$ , where $k = 10$ in the full evaluation setting (6 PhoneAgentBench tasks + 4 open-source benchmarks). When reporting PhoneAgentBench-only results, the average is taken over the 6 constituent tasks.
Baselines. The paper compares against three baselines. Uniform Mixture: all 12 datasets are sampled with equal probability, regardless of their sizes ( $p_i = 1/12$ for all $i$ ). Natural Mixture: sampling probability for each dataset is proportional to its number of samples ( $p_i = n_i / N$ ), meaning larger datasets dominate training. Data Mixing Laws (DML) (Ye et al., 2024): an exponential-function-based method that fits parametric scaling laws to predict downstream task loss from data mixture proportions, then selects the mixture that minimizes predicted loss. DML is identified as the most directly comparable state-of-the-art method and the primary baseline against which DaMo's performance gains are measured.
Generation Budget / Compute Accounting. The paper measures computational cost for Mixture Sampling in H20-hours (hours of computation on NVIDIA H20 GPUs). For DaMo's fitting phase, this cost scales with the number of sampled mixtures: 250 mixtures require 4225 H20-hours (Table 2). For final model training, all methods (DaMo, DML, Uniform, Natural) train the MLLM for the same maximum number of steps (1440) on 8 NVIDIA H20 GPUs with a batch size of 16 and gradient accumulation of 8 (effective batch size 128), making the per-mixture training cost identical across methods. The critical efficiency comparison is therefore not in the final training step but in how many candidate mixtures must be trained and evaluated before the optimal one is found—DaMo requires 250 pilot mixtures to fit the MLP (which is then amortized), while brute-force search would require millions, and DML requires its own fitting procedure (whose cost is not directly compared).
Cross-Validation / Statistical Protocol. For evaluating the MLP's predictive quality, the paper uses 10-fold cross-validation on the 1000 collected (mixture, step, score) tuples, reporting the coefficient of determination R² averaged across folds (Table 2). This is the standard approach for assessing regression model generalization with small sample sizes and provides a more robust estimate than a single train-test split. However, for the final downstream performance comparisons (Tables 3–5), no statistical significance testing, confidence intervals, or multiple-run variance estimates are reported—the paper reports point estimates from single training runs, which limits the ability to distinguish genuine differences from training noise, particularly for small-magnitude improvements.

Main Quantitative Results

DaMo's Predictive Accuracy: The MLP Fit Quality

The fundamental prerequisite for DaMo's approach is that the MLP can predict downstream task performance for unseen data mixtures with sufficient accuracy to guide optimization. Table 2 reports how the MLP's R² evolves with the number of fitting samples:

Mixtures sampled	Checkpoints (total samples)	H20-hours	R² (10-fold CV)
50	200	872	0.58
100	400	1817	0.57
150	600	2581	0.74
200	800	3521	0.78
250	1000	4225	0.81

Several patterns are noteworthy. First, the jump from 100 to 150 mixtures produces a disproportionate improvement in fit quality (R² from 0.57 to 0.74, a 0.17 absolute gain), while further increases yield diminishing returns. This suggests that approximately 150 mixtures (600 checkpoints) capture the major modes of variation in the performance landscape, and the MLP's learning saturates thereafter. Second, the final R² of 0.81—achieved from only ~0.0077% of the full mixture space—is the empirical foundation for DaMo's core claim: that the mapping from data mixture to downstream performance has learnable structure that can be exploited for optimization. An R² of 0.81 means that 81% of the variance in actual downstream scores across mixtures is predictable from the mixture vector alone, leaving 19% attributable to factors not captured (training stochasticity, unmodeled dataset characteristics, measurement noise). Third, there is a curious non-monotonicity in the progression: 50 mixtures achieve R² = 0.58, but 100 mixtures achieve R² = 0.57—a slight degradation. The paper does not comment on this, but it is consistent with the MLP encountering new regions of mixture space at 100 samples that are harder to predict, temporarily reducing cross-validation performance before additional samples at 150 bring the model back on track.

Probability Distribution of Mixture Performance (Figure 4)

Figure 4 provides a critical visualization that contextualizes the optimization challenge and DaMo's effectiveness. Figure 4(a) shows the distribution of overall average scores across different checkpoints when MLLMs are trained on randomly sampled data mixtures. The distribution approximates a normal curve centered at a moderate performance level. The paper notes two key characteristics: "(1) The absence of a right-side long tail indicates that excellent data mixtures are extremely sparse. (2) The performance of random mixture is predominantly mediocre, and baseline methods (vertical dashed line) show no discernible advantage." The vertical dashed lines marking Uniform and Natural mixture performance fall squarely within the central mass, confirming that these heuristics are effectively equivalent to picking a random mixture—they provide no optimization benefit over chance.

Figure 4(b) shows the distribution of actual scores for the top 50 mixtures predicted by DaMo as optimal. The distribution is shifted significantly rightward compared to the random distribution, with the bulk of the mass concentrated at higher scores. Critically, DaMo's top-1 predicted mixture achieves performance at the extreme right tail of what random sampling would produce—a region so sparsely populated that brute-force random search would require orders of magnitude more trials to hit comparable quality. This visualization makes the optimization argument visually: DaMo doesn't just identify mixtures that are somewhat better than average; it identifies mixtures in a performance regime that is essentially inaccessible through random or heuristic search at realistic sampling budgets.

PhoneAgentBench Results: DaMo vs. Baselines (Table 3)

Table 3 presents the main PhoneAgentBench results using the top-1 predicted mixture from DaMo to train the MLLM (InternVL2.5-4B). The top-1 selection means DaMo predicted which single mixture would yield the highest overall average score, and that mixture was then trained and evaluated—this is the most aggressive test of DaMo's predictive accuracy since it relies on the single best prediction rather than a top-k strategy.

The base model (InternVL2.5-4B without any SFT) achieves an overall average score of 44.83% on PhoneAgentBench. The Uniform Mixture baseline reaches 62.94%—an 18.11 percentage point improvement over the base model, demonstrating that SFT itself provides substantial gains regardless of mixture strategy. The Natural Mixture baseline achieves a comparable 63.28%. DML (Ye et al., 2024) reaches 65.16%, providing a modest 2.22 percentage point improvement over Uniform.

DaMo achieves 68.18%, representing:

23.35 percentage point improvement over the no-SFT base model (44.83% → 68.18%)
5.24 percentage point improvement over Uniform Mixture (62.94% → 68.18%)
3.02 percentage point improvement over DML (65.16% → 68.18%), which the paper rounds to 3.38% in the abstract—this discrepancy suggests the abstract reports a slightly different aggregate (possibly the average across multiple runs or the average improvement per-subtask rather than the overall average improvement).

A critical detail: where does the 3.38% figure in the abstract come from? The abstract states "DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods." Reading Table 3 carefully: DaMo's overall average on PhoneAgentBench is 68.18% vs. DML's 65.16%. The absolute improvement is 68.18 − 65.16 = 3.02 percentage points. The relative improvement would be (68.18 − 65.16) / 65.16 = 4.63%. Neither calculation yields exactly 3.38%. The 3.38% likely refers to the average improvement across all individual PhoneAgentBench subtasks rather than the overall average—computing the per-task differences and averaging them could produce this figure, though the paper does not make this explicit.

The per-task breakdown in Table 3 (the specific scores for MT-Plan, MM-RR, MM-NER, Mobile-FC, ACU, APP-Rec) is not shown in the main text but is referenced—the paper states DaMo achieves "more than 23% (from 44.83% to 68.18%) improvement over the native model (without SFT) on PhoneAgentBench, surpassing both uniform and natural mixture strategies." The consistency of gains across individual tasks (the paper claims "stable performance gains across almost all tasks") is important because it demonstrates that DaMo's optimization doesn't sacrifice any single capability to boost the average—a risk with aggregate optimization where gains on some tasks could mask losses on others.

General Benchmark Results: DaMo vs. Baselines (Table 3, columns for open-source benchmarks)

Table 3 also reports performance on the four open-source benchmarks (BFCL-v3, MME-perception, MME-reasoning, OCRBench) when using the mixture optimized for PhoneAgentBench. The results show:

Base model (no SFT): overall average of 38.39% across the four benchmarks
DaMo: overall average of 52.12% across the four benchmarks
Improvement: 13.73 percentage points over the base model

This is a non-trivial finding: optimizing for PhoneAgentBench does not catastrophically degrade general capabilities—in fact, it substantially improves them. The paper interprets this as evidence that "the diverse training datasets collectively build transferable capabilities." The training data used for PhoneAgentBench optimization (which includes OCR, NER, general instruction-following, and multimodal understanding datasets) provides enough general signal that even when the optimization objective is domain-specific, the resulting model retains and improves broad competence.

Compared to DML on these open-source benchmarks, DaMo achieves a 2.57% higher average score (this figure is explicitly stated in the abstract and appears consistent with Table 3's open-source benchmark columns). The per-benchmark breakdown (BFCL-v3, MME-perception, MME-reasoning, OCRBench individual scores) is provided in Table 3 but the specific numbers are not quoted in the main text—the paper only reports that "DaMo outperforms DML by 2.57% in terms of average score" across these four benchmarks.

Task-Specific Optimization: BFCL-v3 Results (Table 4)

Table 4 isolates an important capability: DaMo's ability to optimize for a single downstream task rather than an aggregate. When DaMo is used to predict performance on BFCL-v3 only (ignoring all other benchmarks) and select the optimal mixture for that specific task, the results are striking:

Uniform Mixture: 29.32% on BFCL-v3
Natural Mixture: performance not separately quoted but implied to be similar to Uniform
DML: performance not separately quoted
DaMo (BFCL-v3 only): 47.43% on BFCL-v3

This represents an 18.11 percentage point improvement over Uniform Mixture, or a relative improvement of approximately 61.8%. The abstract characterizes this as "DaMo improves the metrics by 12.47% than other methods"—this 12.47% figure is the relative improvement over DML specifically (since DML is the "other method" being compared against in this context), computed as (DaMo_BFCL − DML_BFCL) / DML_BFCL.

Critically, the paper explicitly notes: "Crucially, this enhancement is sustained even in the absence of any task-curated training data." The optimal mixture for BFCL-v3 was discovered from the existing 12 training datasets—none of which were specifically designed or curated for the BFCL-v3 benchmark. This is the strongest evidence for DaMo's core mechanism: the MLP learns to identify which combinations of available training data produce capabilities that transfer to a target evaluation, even when there is no one-to-one mapping between training datasets and evaluation tasks. This is generalization at the meta-level: the mixture optimizer generalizes across tasks even when the base model's training data wasn't designed for those tasks.

The other task-specific optimizations (DaMo targeting MME-perception only, MME-reasoning only, OCRBench only) are marked with asterisks in Table 4, indicating "these scores correspond to different checkpoints, which are optimized by DaMo on a single task." The specific numbers are provided in the table but not discussed individually in the main text. The consistent pattern across all task-specific optimizations is that DaMo substantially outperforms heuristic baselines, with larger gains on some tasks (BFCL-v3) than others.

Why Task-Specific Optimization Beats Aggregate Optimization

A comparison of Tables 3 and 4 reveals an important tension in multitask optimization. In Table 3, DaMo optimizes for the aggregate of PhoneAgentBench + open-source benchmarks, achieving an overall average score of 52.12% on the open-source benchmarks. In Table 4, DaMo optimizes for BFCL-v3 alone, achieving 47.43% on BFCL-v3—which is substantially higher than the BFCL-v3 score achieved by the aggregate-optimized model (implied to be lower than 47.43% since the aggregate optimization spreads improvement across multiple tasks).

This demonstrates the expected trade-off: optimizing for a single task yields higher performance on that task than optimizing for a multi-task aggregate, because the aggregate objective forces compromise. The practical implication is that DaMo can be deployed in two modes: multi-objective mode (optimize the weighted average of all tasks you care about, accepting that no single task will reach its ceiling) or single-task mode (optimize aggressively for one capability, accepting that other capabilities may regress). The paper doesn't explicitly quantify the regression on non-target tasks when optimizing for a single task, which is a limitation—we don't know whether the BFCL-v3-optimized model maintains reasonable performance on MME or OCRBench, or whether single-task optimization catastrophically damages other capabilities.

Scalability: Transfer to Other Models (Table 5, Figure 5)

Table 5 reports the downstream performance when DaMo (originally trained on InternVL2.5-4B) is transferred to three target models: Qwen2.5VL-3B-Instruct, Qwen2.5VL-7B-Instruct, and InternVL3-14B. Two transfer strategies are compared:

Direct transfer (original DaMo): The optimal mixture identified for InternVL2.5-4B is applied directly to the target model without any modification. The results in Table 5 show that this direct transfer achieves "competitive overall average scores" compared to baselines, with specific numbers provided in the table (the main text only characterizes them qualitatively).

Linear-mapped transfer (calibrated DaMo): A linear transformation ( $g = f(\cdot)\mathbf{W} + b$ ) is learned using 20 calibration samples (20 random mixtures trained on the target model, with the resulting scores used to fit the linear correction). The results show that this calibration "further improves" the scores beyond direct transfer.

The Pearson correlation analysis in Figure 5 provides the mechanistic explanation for why this transfer works:

Top row (before calibration): Scatter plots comparing DaMo's predicted scores (trained on InternVL2.5-4B) against actual scores of target models. The Pearson correlations are:
- Qwen2.5VL-3B: implied to be 0.75–0.95 range (exact value not quoted in main text for individual models)
- Qwen2.5VL-7B: same range
- InternVL3-14B: same range
The paper states: "the Pearson correlation coefficients (r) are generally above 0.75, demonstrating the robust cross-model applicability of DaMo." The scatter plots show a clear positive relationship but with visible spread—the predictions are directionally correct (mixtures predicted to be good tend to be good) but have systematic biases (target models may achieve consistently higher or lower absolute scores than predicted).
Bottom row (after linear calibration): After applying the learned linear mapping, correlations increase to "above 0.9" for all model pairs. The scatter plots tighten around the diagonal, indicating that the rank-ordering of mixtures is strongly preserved after accounting for model-specific baseline differences.

What this demonstrates: The underlying shape of the performance landscape—which mixtures are better than which others—is largely model-agnostic. The model-specific differences manifest primarily as additive offsets (some models are simply better at all tasks) and multiplicative scaling (some models benefit more or less from certain dataset interactions), both of which a linear transformation can capture. This validates the core assumption that makes DaMo practically deployable: train the predictor once on a manageable model, apply it with minimal recalibration to production models.

Table 5 compares DaMo's transfer performance against the Uniform Mixture, Natural Mixture, and DML baselines on the target models. DaMo (both direct and linear-mapped) outperforms all three baselines across all target models. The specific scores and margins are in Table 5; the paper summarizes that "directly applying the original DaMo achieves competitive overall average scores" and "using the linear-mapped DaMo, the scores can be further improved."

A subtle but important point: Table 5 reports performance on both PhoneAgentBench and the open-source benchmarks for each target model, demonstrating that the transfer works for both domain-specific and general capabilities. The consistency across evaluation domains strengthens the claim that the mixture-performance landscape is fundamentally about data-task relationships, not model-specific idiosyncrasies.

Ablation Studies and Robustness Checks

Number of MLP fitting samples (Table 2): The paper systematically varies the number of sampled mixtures from 50 to 250, measuring both R² and computational cost. This is not presented as a formal ablation with controlled variables (different numbers of mixtures also imply different numbers of total checkpoints and different coverage of the mixture space), but it serves as the primary sensitivity analysis for DaMo's core hyperparameter. The key finding is that R² converges around 150–200 mixtures (R² stabilizes at 0.78–0.81), and additional samples provide minimal improvement. The cost at convergence (4225 H20-hours) is substantial but the paper argues it is amortized across multiple model deployments through transfer (Section 5.4).

Top-1 vs. top-50 predicted mixtures (Figure 4b): Though not presented as a formal ablation, the paper's choice to train the top 50 predicted mixtures and report the best performer (rather than solely trusting the top-1 prediction) is a de facto robustness check on the MLP's calibration. Figure 4(b) shows the distribution of actual scores for these top-50 mixtures—if the MLP were perfectly calibrated, the top-1 predicted mixture would always be the best actual performer; the fact that top-50 is used suggests some rank-order noise. The paper does not report what proportion of the top-50 mixtures genuinely outperform baselines (the distribution in Figure 4b suggests the majority do, but exact quantification is absent), nor does it compare the performance of the #1 predicted mixture vs. the best among the top-50—such a comparison would reveal the cost of MLP prediction error in terms of lost performance.

Optimization target: PhoneAgentBench aggregate vs. single tasks (Tables 3 vs. 4): This comparison—using DaMo to optimize for the multi-task average versus for individual benchmarks—reveals the expected tension between specialization and generalization. When optimizing for PhoneAgentBench + open-source benchmarks jointly (Table 3), individual benchmark scores are lower than when optimizing for each benchmark separately (Table 4), but the aggregate is higher. This isn't presented as a formal ablation but functions as one: it shows that DaMo's predictions are sensitive to the specific optimization objective, and the choice of objective (single-task vs. multi-task) has large practical consequences. The paper does not provide Pareto frontier analysis showing the trade-off between different tasks, which would be valuable for practitioners deciding how to weight competing objectives.

Direct transfer vs. linear calibration across models (Figure 5, Table 5): The comparison of direct DaMo transfer against linear-mapped DaMo serves as an ablation on the importance of model-specific calibration. Direct transfer achieves Pearson correlations of 0.75–0.95; calibration lifts these to 0.90+. The practical impact is quantified in Table 5 where linear-mapped DaMo achieves higher scores than direct DaMo across target models. The paper's finding that only 20 calibration samples are needed for meaningful improvement (vs. 250 for full DaMo training) is an implicit ablation on calibration sample efficiency, though the number of calibration samples is not systematically varied—we don't know whether 10 or 50 samples would yield different results.

Fixed mixture vs. dynamic mixture (Appendix D): The paper acknowledges in Appendix D that "preliminary attempts to relax these assumptions—specifically through dynamic data mixture adjustments—remain exploratory" and that they "have yet to establish a systematic methodology for extrapolating optimal dynamic mixtures or quantify the computational costs and performance gains relative to fixed data mixture." This is a notable negative result in that it confirms the difficulty of the dynamic mixing problem, but it is not a controlled ablation—no dynamic mixing baseline is implemented or compared against fixed DaMo, so we cannot quantify what performance is being left on the table.

ReST $^{EM}$ revision model experiment: The paper does not report any ablation varying model architecture for DaMo itself (beyond the transfer experiments), nor does it ablate MLP hyperparameters (depth, width, activation function, learning rate). The sensitivity of DaMo's performance to these choices is therefore unknown—would a deeper or wider MLP achieve R² > 0.81 with the same 1000 samples, or would it overfit? Would a different activation function better capture the non-monotonic patterns in Figure 3? These are left unexplored.

Critical Assessment

Claim 1: "DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods"

What was tested: DaMo's top-1 predicted mixture was trained and evaluated on PhoneAgentBench, compared against Uniform Mixture, Natural Mixture, and DML (Table 3). The overall average score improvement over DML is 3.02 percentage points (68.18% vs. 65.16%), and over Uniform Mixture is 5.24 percentage points. The abstract's 3.38% figure appears to reference a per-task average improvement rather than the overall average, but the specific calculation is not shown.

What was NOT tested: The comparison is against a single prior automated method (DML) and two heuristic baselines. There is no comparison against more sophisticated data selection methods (e.g., DoReMi-style group DRO adapted for SFT, or online bandit approaches like ODM). The claim of "compared to alternative methods" would be stronger with a broader baseline set. Additionally, the comparison is on a single base model (InternVL2.5-4B)—we don't know whether DaMo's advantage over DML holds for other base models without retraining, though the scalability experiments (Table 5) suggest DaMo transfers better than the baselines.

Conditions: The improvement is measured when DaMo's MLP is trained on 250 mixtures (1000 checkpoints) from exactly the same model and datasets used for final training. The paper does not report how DaMo performs if the training datasets change (e.g., adding a 13th dataset)—would the MLP need retraining, or could it generalize? The improvement also relies on the fixed-mix assumption; if dynamic mixing proves substantially better (which remains unexplored), DaMo's fixed-mixture optimum might underperform dynamic strategies.

Verdict: The claim that DaMo outperforms the tested baselines on PhoneAgentBench is supported by Table 3. However, the magnitude of improvement (3.02–5.24 percentage points over DML) should be interpreted cautiously: without confidence intervals or multiple training runs, we cannot distinguish whether this reflects genuine optimization signal or training noise. A single training run can vary by 1–2 percentage points due to random seed effects alone; the paper would benefit from reporting variance across multiple runs of the optimal mixture.

Claim 2: "DaMo outperforms other approaches by 2.57% in terms of average score" on general benchmarks

What was tested: The mixture optimized for PhoneAgentBench + open-source benchmarks was evaluated on BFCL-v3, MME-perception, MME-reasoning, and OCRBench (Table 3). DaMo's average across these four benchmarks exceeds DML's average by a reported 2.57%.

What was NOT tested: Critically, this evaluation uses the mixture optimized for the joint objective (PhoneAgentBench + open-source benchmarks). This is a different mixture than what would be optimal for the open-source benchmarks alone. A fairer comparison would be: DaMo optimized for open-source benchmarks only vs. DML optimized for open-source benchmarks only. Table 4 partially addresses this by showing task-specific DaMo results, but DML's task-specific performance on these benchmarks is not reported—so we cannot determine whether DaMo's advantage is in the mixture optimization or in the joint-vs-separate objective formulation.

Conditions: The 2.57% advantage is contingent on DaMo's MLP being trained on InternVL2.5-4B and the evaluation being on the same model family. The transfer experiments (Table 5) show DaMo maintains advantages over baselines on other models, but the specific 2.57% margin likely varies by model and evaluation suite.

Verdict: The claim is supported for the specific comparison presented (DaMo joint-optimized vs. baselines on general benchmarks), but the comparison design conflates two variables: the optimization method and the optimization objective. A cleaner ablation would report DaMo vs. DML when both optimize for exactly the same objective.

Claim 3: "When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods"

What was tested: DaMo optimized for BFCL-v3 alone achieves 47.43% vs. Uniform Mixture's 29.32% (Table 4). The 12.47% figure is the relative improvement over DML's BFCL-v3 performance.

What was NOT tested: The DML baseline for BFCL-v3-only optimization is not explicitly reported in the paper—Table 4 shows DaMo's BFCL-v3 score and the baseline scores, but the "12.47% than other methods" calculation's denominator is unclear from the main text alone. Without seeing DML's task-specific BFCL-v3 score, we cannot verify this relative improvement figure.

Special condition: The paper notes that this improvement occurs "even in the absence of any task-curated training data"—none of the 12 training datasets was designed for BFCL-v3. This is genuinely impressive and speaks to DaMo's ability to discover transfer relationships between training data and evaluation tasks. However, it also means the result may not generalize: on a benchmark where the base model has near-zero performance and no training dataset provides relevant signal, DaMo would have nothing to optimize.

Verdict: The large absolute gain (29.32% → 47.43%) is credible and important. The 12.47% relative improvement over "other methods" cannot be fully verified from the information provided in the main text without DML's task-specific score.

Claim 4: "DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures"

What was tested: DaMo trained on InternVL2.5-4B is transferred to Qwen2.5VL-3B, Qwen2.5VL-7B, and InternVL3-14B. Pearson correlations of 0.75–0.95 are reported for direct transfer, improving to 0.90+ with 20-sample linear calibration (Figure 5). Table 5 reports that DaMo outperforms baselines on all target models.

What was NOT tested: The transfer experiments test whether DaMo's rank-ordering of mixtures transfers, but the paper selects the optimal mixture for each target model from DaMo's predictions (after calibration). What is not reported is: would a DaMo retrained from scratch on the target model (at full 250-mixture cost) discover a substantially different optimal mixture than the transferred DaMo? The 0.90+ correlation suggests the rank-ordering is similar, but small rank-order differences in the top percentiles could translate to meaningful performance differences. Without comparing transferred-DaMo against retrained-DaMo on target models, we cannot quantify the performance cost of transfer vs. retraining.

Additionally, the transfer is tested across models of similar scale (3B → 7B → 14B) within the broader "small-to-medium" regime. The paper does not test whether DaMo transfers to substantially larger models (e.g., 72B+), where capacity differences might qualitatively change which mixtures are optimal—a 72B model might benefit from different data compositions than a 4B model because it can learn more complex patterns from the same data.

Verdict: The transfer results are strong for the tested models and the finding that only 20 calibration samples suffice for alignment is practically valuable. The claim of "robust scalability" is supported within the tested range (3B–14B, similar architecture families). Extrapolation beyond this range is not validated.

Missing Experiments That Would Strengthen the Paper

Multiple training runs with variance estimates. All reported scores are single-run point estimates. For the optimal mixture, training 3–5 times with different random seeds and reporting mean ± standard deviation would allow significance testing of the differences between DaMo and baselines. The 3.02 percentage point gap over DML on PhoneAgentBench would be more convincing with a standard error below ~0.5 percentage points.

Ablation on MLP architecture. The paper uses a two-layer MLP with 100 neurons per layer and ReLU activations. Varying depth (1 vs. 2 vs. 3 layers), width (50 vs. 100 vs. 200 neurons), and activation function (ReLU vs. GELU vs. tanh) would reveal whether the specific architecture matters or any reasonably-sized neural network works. This is important because the paper's entire contribution rests on the claim that MLPs outperform parametric functions for this problem—understanding what MLP properties drive this advantage would strengthen the argument.

Comparison against random search at equal cost. DaMo requires 4225 H20-hours for the fitting phase (250 mixtures). A natural baseline is: what if you spent those same 4225 H20-hours training and evaluating random mixtures at full 1440-step length (rather than DaMo's 4-checkpoint shortened runs)? How many full-training mixtures could you evaluate for the same cost, and what is the best score you'd find? If random search at equal cost approaches DaMo's performance, the advantage of learned prediction diminishes.

Dynamic mixing baseline. Appendix D acknowledges that dynamic mixing is unexplored. Implementing even a simple curriculum baseline (e.g., start with general data, transition to specialized data) and comparing against fixed DaMo would quantify what is being left on the table. If dynamic mixing provides only marginal gains (as the paper seems to hypothesize), that strengthens the case for fixed mixing; if it provides substantial gains, it changes the research direction.

Pareto frontier across tasks. When optimizing for multiple objectives, there is typically a trade-off surface. Visualizing the trade-off between, say, PhoneAgentBench aggregate and BFCL-v3 would help practitioners choose where on the frontier to operate. The paper currently reports point results (joint optimization vs. single-task optimization) but not the continuous trade-off between any two tasks.

Sensitivity to training dataset composition. All experiments use the same 12 datasets. Adding or removing datasets would test whether DaMo's predictions are robust to changes in the available data pool. If removing one dataset dramatically shifts the predicted optimal mixture for unrelated tasks, that would indicate DaMo captures specific interactions rather than general principles.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Completely Unaccounted For

The foundational prerequisite for DaMo is the collection of 250 mixture × 4 checkpoint = 1000 (mixture, step, score) training samples for the MLP, requiring 4225 H20-hours of GPU compute (Table 2). This cost is substantial — equivalent to fully training the MLLM roughly 250 times at 1440 steps each — yet the paper never amortizes it into any of the headline efficiency or performance numbers. When the paper reports that DaMo achieves a 3.38% improvement over DML on PhoneAgentBench (Table 3), this comparison is between the final trained models — it does not account for how many total GPU-hours each method consumed to arrive at that model. DML requires its own fitting procedure (whose cost the paper does not report, preventing a direct comparison), and the two heuristic baselines (Uniform, Natural) require zero optimization cost. A practitioner comparing methods needs to know: does DaMo's final-model performance advantage survive after subtracting the 4225 H20-hours of pilot experiments from the total budget?

The consequence is that DaMo's practical advantage may be overstated for one-off deployments. If a team needs to fine-tune a single model on a specific set of 12 datasets and evaluate on PhoneAgentBench, spending 4225 H20-hours to train DaMo before even beginning the "real" training run may not be justified — they could instead spend that budget training and evaluating random mixtures directly. The paper's own Figure 4(a) shows that random mixtures occupy a broad distribution, and with 4225 H20-hours of budget, a team could train and fully evaluate roughly 250 random mixtures (at 1440 steps each, rather than DaMo's shortened 360-step checkpoints) and simply pick the best performer. The paper never runs this comparison, so we do not know whether DaMo's learned optimization outperforms equal-cost random search.

The paper partially acknowledges the cost in Table 2 by reporting H20-hours alongside R², and flags it implicitly when noting that the fitting cost "converged" at 250 mixtures. However, it never incorporates this cost into any downstream performance comparison and explicitly states that the experiments "do not account for this cost" — a significant gap. The scalability experiments (Section 5.4, Figure 5) partially address the practical concern by showing that DaMo can be transferred to new models with only 20 calibration samples (~338 H20-hours), meaning the 4225 H20-hour initial investment can be amortized across multiple model deployments. But for the first deployment — or for a team working with a single model — the full cost must be paid upfront, and the paper provides no guidance on whether DaMo is cost-effective in that regime versus simply training and evaluating random mixtures.

The paper suggests no direct mitigation for this one-off deployment scenario. The transfer experiments establish that DaMo's cost is amortizable, but amortization requires multiple downstream deployments — a condition that does not hold for all use cases.

6.2 All Results Are on a Single Model Family and a Single Base Model Scale

Every experiment in the paper — the DaMo fitting phase, all baseline comparisons, and the PhoneAgentBench evaluations — uses InternVL2.5-4B as the base MLLM (Section 5.1, Appendix A.1). The scalability experiments in Section 5.4 extend DaMo to Qwen2.5VL-3B, Qwen2.5VL-7B, and InternVL3-14B, but critically, these extensions only test whether DaMo's predictions transfer — they do not test whether the optimal mixture changes, nor do they retrain DaMo from scratch on the target models. The paper never validates that an MLP trained on Qwen2.5VL-7B from scratch would discover a similar-or-different optimal mixture than the one transferred from InternVL2.5-4B. The 0.90+ Pearson correlations after linear calibration (Figure 5, bottom row) suggest the rank-ordering of mixtures is preserved, but rank-order correlation in the top percentiles — where DaMo selects its optimal mixture — could be lower than the overall correlation, and the paper does not report top-k precision or recall metrics that would quantify this.

The consequence is that we cannot distinguish between two possibilities: (1) the optimal mixture is genuinely model-agnostic (the paper's implicit claim), or (2) the optimal mixture is model-specific, but the transferred DaMo happens to find a mixture that is "good enough" on target models without being truly optimal for them. If possibility (2) holds, then a team deploying a new model architecture would face a difficult choice: accept the transferred mixture's performance (which Table 5 shows is competitive but not necessarily optimal) or invest the full 4225 H20-hours to retrain DaMo from scratch on their specific model. The paper provides no evidence to resolve this ambiguity.

Additionally, all tested models are in the 3B–14B parameter range — the "small-to-medium" regime. There is no evidence about whether DaMo's predictions would transfer to substantially larger models (e.g., 72B+), where capacity differences might qualitatively change which data mixtures are optimal. A 72B model might benefit from more complex task-planning data and less basic instruction-following data compared to a 4B model, because its greater capacity allows it to extract more value from challenging examples. The paper's finding that linear calibration works with only 20 samples may not extend to such capacity gaps, since linear corrections cannot capture qualitative shifts in which datasets are most valuable.

The paper does not acknowledge this as a limitation — it presents the scalability results as evidence that DaMo "maintains robust scalability" without discussing the untested regime of larger models or the difference between prediction transfer and optimal-mixture transfer. The calibration approach with 20 samples is the implicit mitigation, but its adequacy for larger capacity gaps is unverified.

6.3 PhoneAgentBench Is the Only Domain-Specific Evaluation, and Its Construction Is Not Independently Validated

DaMo's headline performance — the 3.38% improvement on PhoneAgentBench and the 23.35 percentage point gain over the no-SFT baseline — depends entirely on PhoneAgentBench being a valid and reliable measure of mobile phone agent capability. The benchmark comprises 1,235 QA pairs constructed by the authors through a combination of manual annotation and synthetic data generation (Section 3, Appendix A.2). The paper describes the data construction procedures in detail (e.g., the 6-phase MMIE generation pipeline, the manual screenshot capture for APP-Rec, the professional annotator filtering for MM-NER), but several aspects raise concerns about benchmark validity:

Cross-validation by internal annotators. The paper states that "three annotators were invited to conduct cross-validation" for MT-Plan data accuracy, and that annotators "manually constructed" questions for ACU, ITR, and other tasks. These annotators are presumably affiliated with OPPO AI Center (all authors share this affiliation). There is no external validation, no inter-annotator agreement metrics reported (e.g., Cohen's kappa, Krippendorff's alpha), and no third-party review of the benchmark's quality. Without independent validation, it is difficult to assess whether the benchmark measures genuine agent capability or reflects annotator-specific biases in question design.

Evaluation metrics vary widely across subtasks and some are coarse. MT-Plan uses a longest-common-subsequence-style plan similarity metric (from T-Eval); MM-NER uses entity F1-score; Mobile-FC uses strict exact-match (1 point for perfect function name + parameter match, 0 otherwise — a binary metric that may overpenalize near-correct answers); ACU uses BLEU score of de-anaphorized output; APP-Rec uses exact-match accuracy. These metrics are incommensurable (a 1-point improvement on BLEU-based ACU is not comparable to a 1-point improvement on exact-match Mobile-FC), yet the "overall average score" treats them as equally weighted and equally scaled. The paper does not analyze whether the overall average is dominated by high-variance or low-variance subtasks, nor does it report per-subtask statistical characteristics that would help interpret the aggregate.

The complexity metrics reported for MT-Plan (Table 7) — edge-to-node ratio for complexity, 1 minus average ROUGE-L for diversity — are compared against T-Eval planning. MT-Plan shows higher complexity and comparable diversity, which the paper presents as evidence of quality. However, these are structural metrics that measure graph topology and lexical variation, not task validity, difficulty calibration, or real-world representativeness. A benchmark can have complex DAGs and diverse queries while still being a poor measure of phone agent capability if the tasks are not representative of real user needs.

The consequence is that DaMo's reported performance gains may not generalize to other evaluations of mobile phone agent capability — particularly third-party benchmarks or real-world deployment metrics. Since PhoneAgentBench is introduced in this paper and has no independent adoption or validation history, the 3.38% improvement over DML and 23.35% improvement over the base model should be interpreted as performance on this specific benchmark, not as a validated measure of real-world phone agent quality.

The paper provides no mitigation for this limitation. PhoneAgentBench is presented as a contribution in its own right (listed as a core contribution in Section 1), and the paper does not discuss plans for external validation, community adoption, or correlation with real-world agent performance. The open-source release of the benchmark (linked in the abstract) would enable third-party validation, but this has not occurred at the time of publication.

6.4 The Fixed-Mixture Assumption May Leave Substantial Performance on the Table, and No Dynamic Baseline Is Tested

DaMo operates entirely within the fixed data mixing space (Section 4.1), where the mixture proportions p remain constant across all training steps. The paper explicitly acknowledges this as a simplification, stating in Appendix D:

"our preliminary attempts to relax these assumptions — specifically through dynamic data mixture adjustments — remain exploratory. We have yet to establish a systematic methodology for extrapolating optimal dynamic mixtures or quantify the computational costs and performance gains relative to fixed data mixture."

This is a candid admission, but the consequence is significant: we have no evidence about whether fixed mixing is close to optimal or whether dynamic/curriculum mixing would provide substantially larger gains. The paper's implicit claim is that getting the proportions right (first-order effect) dominates curriculum scheduling (second-order effect), but this claim is never tested. A simple curriculum baseline — e.g., starting with general instruction-following data (ShareGPT4, Infinity-MM) for the first 50% of training, then transitioning to specialized phone agent data (FC, TP, APP-Rec) for the remaining 50% — would provide a lower bound on what dynamic mixing can achieve. Without such a baseline, the paper cannot claim that DaMo's fixed mixtures are near-optimal; it can only claim they are better than other fixed mixtures and heuristic baselines.

The potential upside of dynamic mixing is suggested by the very patterns that motivate DaMo. Figure 3(a) shows that MMU training initially improves MT-Plan but then causes sharp decline — this is a textbook case where dynamic scheduling (reduce MMU proportion after the peak) should outperform fixed mixing (which must commit to a single MMU proportion for all steps, inevitably either undertraining before the peak or overtraining past it). Figure 3(b) shows the APP-Rec performance surface shifting with training steps, suggesting the optimal APP-Rec:MMU ratio changes over time. DaMo's fixed mixture optimizes for the average over the full training trajectory (since the MLP predicts scores at the final checkpoint), but a dynamic mixture could optimize for the instantaneous optimum at each training stage, potentially achieving higher peak performance.

The paper's proposal for future work — "integrating Monte Carlo Tree Search (MCTS) with reinforcement learning to iteratively determine stage-specific data mixtures" — acknowledges the problem but treats it as an entirely separate research program rather than a missing baseline in the current work. The practical implication for deployers is uncertainty: if a team adopts DaMo's fixed optimal mixture, they don't know whether they're leaving 2% or 15% of potential performance on the table compared to what a well-designed curriculum could achieve.

The mitigation is absent from the current paper. Appendix D describes this as future work with no timeline or preliminary results. A practitioner adopting DaMo today must accept the fixed-mixture assumption as an untested constraint on performance.

6.5 The Method Requires a Validated Evaluation Suite to Define the Optimization Target

DaMo optimizes data mixtures to maximize predicted performance on a specific set of downstream evaluation tasks (PhoneAgentBench subtasks + open-source benchmarks). This creates a dependency: the quality of DaMo's optimization is bounded by the quality of the evaluation suite it targets. If the evaluation suite has biases, blind spots, or metric flaws (as discussed in Limitation 6.3 for PhoneAgentBench), DaMo will optimize toward those biases — potentially producing a model that scores well on the benchmark but performs poorly in real deployment.

This is not merely a hypothetical concern. The paper demonstrates in Table 4 that optimizing DaMo for a single task (BFCL-v3) yields much higher performance on that task (47.43%) than optimizing for the multi-task aggregate. But the paper does not report what happens to other capabilities when optimizing for a single task — does the BFCL-v3-optimized model catastrophically forget phone agent capabilities? Does it maintain reasonable MME or OCRBench scores? Without this information, a practitioner cannot assess whether single-task optimization is a viable strategy or a dangerous form of overfitting to the evaluation metric.

More broadly, DaMo inherits all the well-known problems of benchmark-driven optimization: Goodhart's law ("when a measure becomes a target, it ceases to be a good measure"), benchmark contamination (training data inadvertently containing evaluation-like examples), and metric gaming (models learning to produce outputs that score well under the specific evaluation protocol without possessing the underlying capability). The paper does not discuss these risks, nor does it validate that models optimized via DaMo generalize to held-out tasks beyond the optimization target set. The open-source benchmarks (BFCL-v3, MME, OCRBench) provide some external validation since they were not used to construct the training datasets, but they are still fixed benchmarks with known evaluation protocols that the optimization process could indirectly exploit.

The consequence for deployment is that a model fine-tuned with DaMo's optimal mixture may show inflated benchmark scores relative to its real-world capability. The 23.35 percentage point gain on PhoneAgentBench (44.83% → 68.18%) is measured on exactly the benchmark DaMo was optimizing toward — there is no held-out phone agent evaluation to confirm that this gain reflects genuine capability improvement rather than benchmark-specific optimization.

The paper provides no mitigation for this — no held-out task evaluation, no real-world deployment validation, and no discussion of benchmark overfitting risks. The open-source benchmark results (Table 3) provide some signal that DaMo-optimized models maintain general capabilities, but these benchmarks are part of the optimization target set and thus do not constitute held-out validation. A practitioner deploying DaMo-optimized models should supplement the reported benchmarks with their own independent evaluation before trusting the performance claims.

6.6 No Statistical Significance or Variance Estimates Are Reported for Any Downstream Result

Every performance number in Tables 3–5 is a single-run point estimate. The paper trains one model per mixture configuration, evaluates it once, and reports that number as the definitive score. There are no confidence intervals, no standard deviations from multiple training runs, and no significance tests comparing DaMo against baselines. The closest the paper comes to statistical rigor is the 10-fold cross-validation R² for the MLP fitting (Table 2), but this evaluates the predictor's generalization, not the final model's performance stability.

The consequence is that we cannot distinguish genuine optimization signal from training noise for the headline comparisons. Neural network training is inherently stochastic — different random seeds for data shuffling, dropout, and parameter initialization can produce models whose benchmark scores differ by 1–2 percentage points (or more, depending on benchmark variance and model size). The reported gap between DaMo and DML on PhoneAgentBench is 3.02 percentage points (68.18% vs. 65.16%, from Table 3). If the standard deviation of training runs is ~1.5 percentage points, this gap is approximately 2 standard deviations — suggestive but not conclusive without formal testing. If the standard deviation is ~1.0 percentage point or less, the gap is more robust; if it's ~2.0 percentage points, the gap could be consistent with noise. The paper provides no information to make this assessment.

The problem is amplified for smaller-magnitude comparisons. The 2.57% average improvement over DML on open-source benchmarks (abstract, Table 3) could easily fall within noise if individual benchmark scores have standard deviations of 1–2 percentage points and the average is taken over only 4 benchmarks. Without variance estimates, a practitioner cannot determine whether these improvements are reliable enough to justify adopting DaMo over simpler baselines.

The paper does not acknowledge this limitation and provides no mitigation. Running 3–5 training repetitions with different random seeds for the optimal mixture (and for each baseline) would add roughly 3–5× the final-training cost (not the full 4225 H20-hours of the fitting phase) and would substantially strengthen the empirical claims. The omission is notable given that the paper's core contribution is an optimization method whose value proposition rests on reliably outperforming baselines by modest margins (2–5 percentage points).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a fundamental shift in how data mixture optimization is framed for supervised fine-tuning: it replaces proxy optimization (minimizing validation loss, as in pretraining scaling laws and DML) with direct performance modeling (predicting downstream benchmark scores from mixture proportions). This is not an incremental refinement of existing loss-prediction methods—it is a rejection of their core assumption that loss and downstream performance are monotonically related. The paper's diagnostic analysis in Figure 3 is the decisive evidence: the same training run produces simultaneous improvement, degradation, neutrality, and overfitting across different evaluation tasks, patterns that no monotonic loss function can capture. By building a model that predicts the quantity practitioners actually care about—task scores—rather than an intermediate signal they hope correlates with it, DaMo changes what it means to "optimize" data mixtures.

The magnitude of this shift is best understood as a reframing rather than a paradigm revolution. The paper does not introduce a new learning algorithm for MLLMs, nor does it propose a new class of optimization methods. Instead, it changes the optimization objective and demonstrates that neural networks—with their capacity to model non-monotonic, interactive, multi-modal surfaces—are the appropriate tool for this reframed problem. This is methodologically significant because it challenges the field's reliance on parametric scaling laws imported from pretraining, where data mixing and loss prediction are well-aligned. In SFT, that alignment breaks, and the paper provides both the diagnostic tools (Figure 3) and the solution architecture (MLP surrogate) to address the breakage.

The work resolves a previously unarticulated tension in the data mixing literature. Prior work split into two camps that rarely spoke to each other: pretraining methods (DoReMi, BiMix, ODM) that optimized loss using parametric functions, and fine-tuning practitioners (LLaMA3, Tulu3) who relied on costly manual iteration because automated methods didn't work well. The paper's diagnosis explains why: automated methods were optimizing the wrong thing. DaMo bridges this gap by automating mixture optimization while targeting the right objective—downstream performance—making automated mixture tuning practical for SFT for the first time. The fact that DaMo achieves a 3.02 percentage point improvement over DML (Table 3)—the most directly comparable prior automated method—and a 5.24 percentage point improvement over the Uniform baseline validates that this reframing translates to genuine gains, not just conceptual clarity.

The paper also redirects research attention in several important ways. More attractive directions now include: (a) understanding the shape of mixture-performance landscapes—what features of datasets and tasks determine whether mixtures are cooperative, competitive, or neutral—since DaMo demonstrates these landscapes have learnable structure; (b) developing better neural surrogate architectures for performance prediction, since the MLP is clearly a first-generation solution; (c) amortizing mixture optimization across model deployments, given the transfer results showing that mixture landscapes are largely model-agnostic (Figure 5). Less attractive directions include: (a) further refinement of loss-based parametric scaling laws for SFT, since the paper shows they are structurally incapable of capturing the relevant dynamics; (b) manual heuristic tuning, since Figure 4(a) demonstrates that Uniform and Natural mixtures perform no better than random chance.

However, the paper does not resolve the question of whether fixed mixing (DaMo's operating regime) or dynamic/curriculum mixing is ultimately preferable. Appendix D acknowledges this explicitly, and the absence of a dynamic baseline means the field still does not know whether optimizing proportions captures 90% of achievable gain or only 50%. This remains the central open question that DaMo's framework enables but does not answer.

Follow-Up Research This Work Enables

Quantifying the fixed-mixture performance ceiling: dynamic curriculum baselines against DaMo's optimum. The paper demonstrates that fixed optimal mixtures substantially outperform heuristic and DML baselines, but never tests whether a well-designed curriculum—changing mixture proportions during training—would outperform DaMo's best fixed mixture. A direct follow-up would implement a simple staged curriculum: Phase 1 (general capabilities) using high proportions of ShareGPT4, Infinity-MM, and MMIE for the first 480 steps; Phase 2 (phone-specific capabilities) transitioning to FC, TP, APP-Rec, and MMU for the remaining 960 steps. Train this against DaMo's optimal fixed mixture on InternVL2.5-4B and evaluate on PhoneAgentBench + open-source benchmarks. If the curriculum underperforms DaMo, it strengthens the paper's implicit claim that proportion optimization dominates scheduling effects. If it outperforms, it establishes a new ceiling and motivates the MCTS-based dynamic mixing approach proposed in Appendix D. The cost is modest—two additional training runs on top of the existing DaMo pipeline.

Learning difficulty-aware dynamic schedules using DaMo's performance predictions as a reward signal. The paper's MLP predicts performance at any training step for any mixture, making it a natural building block for reinforcement learning over mixture schedules. A follow-up would frame the dynamic mixing problem as a Markov decision process where the state is the current training step and recent validation scores, the action is choosing the next batch's dataset proportions, and the reward is the improvement in DaMo-predicted downstream performance. Train a lightweight policy network (e.g., a small LSTM) using DaMo's predictions as a learned dynamics model, then validate the discovered schedule against fixed DaMo on PhoneAgentBench. This would directly test whether the performance ceiling identified in the paper can be raised through dynamic allocation, using the paper's own tool (the MLP predictor) as the optimization substrate. A strong result would show gains of 5+ percentage points over fixed DaMo; a null result would confirm that the fixed-mix assumption is not practically limiting.

DaMo on text-only LLMs: does the loss-performance mismatch generalize beyond multimodal settings? The paper's entire analysis is on multimodal LLMs with multimodal tasks. The conflict, overfitting, and neutrality patterns in Figure 3 might be specific to multimodal fine-tuning, where vision-language interactions create complex transfer dynamics. A systematic replication on a text-only LLM—e.g., fine-tuning Llama-3-8B on a mix of MATH, Code-Alpaca, FLAN, and OpenAssistant data, evaluating on GSM8K, HumanEval, MMLU, and AlpacaEval—would test whether the DaMo paradigm generalizes. Train DaMo's MLP on 250 text-only mixtures, measure R², and compare the discovered optimal mixture against Uniform and DML baselines. If the patterns replicate (R² ~0.8, meaningful gains over baselines), it establishes DaMo as a general SFT optimization framework. If they don't, it identifies multimodal interaction as a necessary condition for the loss-performance mismatch, narrowing the scope of DaMo's applicability.

Stress-testing transfer across scale: does DaMo from a 4B model predict optimal mixtures for a 70B+ model? The paper validates transfer across 3B–14B models with 0.90+ Pearson correlations after linear calibration (Figure 5), but capacity differences in this range are modest (roughly 2–5× parameter count). A 70B model represents a ~17× scale increase over the 4B source, which could qualitatively change which mixtures are optimal—a larger model might extract substantially more value from complex task-planning data and less from basic instruction-following data. A stress test would train DaMo on InternVL2.5-4B, then evaluate its predictions on InternVL2.5-78B (or the largest available variant) using 50 calibration samples, computing both overall Pearson correlation and top-k precision (what fraction of DaMo's top-10 predicted mixtures are genuinely in the top-10 for the 78B model?). If top-k precision remains high (80%+), it validates DaMo's practical scalability to production-scale models. If it degrades substantially (below 50%), it establishes a boundary condition and motivates architecture-aware mixture optimization.

Adversarial evaluation: does DaMo over-optimize to benchmark idiosyncrasies rather than genuine capability? The paper optimizes DaMo to maximize scores on PhoneAgentBench and open-source benchmarks—the same benchmarks used for evaluation. This creates a risk of benchmark overfitting: the optimal mixture might improve scores on these specific test sets without improving real-world agent performance. A follow-up would construct a held-out phone agent evaluation suite—e.g., 200 new task-planning queries, 200 new function-calling scenarios, and 200 new screen-understanding tasks, constructed by different annotators using different templates—and evaluate DaMo-optimized models against Uniform-optimized models on this independent set. If the 23.35 percentage point gain on PhoneAgentBench (44.83% → 68.18%, Table 3) shrinks to, say, a 10-point gain on held-out data, it would indicate substantial benchmark overfitting. If the gain remains proportionally similar, it validates that DaMo discovers genuinely better mixtures rather than benchmark-specific hacks. This is the single most important validity check for practical adoption.

Feature attribution on the trained MLP to extract interpretable data mixing rules. The paper treats DaMo's MLP as a black-box predictor—it maps mixtures to scores without explaining why certain mixtures work. A follow-up would apply SHAP or integrated gradients to the trained MLP, quantifying which input features (dataset proportions) most influence predictions for each downstream task. For instance: "Increasing Task-Planning data from 5% to 15% has +3.2% predicted effect on MT-Plan score; increasing MMU data from 5% to 15% has −1.8% predicted effect on APP-Rec score." This would extract interpretable rules that practitioners can use even without running DaMo—e.g., "for phone agent tasks, prioritize at least 10% Task-Planning data and keep MMU below 20%." The analysis could also reveal whether the MLP has learned sensible transfer relationships (e.g., OCR training data should predict OCRBench performance) or surprising ones that merit deeper investigation. This bridges the gap between DaMo's predictive power and the scientific understanding of why certain mixtures work.

Practical Applications and Downstream Use Cases

Cost-efficient multitask fine-tuning for on-device assistant models. Mobile phone manufacturers (like OPPO, the authors' affiliation) deploying on-device MLLMs face a concrete problem: they have a fixed collection of training datasets covering capabilities like function calling, screen understanding, and task planning, and they need to produce a single model that performs well across all of them within a limited training budget. The paper provides a direct recipe: (1) sample 150–200 random mixtures from the fixed-mix space; (2) train the on-device model (e.g., a 3B-parameter variant) on each mixture with checkpointed evaluations; (3) fit DaMo's MLP on the resulting (mixture, step, score) tuples (4225 H20-hours for 250 mixtures, or proportionally less based on the convergence pattern in Table 2); (4) extrapolate the optimal mixture and train the production model. The 23.35 percentage point gain over the no-SFT baseline on PhoneAgentBench (Table 3) provides an order-of-magnitude estimate of the value: without DaMo, a team might spend weeks on manual mixture tuning to achieve gains that DaMo finds automatically. For a manufacturer shipping quarterly model updates, the 4225 H20-hour upfront cost amortizes over multiple release cycles, and the scalability results (Figure 5) mean the same DaMo instance can serve future model generations with only 20-sample recalibration.

Data generation prioritization for self-improving agent systems. When building mobile agents that improve through self-play or environment interaction, the training data mix changes dynamically as the agent encounters new scenarios. DaMo provides a mechanism for prioritizing which types of newly collected data to include in the next fine-tuning round. If a deployed agent collects 10,000 new task-planning examples, 5,000 new function-calling examples, and 50,000 new screen-understanding examples each week, the team needs to decide what proportion of each to include in the next training cycle. Retraining DaMo's MLP with the new dataset sizes and compositions would predict the performance impact of different inclusion ratios, enabling data-aware scheduling. The key efficiency comes from DaMo's ability to predict performance without training: rather than running 50 experimental training runs with different inclusion ratios to find the best one, the team runs DaMo inference (microseconds) and trains once on the predicted optimal mixture. The BFCL-v3 result (Table 4)—where DaMo discovered a mixture achieving 47.43% without any BFCL-curated training data—demonstrates that the method can identify valuable data-task relationships that aren't obvious from dataset labels alone, which is exactly the capability needed when new data streams arrive without clear task annotations.

Cross-model data mixture transfer for model family development. Organizations developing multiple model sizes simultaneously (e.g., a 3B on-device model, a 7B mid-range model, and a 14B cloud model) currently face the prospect of repeating mixture optimization for each model size—or worse, applying the same heuristic mixture to all sizes and accepting suboptimal performance. The paper's scalability results (Figure 5, bottom row) show that DaMo trained on a 4B model, with 20-sample linear calibration per target model, achieves 0.90+ Pearson correlations and competitive downstream performance (Table 5) on 3B, 7B, and 14B models. The practical workflow is: invest 4225 H20-hours to train DaMo on the smallest model in the family; spend ~338 H20-hours per additional model (20 calibration samples); apply the calibrated DaMo to select optimal mixtures for each. Compared to training independent DaMo instances for each model size (4225 H20-hours each), this represents a ~12.5× cost reduction per additional model. For a family of 4 model sizes, the total cost is 4225 + 3 × 338 = 5239 H20-hours, versus 4 × 4225 = 16,900 H20-hours for independent optimization—a 3.2× total savings. The performance cost of transfer vs. independent optimization is not quantified in the paper (no retrained-from-scratch DaMo on target models), but Table 5 shows transferred DaMo outperforms DML and heuristic baselines on all target models, establishing that transfer is competitive even if not provably optimal.

When to Prefer This Method

The paper explicitly positions DaMo against three alternatives: heuristic mixture strategies (Uniform, Natural), which cost nothing to devise but perform at random-chance levels (Figure 4a); DML (Ye et al., 2024), which automates mixture optimization but uses loss prediction that cannot capture non-monotonic SFT dynamics; and manual iteration (as documented for LLaMA3/Tulu3), which is effective but expensive in human and compute time. The choice between these depends on the deployment context, specifically the number of models to be trained, the available compute budget for optimization vs. final training, and whether the training dataset composition is stable or changing.

Prefer DaMo when:

The training dataset composition is fixed and multiple models will be fine-tuned on it (e.g., a model family, or periodic retraining on updated data). The 4225 H20-hour upfront cost amortizes across deployments; the scalability experiments (Figure 5, Table 5) explicitly validate this amortization with only 20 calibration samples per new model.
Downstream tasks exhibit known or suspected task interference (e.g., training on dataset A degrades performance on task B). The diagnostic patterns in Figure 3—Conflict, Overfitting—are exactly the regimes where loss-based methods fail and DaMo's direct performance modeling provides the largest advantage.
A single task is the priority and you need to discover which existing training data transfers to it. The BFCL-v3 result (Table 4: 47.43% vs. 29.32% Uniform, without any BFCL-curated training data) demonstrates DaMo's ability to identify non-obvious transfer relationships, which is valuable when labeled data for the target task is scarce but diverse auxiliary data is available.
The evaluation suite is multi-dimensional and you need to balance competing objectives. DaMo's MLP predicts scores for all tasks simultaneously, enabling joint optimization with explicit or implicit weighting. The paper shows this joint optimization preserves general capabilities while improving domain-specific ones (Table 3: +13.73% on open-source benchmarks while optimizing for PhoneAgentBench).

Prefer heuristic mixtures (Uniform or Natural) when:

Only a single model will be trained, and the total compute budget (optimization + training) is tightly constrained. The 4225 H20-hour DaMo fitting cost exceeds the cost of training and evaluating roughly 250 random mixtures at full length (the paper never runs this equal-cost comparison). If 250 random evaluations are affordable and the team can simply pick the best performer, DaMo's predictive advantage may not justify its upfront cost. This is the paper's most significant deployment caveat: the amortization argument fails for one-off projects.
The training datasets are known to be mutually compatible (no evidence of task interference, no overfitting patterns). In this regime, the smooth loss curves that DML and even heuristic methods rely on may be adequate, and DaMo's additional modeling capacity provides marginal benefit. The paper provides no evidence about what fraction of real-world SFT scenarios exhibit interference vs. compatibility, so this is a judgment call based on domain expertise.

Prefer DML when:

The computational budget cannot support 250 pilot training runs, and the dataset-task relationships are expected to be relatively well-behaved (monotonic improvement with data volume, minimal antagonistic interactions). DML's parametric fitting requires fewer pilot experiments (the paper does not report DML's fitting cost for direct comparison, but parametric models typically need fewer samples than neural networks). However, the paper's results suggest DML underperforms DaMo by 2.57–3.02 percentage points on average (Tables 3–4), so this choice trades performance for reduced optimization cost—a tradeoff the paper does not quantify but is implicit in the method comparison.