Qwen2.5 Technical Report

🎯 Pitch

By tripling its pre-training data to 18 trillion tokens and carefully enriching that data for math and coding, Qwen2.5-72B matches or beats Llama-3-405B—a model five times its size—on key benchmarks like MATH (83.1 vs. 73.8). A smaller 7B variant nearly triples its predecessor’s math score (75.5 vs. 52.9), showing that data selection can substitute for parameter count when paired with multi-stage reinforcement learning.

1. Executive Summary

This report introduces Qwen2.5, a series of large language models spanning 0.5B to 72B parameters plus MoE variants, that scales pre-training data from 7 trillion to 18 trillion tokens and applies a two-stage post-training pipeline—supervised fine-tuning on over 1 million examples followed by multi-stage reinforcement learning (offline DPO and online GRPO)—to enhance reasoning, code, math, and instruction-following capabilities. The flagship open-weight Qwen2.5-72B-Instruct achieves competitive parity with Llama-3-405B-Instruct on multiple benchmarks (81.2 vs. 69.3 on Arena-Hard, 83.1 vs. 73.8 on MATH) despite being roughly 5× smaller, while the smaller Qwen2.5-7B-Instruct reaches 75.5 on MATH—a 22.6-point improvement over its Qwen2-7B predecessor. The gains are most pronounced on mathematics and coding tasks where the pre-training data mixture was deliberately enriched, establishing that data scaling with targeted domain emphasis can substitute for parameter count only when the base model's architecture and post-training pipeline are jointly optimized.

2. Context and Motivation

The Core Problem: Making Open-Weight Models Competitive Through Better Training Rather Than Just Bigger Models

The fundamental challenge this paper addresses is deceptively practical: can you build open-weight language models that rival the largest proprietary systems without simply scaling parameter count? In the two years leading up to Qwen2.5, the LLM field had settled into a pattern where each generation of models achieved performance improvements primarily by growing larger — Llama-2 to Llama-3 to Llama-3.1 increased from 70B to 405B parameters, GPT-3 to GPT-4 represented orders-of-magnitude scaling, and the implicit assumption was that parameter count determined capability ceiling. The Qwen team challenges this assumption by seeking performance gains through data quality and training methodology rather than raw scale.

This matters for several practical reasons the paper implies throughout its introduction:

Democratization of access: Open-weight models can be run locally, fine-tuned on private data, and deployed without API costs. But if open-weight models remain a generation behind proprietary ones, they lock users out of the best capabilities. Closing this gap through training improvements rather than waiting for the community to train ever-larger models (which requires resources few possess) accelerates democratization.
Deployment feasibility: A 405B parameter model requires roughly 810 GB of memory at FP16 precision — multiple high-end GPUs just to load. A 72B model at competitive quality cuts hardware requirements by roughly 5×, making on-premise deployment practical for many more organizations. This is explicitly quantified in the paper's flagship comparison: Qwen2.5-72B-Instruct matches or exceeds Llama-3-405B-Instruct on Arena-Hard (81.2 vs. 69.3), MATH (83.1 vs. 73.8), and MMLU-redux (86.8 vs. 86.2) while being much smaller.
Small model regimes: The paper extends down to 0.5B, 1.5B, and 3B parameters — sizes that can run on edge devices and mobile phones. Prior to Qwen2.5, there were relatively few competitive open-weight models in this range (the paper notes these sizes are "under-represented in the current field of open foundation models"). Strong small models enable entirely new deployment scenarios.

Where Prior Approaches Fall Short

The paper identifies specific limitations in the existing landscape that motivate their approach:

Data quality is often treated as an afterthought relative to data quantity. The paper's Figure 1 (the radar-style chart) tells a revealing story: Qwen1.5-72B was trained on 3T tokens, Qwen2-72B on 7T tokens, and Qwen2.5-72B on 18T tokens. The raw token count doubled at each step, but the paper emphasizes that token count alone doesn't capture the improvement — deliberate data mixture matters. Specifically, prior open-source pre-training datasets heavily overrepresent certain web domains (e-commerce, social media, entertainment) that contain "repetitive, template-based, or machine-generated content" (Section 3.1), while underrepresenting high-quality domains like technology, science, and academic research. The Qwen team's remedy — using Qwen2-Instruct models as data quality filters and strategically rebalancing the domain mixture — represents an explicit rejection of the "scrape more web data" approach.

Mathematics and code are treated as post-training specializations rather than pre-training foundations. Many prior models (including Qwen2) addressed math and code primarily during the post-training phase through targeted instruction tuning. The Qwen2.5 approach integrates these capabilities much earlier by incorporating the training data from Qwen2.5-Math and Qwen2.5-Coder directly into pre-training. This is a non-trivial design choice: if mathematical reasoning requires certain neural circuits to form during pre-training (rather than being overlaid during fine-tuning), then the timing of data exposure matters. The paper's results — Qwen2.5-72B achieving 62.1 on MATH (vs. Qwen2-72B's 50.9, a 22% relative improvement) at the base model level, before any instruction tuning — support this hypothesis.

Reinforcement learning for post-training is typically done in a single, undifferentiated stage. Prior work (including Qwen2) applied RLHF as a unified process. Qwen2.5 decomposes RL into two distinct stages (Section 4): offline RL (DPO on pre-validated training pairs for reasoning, factuality, and instruction-following) followed by online RL (GRPO with a reward model for truthfulness, helpfulness, conciseness, and other human preference dimensions). The motivation is explicit: some capabilities (reasoning, factuality) are difficult for reward models to evaluate reliably, so they benefit from verified offline signals, while others (output quality, style, safety) are better suited to reward-based online optimization. This decomposition is not obvious — prior approaches typically treated all RL as a single phase — and the paper argues it enables better optimization of both categories.

Synthetic data quality is variable and often unchecked. Many models use LLM-generated synthetic data, but the quality control mechanisms vary widely. Qwen2.5 employs a layered verification system: Qwen2-72B-Instruct and Qwen2-Math-72B-Instruct generate synthetic data (Section 3.1), which is then filtered through a proprietary general reward model and the specialized Qwen2-Math-RM-72B. This dual-filter approach acknowledges that general and specialized quality assessment capture different types of errors.

Long-context capabilities are often achieved through post-hoc extension techniques with limited training. The paper notes that Qwen2 had a generation length limited to 2K tokens (Section 1, "Better in Use"), which is a significant practical limitation for tasks requiring detailed responses. The fix — extending generation length to 8K tokens — required deliberate data construction during SFT, including back-translation techniques to create long-response training pairs (Section 4.1). Additionally, for Qwen2.5-Turbo's 1M-token context, the paper implements progressive context length expansion through four stages (32K → 65K → 131K → 262K tokens) with carefully curated data mixtures (40% max-length, 60% shorter) at each stage. This is more sophisticated than simply applying RoPE extension at inference time.

Structured data understanding and tool use are underdeveloped in prior open-weight models. The paper explicitly identifies "structured input and output (e.g., tables and JSON)" and "easier tool use" as limitations of Qwen2 that Qwen2.5 addresses (Section 1). The control token expansion from 3 to 22 tokens (Section 2) — including two specifically for tool functionality — signals that the model's vocabulary was architecturally limited for these use cases.

How This Paper Positions Itself

The paper positions Qwen2.5 as an iteration that achieves performance breakthroughs through coordinated improvements across the entire training stack rather than through any single innovation. This is a deliberate positioning strategy: rather than claiming a novel algorithm or architecture, the paper argues that the combination of scaled high-quality data, staged pre-training, decomposable RL, and targeted SFT data construction produces outsized gains.

The introduction frames this explicitly in two ways:

First, by situating Qwen2.5 within the broader Qwen lineage (Figure 1 and the narrative around data scaling), the paper establishes that the improvements are cumulative and build on lessons from Qwen1.5 and Qwen2. The 18T token pre-training corpus isn't just bigger — it's better filtered, better mixed, and supplemented with specialized math/code data that prior versions lacked. This is important because it reframes the contribution from "we trained on more data" to "we learned what data matters and how to curate it."

Second, by emphasizing the model size range (0.5B to 72B plus MoE), the paper positions Qwen2.5 as a platform rather than a point solution. The availability of 3B, 14B, and 32B models fills gaps in the open-weight ecosystem that the authors characterize as "more cost-effective for resource-limited scenarios" — a practical orientation rather than a purely research-driven one. The inclusion of quantized versions and over 100 model variants (Hugging Face Hub, ModelScope, Kaggle) reinforces this deployment-focused positioning.

The paper also explicitly connects to the trend of inference-time scaling (citing o1) in the introduction, placing Qwen2.5 as a foundation for such techniques rather than a competitor. The statement that "Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models" positions the release as infrastructure for the broader ecosystem rather than an endpoint.

The Specific Gap: Systematic Underinvestment in Training Data Quality and Mixture

Perhaps the most significant gap the paper addresses — and the one that motivates much of the experimental design — is the systematic underinvestment in pre-training data quality engineering relative to architectural innovation in the open-weight community. While papers frequently mention "high-quality data" as important, the Qwen2.5 team provides concrete methodology: using Qwen2-Instruct as a multidimensional data quality filter, applying specialized reward models for synthetic data filtering, implementing domain-based down-sampling and up-sampling, and staging the pre-training data mixture across training phases.

This matters because it challenges the prevailing narrative that model capability is primarily a function of parameter count and total training FLOPs. If data curation and mixture can yield the kind of gains Qwen2.5 demonstrates (e.g., Qwen2.5-7B base model scoring 49.8 on MATH vs. Llama3-8B's 20.5 — a 2.4× difference at comparable parameter counts), then the efficient allocation of data engineering effort may be as important as the efficient allocation of compute that scaling laws typically address.

The paper doesn't frame this as a rejection of scaling laws — indeed, they develop their own scaling laws for hyperparameter optimization in Section 3.2 — but rather as a complement. The implicit argument is that compute-optimal training (Hoffmann et al., 2022) must consider not just how many tokens but also which tokens, and that this dimension has been underexplored in the open-weight literature.

3. Technical Approach

3.1 Reader Orientation

Qwen2.5 is a series of Transformer-based language models whose capabilities are the product of a coordinated training pipeline spanning data curation, pre-training, and two-stage post-training—none of which is individually novel, but whose combination with specific quality-control mechanisms yields substantial performance improvements. The paper solves the problem of building open-weight models that rival much larger proprietary systems by redirecting investment from raw parameter scaling toward three leverage points: (1) aggressive data quality filtering and domain rebalancing during pre-training, (2) staged pre-training that integrates specialized math and code data early, and (3) decomposable post-training that separates capabilities better learned from verified offline signals (reasoning, factuality) from those better learned through online reward optimization (style, safety, conciseness).

3.2 Big-Picture Architecture (Diagram in Words)

The Qwen2.5 system has five major components arranged in a sequential pipeline:

Data Curation Engine — a multi-stage filtering and mixture system that transforms raw web-scale corpora into 18 trillion high-quality pre-training tokens. Qwen2-Instruct models serve as quality filters; Qwen2.5-Math and Qwen2.5-Coder data are integrated directly; synthetic data is generated by Qwen2-72B-Instruct and Qwen2-Math-72B-Instruct and filtered by reward models; domain proportions are rebalanced through down-sampling of overrepresented domains (e-commerce, social media) and up-sampling of underrepresented ones (technology, science).
Pre-Training — a Transformer decoder with GQA, SwiGLU activations, RoPE, QKV bias, and RMSNorm pre-normalization, trained in two phases: an initial phase at 4,096-token context length followed by a long-context extension phase to 32,768 tokens (or progressively to 262,144 tokens for Turbo). Scaling laws derived from small-model experiments determine batch size and learning rate for each model size.
Supervised Fine-Tuning (SFT) — over 1 million examples spanning long-sequence generation, mathematics with chain-of-thought, coding across ~40 languages, instruction-following with code-based validation, structured data understanding, logical reasoning (70,000 new queries), cross-lingual transfer, and robust system prompts. Responses are filtered through a critic model and multi-agent scoring system; only those deemed "flawless by all scoring systems" are retained.
Offline Reinforcement Learning (DPO) — approximately 150,000 training pairs constructed by resampling from the SFT model on objective-domain queries (math, coding, instruction-following, logic), with responses that pass quality checks serving as positive examples and failures as negative examples. Trained for one epoch with the Online Merging Optimizer at learning rate 7 × 10⁻⁷.
Online Reinforcement Learning (GRPO) — a reward model trained on preference pairs from both open-source and proprietary queries guides Group Relative Policy Optimization, where 8 responses are sampled per query, queries are prioritized by response score variance, and training proceeds with a 2,048 global batch size.

Information flows sequentially: raw data → curated pre-training corpus → base model (via two-phase pre-training) → SFT model (via supervised fine-tuning on 1M+ examples) → DPO model (via offline preference optimization) → final instruction-tuned model (via online GRPO). For MoE models (Turbo, Plus), the architecture replaces standard FFN layers with fine-grained expert segmentation and shared expert routing.

3.3 Roadmap for the Deep Dive

First, the architecture and tokenizer (Section 2 of paper), since all modeling decisions—GQA for KV cache efficiency, SwiGLU for activation, RoPE for position encoding, QKV bias, RMSNorm—establish the computational substrate everything else builds on. The tokenizer expansion from 3 to 22 control tokens is especially important because it enables the tool-use and structured output capabilities that distinguish Qwen2.5 from its predecessor.
Second, pre-training data curation (Section 3.1), because the paper's central claim is that data quality improvements, not architectural novelty, drive most of the performance gains. Understanding the five data improvement axes (filtering, math/code integration, synthetic data generation, mixture rebalancing, and scaling to 18T tokens) is prerequisite to interpreting any benchmark result.
Third, scaling laws and hyperparameter selection (Section 3.2), since the paper uses these not to determine model size (as in Hoffmann et al., 2022) but to transfer optimal hyperparameters across architectures—a subtle but important inversion of the standard scaling law application.
Fourth, long-context pre-training (Section 3.3), because the two-phase approach (4K → 32K tokens, with ABF for RoPE) and the progressive expansion strategy for Turbo (four stages to 262K tokens) represent concrete engineering choices with measurable impact on RULER, LV-Eval, and LongBench-Chat.
Fifth, the SFT data construction and training (Section 4.1), covering all nine data enhancement areas and the associated quality control mechanisms, since SFT is where domain-specific capabilities are operationalized.
Sixth, the two-stage RL design (Sections 4.2–4.3), covering offline DPO for reasoning/factuality and online GRPO for human preference dimensions, since this decomposition is the paper's key post-training innovation.
Seventh, long-context fine-tuning (Section 4.4), which uses a two-stage SFT approach (short instructions only, then mixed short/long) and an RL-on-short-only strategy to achieve 1M-token context for Turbo without sacrificing short-context performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an engineering systems paper whose core idea is that coordinated optimization across data curation, pre-training staging, and decomposable post-training can produce models that rival much larger alternatives. There is no single novel algorithm; the contribution is the demonstrated effectiveness of the combination of known techniques applied with unusual rigor to data quality and training methodology.

Architecture and Tokenizer

The dense Qwen2.5 models (0.5B through 72B parameters) use a standard Transformer decoder architecture with specific component choices carried forward from Qwen2. The architecture is not novel—every component has prior literature—but the specific combination and parameterization matter for both training stability and inference efficiency.

Grouped Query Attention (GQA): Rather than giving every attention head its own full set of key-value projections (as in multi-head attention), GQA shares K and V projections across groups of query heads. Table 1 shows the Q/KV head ratios: for the 72B model, there are 64 query heads but only 8 KV heads (an 8:1 sharing ratio); for the 7B model, 28 query heads and 4 KV heads (7:1). The paper's smaller models (0.5B, 1.5B, 3B) use 2 KV heads. The primary motivation is KV cache efficiency during inference: each token's key-value pairs must be stored for autoregressive generation, so reducing the number of distinct KV heads by 4–8× proportionally reduces memory consumption. This is not a performance innovation but an operational one—it makes deployment of larger models practical on memory-constrained hardware.

SwiGLU Activation: The feed-forward layers use the SwiGLU activation function, which combines a gating mechanism (Sigmoid-activated linear projection) with a value projection (SiLU/sigmoid linear unit). Compared to standard ReLU, SwiGLU provides a smoother, non-monotonic activation landscape that has been empirically shown to improve training dynamics and downstream performance in prior work (Dauphin et al., 2017). The paper does not ablate this choice—it is inherited from Qwen2 and treated as a standard architectural component.

Rotary Positional Embeddings (RoPE): Position information is encoded by applying a rotation matrix to query and key vectors based on their absolute position, such that the dot product between queries and keys depends only on relative position. The base frequency starts at 10,000 for standard pre-training and is increased to 1,000,000 during long-context extension using the ABF technique (Adjusted Base Frequency, Xiong et al., 2023). For Qwen2.5-Turbo, the base frequency reaches 10,000,000 at the longest stage. Increasing the RoPE base frequency stretches the wavelength of the position encoding, allowing the model to distinguish positions at much greater distances without retraining positional embeddings—a critical enabler for the 32K to 1M token context windows.

QKV Bias: The attention mechanism includes bias terms in the query, key, and value projections. This is referenced as "QKV bias (Su, 2023)" and is related to length extrapolation—the bias term interacts with RoPE to improve the model's ability to generalize to sequence lengths beyond those seen during pre-training. The paper does not elaborate on the mechanism, but the citation to Su (2023) connects it to improved attention score distributions at long range.

RMSNorm with Pre-Normalization: Layer normalization using root mean square statistics (rather than mean-and-variance statistics as in LayerNorm) is applied before each sub-layer (attention and FFN), not after. Pre-normalization has become standard since it stabilizes training by ensuring that residual streams are normalized before transformation rather than after.

MoE Extension for Turbo and Plus: Building on the dense architecture, the MoE models replace standard FFN layers with MoE layers containing multiple expert FFNs and a routing mechanism that dispatches each token to the top-K experts. Two architectural choices from Qwen1.5-MoE are carried forward: (1) fine-grained expert segmentation (Dai et al., 2024)—rather than having a small number of large experts, each MoE layer contains many smaller experts, giving the router more granular control over token assignment; and (2) shared expert routing (Rajbhandari et al., 2022; Dai et al., 2024)—some experts are always activated regardless of the routing decision, ensuring a baseline computation for all tokens while the routed experts provide specialization. The paper claims these "yielded substantial improvements in model performance across downstream tasks" but does not provide ablations—these are architectural choices validated in prior work.

Tokenizer: The tokenizer uses byte-level BPE (BBPE) with a vocabulary of 151,643 regular tokens and 22 control tokens. The control token expansion from 3 (in prior Qwen versions) to 22 is functionally significant: two new tokens are allocated for tool functionality (likely marking tool calls and tool outputs in conversation), and the remaining 17 are "allocated for other model capabilities." The paper states this "establishes a unified vocabulary across all Qwen2.5 models, enhancing consistency and reducing potential compatibility issues"—a practical engineering concern when a single model family spans 0.5B to 72B parameters and needs to support consistent conversational formatting.

Architectural Details by Model Size (Table 1):

Model	Layers	Q/KV Heads	Tie Embedding	Context/Gen Length
0.5B	24	14/2	Yes	32K/8K
1.5B	28	12/2	Yes	32K/8K
3B	36	16/2	Yes	32K/8K
7B	28	28/4	No	128K/8K
14B	48	40/8	No	128K/8K
32B	64	40/8	No	128K/8K
72B	80	64/8	No	128K/8K

The "tie embedding" column indicates whether input token embeddings and output projection weights are shared. Models ≤3B tie embeddings (reducing parameter count for a given vocabulary size, which matters at small scales), while models ≥7B do not (allowing independent learning of input representations and output distributions, which matters at larger scales). Context lengths are 32K for small models and 128K for larger ones, with generation length uniformly capped at 8K tokens (a deliberate increase from Qwen2's 2K limitation).

Pre-Training Data Curation

The pre-training data pipeline is the most substantively detailed portion of the paper and the component the authors attribute most performance gains to. The dataset expands from Qwen2's 7 trillion tokens to 18 trillion tokens, but the paper emphasizes that quality improvements are at least as important as quantity.

Data Quality Filtering via Qwen2-Instruct: The primary filtering mechanism uses Qwen2-Instruct models as "data quality filters that perform comprehensive, multi-dimensional analysis to evaluate and score training samples." This is a form of model-based data cleaning where a previously trained LLM acts as a classifier/judge over candidate pre-training documents. The paper claims this is a "significant advancement" over Qwen2's filtering approach because Qwen2-Instruct's "expanded pre-training on a larger multilingual corpus" provides "more nuanced quality assessment, resulting in both improved retention of high-quality training data and more effective filtering of low-quality samples across multiple languages."

The operational mechanism is not described in detail—we don't know the exact scoring dimensions, thresholds, or whether this is a binary accept/reject or a soft weighting scheme. But the key design choice is clear: use a model trained on an earlier generation of data to curate data for the next generation, creating a bootstrapping quality flywheel.

Math and Code Data Integration: Rather than treating math and code as post-training specializations, Qwen2.5 incorporates the training data from Qwen2.5-Math (Yang et al., 2024b) and Qwen2.5-Coder (Hui et al., 2024) directly into pre-training. The paper states this "proves highly effective, as these specialized datasets are instrumental in achieving state-of-the-art performance on mathematical and coding tasks." This is a non-trivial architectural decision with implications for how capabilities form: if mathematical reasoning circuits benefit from being established during the pre-training phase (when the model is learning general linguistic and world knowledge), then deferring math data to post-training may leave performance on the table. The base model MATH scores support this: Qwen2.5-72B base achieves 62.1 on MATH vs. Qwen2-72B base's 50.9—a 22% relative improvement before any instruction tuning.

Synthetic Data Generation and Filtering: For domains where high-quality natural data is scarce—specifically mathematics, code, and knowledge—the paper generates synthetic data using Qwen2-72B-Instruct and Qwen2-Math-72B-Instruct. The synthetic data is then filtered through a two-stage quality control pipeline: a "proprietary general reward model" and the "specialized Qwen2-Math-RM-72B." This dual-filter approach addresses a subtle problem: a general reward model might approve synthetic math content that is stylistically plausible but mathematically incorrect, while a math-specific reward model might reject synthetic content that is correct but stylistically atypical. Using both in sequence (the paper implies both filters must approve) reduces both false positives and false negatives.

The specific generation methodology for synthetic data is not described—we don't know the prompts, temperature settings, or rejection sampling thresholds. The paper treats this as an established technique rather than a novel contribution.

Data Mixture Rebalancing: The paper identifies a specific problem with web-scale data: "domains like e-commerce, social media, and entertainment are significantly overrepresented in web-scale data, often containing repetitive, template-based, or machine-generated content." Conversely, "domains such as technology, science, and academic research, while containing higher-quality information, are traditionally underrepresented." The solution is two-fold: (1) use Qwen2-Instruct models to classify documents into domains, and (2) strategically down-sample overrepresented domains while up-sampling high-value domains.

The granularity of domain classification and the exact sampling ratios are not specified, but the principle is clear: raw web frequency is a poor proxy for training value, and deliberate rebalancing can compensate. This is an explicit rejection of the "train on everything" approach that characterized earlier web-scale datasets like CommonCrawl.

Staged Pre-Training Data: The paper mentions that "the pre-training is staged to allow transitions among different mixtures" (Section 1), suggesting that the data mixture changes during the course of pre-training—possibly starting with more general web data and gradually increasing the proportion of specialized math/code/synthetic data, or varying the domain balance across training phases. The specific staging schedule is not detailed, but the concept is analogous to curriculum learning: the model sees easier or more general data early in training and more specialized or complex data later.

Scaling Laws for Hyperparameter Selection

The paper develops scaling laws following the tradition of Hoffmann et al. (2022) and Kaplan et al. (2020), but with an important inversion of purpose: rather than using scaling laws to determine the optimal model size for a given compute budget, Qwen2.5 uses them to determine optimal hyperparameters (batch size and learning rate) for models of already-chosen sizes.

Experimental Design: The scaling law experiments cover dense models from 44M to 14B parameters and MoE models with 44M to 1B activated parameters, trained on datasets ranging from 0.8B to 600B tokens. This is a substantial experimental campaign—training dozens of small models at varying scales to establish the relationship between architecture and optimal training configuration.

What the Scaling Laws Predict: The paper identifies two key relationships:

How the optimal learning rate $\mu_{\text{opt}}$ varies with model size $N$ and pre-training data size $D$
How the optimal batch size $B_{\text{opt}}$ varies with model size $N$ and pre-training data size $D$

The specific functional forms are not provided in the paper—we only have the qualitative statement that "we systematically study the relationship between model architecture and optimal training hyper-parameters" and that the laws "help determine key training parameters like batch size $B$ and learning rate $\mu$ for both dense models and MoE models of varying sizes."

MoE vs. Dense Comparison: The scaling laws are also used to "predict and compare the performance of MoE models with varying parameter counts against their dense counterparts." This guides how many activated and total parameters the MoE models need to achieve "performance parity with specific dense model variants (such as Qwen2.5-72B and Qwen2.5-14B)." In other words, the scaling laws tell the team what MoE configuration is needed to match a given dense model, enabling the cost-performance tradeoffs that make Turbo and Plus competitive.

Why This Design Choice: The standard application of scaling laws (as in Chinchilla) answers the question "given a FLOPs budget, what model size and data quantity should I use?" The Qwen2.5 team already knows their model sizes (they're building a family from 0.5B to 72B) and data quantity (18T tokens). Their question is different: "given these fixed model sizes and this fixed data budget, what training hyperparameters minimize final loss?" By training small models and extrapolating, they avoid expensive hyperparameter sweeps at the full 72B scale. This is a practical, engineering-driven use of scaling laws rather than a scientific investigation of scaling relationships.

Limitations: The paper does not report the scaling law functional forms, the quality of fit, or the extrapolation error when going from 44M–14B experiments to the full 72B model. These details matter for assessing how much to trust the hyperparameter recommendations.

Long-Context Pre-Training

The paper implements a two-phase pre-training approach for context length, with standard models and Turbo following different expansion trajectories.

Standard Models (0.5B through 72B):

Phase 1: Pre-training at 4,096-token context length. This is the computationally efficient phase where the model learns the bulk of its knowledge and capabilities. Training on shorter sequences is faster per token and avoids the quadratic attention cost of long sequences.
Phase 2: Context length extension from 4,096 to 32,768 tokens during the final pre-training stage. During this phase, the RoPE base frequency is increased from 10,000 to 1,000,000 using the ABF technique (Adjusted Base Frequency, Xiong et al., 2023).

What ABF does: RoPE encodes position through rotation angles determined by $\theta_i = \text{base}^{-2i/d}$ where $i$ indexes the head dimension and $d$ is the head dimension. Increasing the base frequency from 10,000 to 1,000,000 makes these rotation angles smaller for a given position, which means the dot product between far-apart positions decays more slowly—effectively extending the range over which the model can distinguish relative positions. ABF adjusts this base frequency to smoothly transition the model's positional understanding from the 4K regime to the 32K regime without requiring full re-training.

Qwen2.5-Turbo (Progressive Expansion): Turbo follows a more aggressive four-stage expansion:

4,096 → 32,768 tokens (first extension)
32,768 → 65,536 tokens
65,536 → 131,072 tokens
131,072 → 262,144 tokens (final training context)

The RoPE base frequency is set to 10,000,000 (10× higher than standard models) to support this extreme range. At each stage, the training data is curated to contain 40% sequences at the current maximum length and 60% shorter sequences. This 40/60 ratio is a specific engineering choice: 40% provides enough long-context exposure to learn the positional relationships, while 60% shorter sequences maintain the model's performance on short-context inputs (which would degrade if the model only saw long sequences). The paper claims this "enables smooth adaptation to increasing context lengths while maintaining the model's ability to effectively process and generalize across sequences of varying lengths."

Inference-Time Length Extrapolation: Beyond the training context length, the paper applies two techniques to further extend sequence capacity at inference:

YARN (Yet Another RoPE extensioN, Peng et al., 2023): A method for extending RoPE-based models to longer contexts by adjusting the rotation frequencies. YARN interpolates the position indices for dimensions corresponding to high frequencies (which encode local position information) while extrapolating dimensions corresponding to low frequencies (which encode global position). This mixed interpolation/extrapolation preserves local attention patterns while allowing the global position encoding to reach unseen distances.
Dual Chunk Attention (DCA, An et al., 2024): An attention mechanism that processes long sequences in chunks while maintaining cross-chunk information flow. DCA decomposes attention into intra-chunk attention (within each segment) and inter-chunk attention (between segments), reducing the quadratic complexity of full attention while preserving the model's ability to attend across chunk boundaries.

Together, YARN and DCA achieve a four-fold increase in sequence length capacity over the training context: Qwen2.5-Turbo can handle up to 1 million tokens (4× 262K), and other models can process up to 131,072 tokens (4× 32K). The paper verifies this with an ablation in Tables 16–17: without DCA+YARN, RULER scores at 128K context drop from the 80s to the 30s–60s depending on model size (e.g., Qwen2.5-72B-Instruct drops from 88.4 to 67.0, Qwen2.5-7B-Instruct drops from 55.1 to 31.4), demonstrating that the extrapolation techniques are essential, not optional, for long-context performance.

MInference for Long-Context Speed: For Qwen2.5-Turbo at 1M tokens, the paper implements a sparse attention mechanism based on MInference (Jiang et al., 2024b) that "reduces the computational load of the attention mechanism by 12.5 times." This translates to a 3.2–4.3× speedup in time-to-first-token across hardware configurations (Figure 3). The sparse attention likely identifies and skips attention computations between query-key pairs that contribute negligibly to the output, exploiting the observation that attention patterns in very long sequences tend to be highly sparse.

Supervised Fine-Tuning

The SFT phase is where general pre-trained capabilities are operationalized into specific behaviors. Qwen2.5 constructs over 1 million SFT examples spanning nine enhancement areas, each with distinct data construction methodologies and quality control mechanisms.

Training Configuration: The model is fine-tuned for two epochs with a sequence length of 32,768 tokens—this is notable because it means the SFT phase itself operates at long context, training the model to handle extended instructions and generate long responses in a single pass. The learning rate follows a cosine-like schedule decaying from 7 × 10⁻⁶ to 7 × 10⁻⁷. Weight decay of 0.1 is applied for regularization, and gradient norms are clipped at a maximum value of 1.0 to prevent training instability.

(1) Long-Sequence Generation: Qwen2's generation length was limited to approximately 2,000 tokens—a serious practical constraint for tasks requiring detailed explanations, long-form content, or multi-step reasoning chains that produce substantial intermediate work. To train Qwen2.5 to generate up to 8,192 output tokens, the team constructs long-response datasets using back-translation: they take long text from pre-training corpora and use a model to generate queries that those long texts would appropriately answer. Output length constraints are explicitly imposed during data construction, and Qwen2 is used to filter out low-quality pairs.

The back-translation approach solves a data scarcity problem: there are relatively few naturally occurring examples of very long, high-quality responses to specific queries in standard instruction datasets. By starting with the long text (which exists abundantly in pre-training data) and generating a query backward, they can create synthetic long-response training pairs without requiring human annotators to write both the query and the multi-thousand-word response.

(2) Mathematics: The SFT data incorporates chain-of-thought reasoning from Qwen2.5-Math (Yang et al., 2024b), spanning "public datasets, K-12 problem collections, and synthetic problems." The quality control mechanism is rejection sampling with reward modeling: the model generates multiple solution attempts per problem, a reward model scores each attempt, and only solutions that match the annotated ground-truth answer AND receive high reward model scores are retained. This dual criterion—correctness (matching the answer) AND quality (high reward model score)—ensures that the training data contains not just right answers but well-reasoned right answers with clear step-by-step exposition.

(3) Coding: The coding SFT data is drawn from Qwen2.5-Coder (Hui et al., 2024) and constructed through a multi-step pipeline:

Multiple language-specific agents collaborate to generate instruction pairs across nearly 40 programming languages—far more than the typical Python-focused coding datasets.
Instruction data is expanded by synthesizing examples from code-related Q&A websites and gathering algorithmic code snippets from GitHub.
A comprehensive multilingual sandbox performs static code checking and validates code snippets through automated unit testing. This is a critical quality guarantee: unlike natural language, code correctness can be mechanically verified, and the sandbox ensures that training examples contain executable, correct code (Dou et al., 2024).

(4) Instruction-Following: The paper implements a code-based validation framework for instruction-following data. LLMs generate both instructions AND corresponding verification code AND comprehensive unit tests. The generated code is executed, and only instruction-response pairs that pass the unit tests are retained—this is execution feedback-based rejection sampling applied to the meta-task of verifying that responses follow their instructions. The paper cites Dong et al. (2024) for this approach, and the key insight is that instruction-following fidelity can be operationalized as a verifiable property when the instruction constrains the response format or content in ways that unit tests can check.

(5) Structured Data Understanding: The SFT data includes "traditional tasks, such as tabular question-answering, fact verification, error correction, and structural understanding, as well as complex tasks involving structured and semi-structured data." A key enhancement is incorporating reasoning chains into responses—rather than just extracting facts from tables, the model learns to articulate the inferential steps connecting structured data to conclusions. This is important for Qwen2.5's claimed improvements in "structured input and output (e.g., tables and JSON)."

(6) Logical Reasoning: The paper introduces 70,000 new queries spanning multiple-choice, true/false, and open-ended formats across diverse reasoning types: deductive reasoning, inductive generalization, analogical reasoning, causal reasoning, and statistical reasoning. The quality control involves "iterative refinement" where "data containing incorrect answers or flawed reasoning processes" is systematically filtered out. This creates a curriculum of progressively cleaner reasoning examples.

(7) Cross-Lingual Transfer: To transfer capabilities from high-resource languages (English, Chinese) to low-resource languages, the paper uses a translation model to convert instructions into various languages and generate corresponding responses. Semantic alignment between each multilingual response and its original is evaluated to ensure "the logical structure and stylistic nuances of the original responses" are preserved. This is a practical approach to a difficult problem: collecting high-quality instruction data in dozens of low-resource languages is prohibitively expensive, but translation plus quality checking can approximate it.

(8) Robust System Instruction: The team constructs "hundreds of general system prompts" to improve the diversity of system prompts seen during post-training, ensuring the model doesn't overfit to a particular prompt style. Evaluations with different system prompts show that the model "maintains good performance and reduced variance, indicating improved robustness" (citing Lu et al., 2024b).

(9) Response Filtering: The final quality gate uses multiple automatic annotation methods, including "a dedicated critic model and a multi-agent collaborative scoring system." Responses undergo rigorous assessment, and only those deemed flawless by all scoring systems are retained. This is an intentionally conservative filtering strategy: false negatives (rejecting good responses) are preferred over false positives (including flawed responses), trading data quantity for data quality.

Offline Reinforcement Learning (DPO)

The offline RL stage addresses a specific limitation of online RL: reward models struggle to evaluate reasoning, factuality, and instruction-following in complex domains because these properties require verifying multi-step logical chains that reward models (themselves LLMs) may not reliably assess.

Data Construction: Using the SFT model as a generator, the team samples responses for a set of queries in "objective query domains such as mathematics, coding, instruction following, and logical reasoning" where correctness can be verified through execution feedback or answer matching. Responses that pass quality checks become positive examples (chosen); responses that fail become negative examples (rejected). This creates preference pairs where the "preference" is grounded in objective correctness rather than subjective human judgment.

The paper additionally applies "both human and automated review processes" (citing Cao et al., 2024) to "further enhance the reliability and accuracy of the training signals." The dataset comprises approximately 150,000 training pairs.

DPO Training: Direct Preference Optimization (Rafailov et al., 2023) trains the model to increase the likelihood of chosen responses relative to rejected ones without explicitly training a separate reward model. The DPO objective can be written in terms of the policy model $\pi_\theta$ and a reference model $\pi_{\text{ref}}$ (typically the SFT model) as:

$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

where $(x, y_w, y_l)$ is a preference triple (prompt, chosen/winning response, rejected/losing response), $\sigma$ is the logistic sigmoid function, $\beta$ is a temperature parameter controlling how far $\pi_\theta$ can deviate from $\pi_{\text{ref}}$ , and the expectation is over the training dataset $\mathcal{D}$ .

What it computes: For each preference pair, DPO computes the log-ratio of the policy model's probability for the chosen response versus the rejected response, relative to the reference model's log-ratios. If the policy assigns higher relative probability to the chosen response than the reference does, the term inside the sigmoid is positive and the loss is low. If the policy deviates too far from the reference in the wrong direction, the loss penalizes it. The $\beta$ parameter controls the strength of this regularization.

Why this form: DPO reparameterizes the RLHF objective (which normally requires explicitly training and sampling from a reward model) into a direct policy optimization objective. The key insight is that the optimal policy under a Bradley-Terry preference model can be expressed in terms of the policy and reference probabilities alone, eliminating the need for a separate reward model. For offline RL, this is advantageous because (a) it avoids the instability of training a separate reward model on data where reward signals are noisy, and (b) it directly optimizes the policy on the verified preference pairs without an intermediate reward approximation step.

Training Details: The model trains for one epoch using the Online Merging Optimizer (Lu et al., 2024a) with learning rate 7 × 10⁻⁷. Training for only one epoch on 150K pairs is conservative—the paper is avoiding over-optimization of the DPO signal, which can cause the model to collapse to degenerate outputs that exploit the preference data.

Online Reinforcement Learning (GRPO)

The online RL stage addresses capabilities that are well-suited to reward model evaluation: truthfulness, helpfulness, conciseness, relevance, harmlessness, and debiasing. Unlike the offline stage where correctness is objective, these qualities require nuanced judgment that a reward model can provide.

Reward Model Training Data: The reward model is trained on preference pairs drawn from two sources: publicly available open-source data and a "proprietary query set characterized by higher complexity." Responses are generated from Qwen model checkpoints at various training stages (SFT, DPO, and RL) at different temperature settings to introduce diversity. Preference labels are created through both human annotation and automated labeling, and the DPO training data is also integrated into this dataset.

The labeling criteria are explicitly enumerated as six dimensions: truthfulness (factual accuracy, fidelity to provided context), helpfulness (usefulness, engagement, educational value, instruction-following precision), conciseness (brevity without sacrificing clarity), relevance (alignment with query, history, and context), harmlessness (avoiding content that could enable illegal, immoral, or harmful behavior), and debiasing (freedom from gender, race, nationality, and political bias, with fair treatment of all topics).

GRPO Training: Group Relative Policy Optimization (Shao et al., 2024) is an online RL algorithm that extends the standard PPO approach with grouped relative comparisons. For each query, the model samples 8 responses—a group—and the reward model scores all 8. The GRPO objective can be understood as optimizing the policy to increase the probability of responses that score above the group mean while decreasing the probability of below-mean responses.

The training configuration includes a global batch size of 2048 and 2048 samples per episode, with "a pair of queries and responses as a sample." Queries are not processed in random order: instead, they are prioritized by the variance of their response scores as evaluated by the reward model. Queries with higher score variance are trained on first, because high variance indicates that the model's current policy produces inconsistent-quality responses for that query, making it a more informative training example. Low-variance queries (where the model already consistently produces good or consistently produces bad responses) provide less useful gradient signal. This is a curriculum learning strategy based on the reward model's uncertainty.

Design Rationale for Two-Stage RL: The decomposition into offline DPO and online GRPO reflects a theory about which capabilities benefit from which training signal:

Reasoning, factuality, instruction-following → offline DPO: These capabilities require verifying multi-step logical chains or factual claims, which reward models (being LLMs themselves) cannot reliably assess online. Pre-verified preference pairs from objective quality checks provide cleaner training signals.
Output quality, style, safety → online GRPO: These are inherently subjective and contextual qualities that reward models CAN assess reasonably well, especially when trained on human preference data across diverse outputs. Online sampling allows the model to explore the output space and receive immediate feedback.

The paper's claim is that this decomposition "enables the model to acquire those complex skills effectively" during offline RL and then "enables the model to generate responses that are precise, coherent, and well-structured while maintaining safety and readability" during online RL. The two stages are complementary rather than redundant.

Long-Context Fine-Tuning for Turbo

Qwen2.5-Turbo requires additional post-training to handle its 1M-token context window, and the approach uses a two-stage SFT strategy plus short-instruction-only RL.

SFT Stage 1: Short Instructions Only. The model is fine-tuned exclusively on short instructions (up to 32,768 tokens) using "the same data and training steps as those employed for the other Qwen2.5 models." This stage ensures strong performance on standard-length tasks and serves as a baseline before long-context specialization.

SFT Stage 2: Mixed Short and Long. The fine-tuning process combines short instructions (up to 32,768 tokens) with long instructions (up to 262,144 tokens). This hybrid approach prevents catastrophic forgetting of short-context capabilities while extending the model's effective operating range. The paper claims it "effectively enhances the model's instruction-following ability in long context tasks while maintaining its performance on short tasks."

RL Stage: Short Instructions Only. Despite Turbo's ultra-long context capability, the RL stage uses only short instructions (following the same strategy as other Qwen2.5 models). The paper gives two explicit reasons for this design choice: (1) "RL training is computationally expensive for long context tasks" (the quadratic attention cost makes GRPO with 8 samples per query at 262K tokens extremely expensive), and (2) "there is currently a scarcity of reward models that provide suitable reward signals for long context tasks" (reward models struggle to evaluate response quality when the context spans hundreds of thousands of tokens). Remarkably, the paper finds that "adopting RL on short instructions alone can still significantly enhance the model's alignment with human preferences in long context tasks," suggesting that the RL-trained behaviors (truthfulness, helpfulness, conciseness) transfer to long contexts even when the RL training was only on short contexts.

Summary of Design Choices and Their Justifications

This section is not a list of methods—it explains why each choice was made over alternatives.

Qwen2-Instruct as data filter: Using a previous-generation model to curate training data for the next generation creates a bootstrapping quality flywheel. The alternative—rule-based filters or simple classifiers—lacks the linguistic and semantic understanding to distinguish high-quality from low-quality content in diverse languages and domains.

Math and code data in pre-training rather than post-training only: The paper's results suggest that specialized reasoning capabilities benefit from being established during pre-training when the model is building its fundamental representations. Deferring to post-training may leave performance on the table because the neural circuits for mathematical reasoning are more plastic and integrative during pre-training than during fine-tuning.

Offline DPO for reasoning, online GRPO for style: This decomposition reflects an accurate assessment of current reward model limitations. Reward models cannot reliably grade mathematical proofs or multi-step logical deductions (they hallucinate evaluations), so pre-verified preference pairs provide cleaner signals for these capabilities. Conversely, reward models CAN assess output quality dimensions like conciseness and helpfulness reasonably well, making online exploration through GRPO suitable.

Progressive context expansion with 40/60 long/short mixture: Expanding context length in stages (rather than jumping directly to 262K) allows the model to gradually adapt its positional representations. The 40% long/60% short ratio at each stage prevents catastrophic forgetting of short-context performance by ensuring the model continues to see diverse sequence lengths.

YARN + DCA rather than full attention at long context: Full attention for 1M tokens is computationally prohibitive (quadratic in sequence length). The combination of YARN (position encoding extension) and DCA (chunk-based sparse attention) provides a practical path to ultra-long context that preserves strong performance on both long and short sequences, as demonstrated in Tables 16–17.

Two-epoch SFT with 1M+ examples and aggressive filtering: The paper's "flawless by all scoring systems" retention criterion for SFT data is intentionally conservative. By training for two epochs on a smaller set of extremely high-quality examples rather than one epoch on a larger noisier set, the model sees each clean example twice—reinforcing correct behaviors without contamination from flawed examples.

Online Merging Optimizer for DPO: The Online Merging Optimizer (Lu et al., 2024a) is a specialized optimizer for preference optimization that addresses the "alignment tax"—the tendency for RLHF to degrade the model's general capabilities while improving alignment. The paper cites it as a mechanism to "boost rewards and mitigate tax," making it a deliberate choice to preserve pre-existing capabilities during DPO training.

GRPO query prioritization by score variance: This is a curriculum learning strategy: train on the hardest (highest-variance) queries first, then refine on the easier ones. High variance means the model is uncertain or inconsistent, providing steeper gradients and more informative updates. Low variance means the model has already converged for that query type, making further training on it less valuable.

4. Key Insights and Innovations

Innovation 1: Pre-Training Data Quality Engineering as a First-Class Scientific Discipline

The dominant narrative in the LLM scaling literature, from Kaplan et al. (2020) through Hoffmann et al. (2022) to Llama-3 (Dubey et al., 2024), treats pre-training data primarily as a quantity variable: more tokens → lower loss → better downstream performance. When quality is discussed, it is usually in broad strokes ("we filtered out toxic content," "we deduplicated") rather than as a systematically optimized lever with quantifiable impact. Qwen2.5 challenges this framing by demonstrating that data quality engineering—deliberate filtration, domain rebalancing, synthetic data augmentation, and staged mixture scheduling—can produce capability gains that rival or exceed those from parameter scaling.

The intellectual move here is subtle but significant. The paper does not simply claim "our data is high quality"; it operationalizes data quality through a specific methodological stack: using Qwen2-Instruct as a multidimensional quality filter (a bootstrapping flywheel where generation n curates data for generation n+1), applying domain classification to deliberately down-sample overrepresented web domains while up-sampling science and technology content, integrating specialized math and code corpora during pre-training rather than deferring them to post-training, and subjecting synthetic data to dual-filter validation through both general and domain-specific reward models. Each of these techniques exists in prior work; what is distinctive is treating them as an integrated data engineering discipline whose collective impact is on par with the 2.6× increase in token count (from 7T to 18T).

The evidence that makes this more than a "we cleaned our data" claim is the base model MATH benchmark comparison: Qwen2.5-72B base achieves 62.1 vs. Qwen2-72B base's 50.9—a 22% relative improvement before any instruction tuning. Since both models share the same architecture and differ primarily in pre-training data quality and quantity, this isolated data scaling effect is substantial. Similarly, Qwen2.5-7B base reaches 49.8 on MATH vs. Llama3-8B's 20.5 at comparable parameter counts—a gap that cannot be explained by parameter count, training FLOPs, or architecture alone, and must be attributed primarily to the deliberate enrichment of mathematical content during pre-training.

Prior to this work, the open-weight community's implicit assumption was that data quality improvements were incremental refinements—important hygiene but not capability-defining. Qwen2.5's results suggest they are instead a multiplicative factor: high-quality pre-training data does not just raise the baseline; it determines whether post-training can unlock capabilities at all. This reframes the data engineering pipeline from an operational detail to a core research contribution.

Innovation 2: Decomposable Post-Training as a Theory of Capability Acquisition

The standard post-training paradigm for instruction-tuned LLMs (Ouyang et al., 2022; Touvron et al., 2023b; Dubey et al., 2024) applies SFT followed by a single, undifferentiated RLHF stage. The implicit assumption is that RLHF is a general alignment mechanism: it improves whatever dimensions human raters evaluate, and separating it into stages is unnecessary because a single reward model can capture all relevant preferences.

Qwen2.5 introduces a theoretical distinction that challenges this assumption: capabilities vary in their verifiability by learned reward models, and the optimal training signal differs accordingly. Specifically, the paper argues that reasoning, factuality, and instruction-following require verifying multi-step logical or factual chains that reward models (being LLMs themselves) hallucinate when evaluating. These capabilities benefit from offline verified signals—preference pairs where correctness is established through execution feedback, answer matching, or human verification before training begins. Conversely, output qualities like conciseness, helpfulness, harmlessness, and stylistic appropriateness are inherently subjective and contextual—reward models can assess them reasonably well online, making exploration-based RL suitable.

This is not an architectural innovation—DPO and GRPO are established algorithms from Rafailov et al. (2023) and Shao et al. (2024), respectively. It is a conceptual innovation about the structure of capability acquisition: the paper posits that different capabilities have different "learnability surfaces" with respect to reward signals, and a single-phase RL approach conflates capabilities with incompatible training requirements. The offline DPO stage addresses capabilities where reward model noise is the binding constraint (you need verified correctness, not estimated preference); the online GRPO stage addresses capabilities where exploration diversity is the binding constraint (you need the model to sample varied outputs and receive relative quality feedback).

The significance of this decomposition extends beyond the specific DPO-then-GRPO ordering. It opens a research question that the paper does not fully answer but definitively raises: what is the optimal allocation of post-training compute across different types of training signal? If verified offline signals are more expensive to construct (requiring execution environments, human review, or answer verification) but produce more reliable updates, while online signals are cheaper but noisier, then the post-training process becomes an optimization problem over data acquisition strategies. This parallels the pre-training scaling law question but for post-training, and it suggests that future work should characterize the "scaling laws of post-training signal types."

Evidence for the effectiveness of the decomposition is distributed across the benchmark results rather than isolated in a single ablation. The paper does not provide a controlled experiment comparing two-stage vs. single-stage RL (this is a limitation—see Section 6 of the analysis), but the aggregate performance improvements on reasoning-heavy benchmarks (MATH: 83.1 for Qwen2.5-72B-Instruct vs. 69.0 for Qwen2-72B-Instruct; GPQA: 49.0 vs. 42.4) and alignment benchmarks simultaneously (Arena-Hard: 81.2 vs. 48.1; IFEval: 84.1 vs. 77.6) are consistent with the claim that decomposing RL preserves reasoning gains from offline training while adding human preference alignment from online training.

Innovation 3: Small Models as First-Class Citizens—The "Capability Density" Reframing

The LLM field has a well-documented tendency to focus attention on the largest available model in any release. Llama-3's headline result was the 405B model; GPT-4's capabilities defined a generation. Smaller models are typically treated as compressed or distilled versions—useful for deployment but intellectually derivative. Qwen2.5 inverts this framing by treating model size as a spectrum to be optimized across, with deliberate investment in making the 0.5B, 1.5B, 3B, 7B, 14B, and 32B models independently competitive rather than merely scaled-down afterthoughts.

The evidence for this being more than marketing is the quantitative pattern: Qwen2.5-3B-Instruct (with only 2.8B non-embedding parameters) achieves 65.9 on MATH, outperforming the larger Phi3.5-mini (3.6B, 48.5) and MiniCPM3-4B (4.0B, 46.6). Qwen2.5-7B base reaches 49.8 on MATH—matching some 70B-class models from the previous generation. Qwen2.5-0.5B base outperforms Gemma2-2.6B on several math and coding benchmarks (GSM8K: 41.6 vs. 30.3; MATH: 19.5 vs. 18.3; HumanEval: 30.5 vs. 19.5) despite having ~5× fewer parameters. These are not marginal gains from aggressive compression; they represent genuine improvements in what the paper might call capability density—the amount of useful capability extracted per parameter.

The conceptual contribution here is the demonstration that the same data engineering and training methodology investments that benefit large models produce disproportionately large benefits for small models. This is non-obvious: one might expect that small models, with their limited capacity, would saturate earlier and benefit less from enriched training data. The paper's results suggest the opposite—that small models benefit more from high-quality, domain-targeted pre-training data because their limited capacity makes every parameter more precious. A 0.5B model trained on unfiltered web data wastes capacity on template boilerplate and low-quality content; the same model trained on Qwen2.5's curated mixture concentrates its representational budget on content that matters for downstream capabilities.

If this pattern generalizes, it has significant implications for the field's research priorities. It suggests that investment in data quality engineering and training methodology may have higher marginal returns at small model sizes than at large ones—the opposite of the conventional wisdom that "scaling engineering" is what makes large models work and small models are just shrunk versions. It also raises the possibility that the "emergent abilities" narrative (Wei et al., 2022) may partially reflect not an inherent property of model scale but a property of the data that larger models are typically trained on—when small models are trained on equivalently curated data, they may exhibit capabilities previously thought to require scale.

Innovation 4: Context Length Extension as a Training Design Problem, Not an Inference Hack

The dominant approach to extending LLM context length beyond training has been inference-time intervention: apply RoPE interpolation (Chen et al., 2023; Peng et al., 2023) or alternative position encodings at inference without additional training, accepting some quality degradation. More sophisticated approaches use lightweight continued training (Xiong et al., 2023) to adapt the model to longer sequences. Qwen2.5 stakes out a more ambitious position: context length extension should be treated as a curriculum design problem integrated into both pre-training and post-training, with deliberate staging, data mixture control, and task-specific tradeoffs.

What distinguishes this from prior long-context work is the specificity of the design choices and their motivation. For standard models, the two-phase approach (4K pre-training → 32K extension with ABF increasing RoPE base frequency to 1,000,000) is relatively standard. But for Qwen2.5-Turbo, the progressive four-stage expansion (32K → 65K → 131K → 262K) with precisely 40% max-length and 60% shorter sequences at each stage represents an explicit curriculum: the model never faces a distributional shock; it smoothly adapts its positional representations at each step while maintaining generalization across lengths. The choice of 40/60 rather than 50/50 or some other ratio reflects an optimization over two competing objectives (long-context performance and short-context retention) that the paper could have treated as a hyperparameter to tune but instead treated as a curriculum design principle.

The post-training insight is equally significant: Turbo's RL stage uses only short instructions despite the model's 1M-token capability. The paper explicitly justifies this with two constraints—computational cost (quadratic attention at 262K tokens with 8 samples per query is prohibitive) and reward model scarcity (no reward model can reliably evaluate response quality at ultra-long context). Yet they find that short-instruction RL transfers effectively to long-context behavior. This is a negative result with positive implications: it suggests that the human preference dimensions learned through online RL (truthfulness, helpfulness, conciseness) are context-length invariant—a model that learns to be concise and truthful on short queries generalizes these behaviors to long queries without explicit long-context RL training. If validated, this substantially reduces the cost barrier to aligning ultra-long-context models.

The ablation in Tables 16–17 quantifies the contribution of the inference-time extrapolation components (YARN+DCA) versus the training curriculum. Without YARN+DCA, RULER scores at 128K context drop dramatically: Qwen2.5-72B-Instruct falls from 88.4 to 67.0; Qwen2.5-7B-Instruct from 55.1 to 31.4. This demonstrates that the training curriculum alone is insufficient—the inference-time techniques provide a multiplicative factor on top of the training foundation. The combination of both (training curriculum for positional adaptation + inference techniques for further extension) is what achieves the paper's claimed four-fold context expansion.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two tiers of evaluation:
- Public benchmarks for standardized comparison against other models: MMLU, MMLU-Pro, MMLU-redux, BBH, ARC-C, TruthfulQA, Winogrande, HellaSwag (general tasks); GPQA, TheoremQA, MATH, MMLU-stem, GSM8K (mathematics and science); HumanEval, HumanEval+, MBPP, MBPP+, MultiPL-E, LiveCodeBench (coding); and a suite of multilingual benchmarks grouped into exam, understanding, mathematics, and translation categories. For instruction-tuned models, additional benchmarks include IFEval, MT-Bench, Arena-Hard, LiveBench 0831, and Reward Bench for alignment assessment.
- In-house automatic evaluation benchmarks in both English and Chinese, covering instruction following, knowledge utilization, comprehension, coding, math, and reasoning. These are proprietary datasets designed to "assess various aspects of model performance" that the authors believe public benchmarks insufficiently capture.
- Long-context benchmarks: RULER, LV-Eval, and LongBench-Chat at context lengths up to 256K tokens.
- Reward model benchmarks: Reward Bench, RMB, PPE, and an internally collected out-of-domain Chinese human preference benchmark (Human-Preference-Chinese).
Base model(s). The Qwen2.5 series includes dense models at 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters, and MoE models (Qwen2.5-Turbo and Qwen2.5-Plus) for API service. Base models are evaluated after pre-training; instruction-tuned models after the full post-training pipeline (SFT + offline DPO + online GRPO). Comparisons are made against prior Qwen versions (Qwen1.5, Qwen2) and leading open-weight models at comparable sizes: Llama-3 / Llama-3.1 (70B, 8B, 405B), Mistral-7B, Mixtral-8x22B, Gemma2 (2.6B, 9B, 27B), Yi-1.5-34B, Phi3.5-Mini, and MiniCPM3-4B, along with proprietary models GPT-4o, GPT-4o-mini, and Claude3.5-sonnet where noted.
Metrics. All benchmarks report accuracy (percentage of correctly answered questions) as the primary metric, with dataset-specific evaluation protocols: 5-shot for MMLU, MMLU-Pro, MMLU-redux; 0-shot for TruthfulQA, HumanEval, MBPP; 3-shot for BBH; 4-shot for GSM8K and MATH; etc. For long-context evaluation, RULER reports aggregate accuracy across length levels, LV-Eval uses keyword recall to "mitigate the high rate of false negatives present in the original metrics," and LongBench-Chat reports a composite score. For alignment, Arena-Hard reports a score from 0–100, MT-Bench uses GPT-4 as judge on a 1–10 scale, IFEval reports strict-prompt subset accuracy, and LiveBench 0831 reports an aggregate score. For reward model evaluation, Reward Bench and RMB report accuracy/preference metrics; PPE reports both human preference accuracy and objective benchmark scores; Human-Preference-Chinese reports accuracy. The paper does not report confidence intervals or statistical significance tests for any benchmark comparisons.
Baselines. The paper compares against an extensive set of prior and contemporary models organized by size tier:
- 70B+ class (Table 2 for base models, Table 6 for instruct): Llama-3-70B, Llama-3.1-70B-Instruct, Llama-3-405B, Llama-3.1-405B-Instruct (Dubey et al., 2024), Mixtral-8x22B (Jiang et al., 2024a), Qwen2-72B and Qwen2-72B-Instruct (Yang et al., 2024a), GPT-4o, GPT-4o-mini (OpenAI, 2024a).
- 14B–34B class (Table 3 for base models, Table 7 for instruct): Yi-1.5-34B (Young et al., 2024), Gemma2-27B (Gemma Team et al., 2024), Qwen1.5-32B (Qwen Team, 2024b), Qwen2-57BA14B-Instruct (a Qwen2 MoE model with 14B activated parameters), GPT-4o-mini.
- 7B–9B class (Table 4 for base models, Table 8 for instruct): Mistral-7B (Jiang et al., 2023a), Llama3-8B / Llama3.1-8B-Instruct (Dubey et al., 2024), Gemma2-9B (Gemma Team et al., 2024), Qwen2-7B (Yang et al., 2024a).
- 0.5B–4B class (Table 5 for base models, Tables 9–10 for instruct): Qwen2-0.5B / 1.5B (Yang et al., 2024a), Gemma2-2.6B (Gemma Team et al., 2024), Gemma2-2B, Phi3.5-Mini (Abdin et al., 2024), MiniCPM3-4B (Hu et al., 2024).
- Reward models (Table 15): Nemotron-4-340B-Reward (Adler et al., 2024), Llama-3.1-Nemotron-70B-Reward (Wang et al., 2024c), Athene-RM-70B (Frick et al., 2024a).
Generation budget / compute accounting. The paper does not report a unified compute budget or FLOPs count for its comparisons. For base models, comparison is on benchmark performance alone with no generation budget constraint. For instruction-tuned models, evaluation follows standard benchmark protocols with no explicit compute budget normalization—models are compared on accuracy regardless of inference cost. This differs from the reference example paper's approach of comparing methods at matched generation budgets. The paper notes that MoE models (Turbo, Plus) have "significantly lower training and inference costs" than comparable dense models, but does not quantify these costs in the evaluation section. For long-context inference speed, Figure 3 reports time-to-first-token (TTFT) speedups of 3.2–4.3× using the sparse attention mechanism relative to full attention on various hardware configurations, but this is not integrated into the accuracy comparisons.
Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance testing for any benchmark results. Test data contamination is addressed through n-gram matching during data construction: a training sequence is removed if it shares a longest common subsequence with a test sequence that is both ≥13 tokens long and ≥60% of the minimum sequence length. This is the same protocol used in Qwen2. No further statistical methodology is described.

Main Quantitative Results

Base Model Performance at the 70B+ Scale

The flagship Qwen2.5-72B base model achieves 86.1 on MMLU, surpassing Llama-3-405B (85.2) and Qwen2-72B (84.2), while trailing Llama-3-405B only on MMLU-Pro (58.1 vs. 61.6) (Table 2). On BBH, Qwen2.5-72B scores 86.3, exceeding Llama-3-405B (85.9) and marking a 3.9-point improvement over Qwen2-72B (82.4).

The mathematics gains are the most dramatic: Qwen2.5-72B reaches 62.1 on MATH (4-shot), compared to Qwen2-72B's 50.9 (+11.2 points) and Llama-3-405B's 53.8 (+8.3 points). On GSM8K, Qwen2.5-72B scores 91.5 vs. Llama-3-405B's 89.0 and Qwen2-72B's 89.0 (+2.5 points). On the graduate-level GPQA, Qwen2.5-72B achieves 45.9, substantially ahead of Qwen2-72B (37.4) and Llama-3-70B (36.3)—though Llama-3-405B is not reported for this benchmark.

Coding performance shows a more mixed picture: Qwen2.5-72B reaches 59.1 on HumanEval, which is below Qwen2-72B's 64.6 (a 5.5-point regression), and trails Llama-3-405B's 61.0. However, on MBPP, Qwen2.5-72B achieves 84.7—the highest in the table, surpassing Llama-3-405B (73.0), Qwen2-72B (76.9), and Mixtral-8x22B (71.7). This discrepancy (HumanEval down, MBPP up) is not discussed in the paper but suggests the coding data mixture may favor certain programming patterns over others.

Multilingual performance improves substantially: Qwen2.5-72B reaches 89.6 on Multi-Understanding (vs. Qwen2-72B's 80.7, a +8.9 point gain), 78.7 on Multi-Exam (vs. 76.6), and 39.0 on Multi-Translation (vs. 37.8). The Multi-Understanding gain is the single largest improvement across all benchmark categories in Table 2.

Qwen2.5-Plus, the proprietary MoE model, achieves 64.0 on MMLU-Pro—5.9 points higher than Qwen2.5-72B (58.1), and 2.4 points above Llama-3-405B (61.6). It also leads on TheoremQA (48.5 vs. Qwen2.5-72B's 42.4), MATH (64.4 vs. 62.1), GSM8K (93.0 vs. 91.5), and Multi-Mathematics (82.4 vs. 76.7). On MMLU and BBH, Plus slightly trails the dense 72B model (85.4 vs. 86.1, 85.8 vs. 86.3). The paper notes that Plus has "significantly lower training and inference costs" than the 72B dense model, but does not quantify these costs.

Base Model Performance at the 14B–32B Scale

Qwen2.5-32B achieves 83.3 on MMLU—outperforming all listed larger models except Qwen2.5-72B and Llama-3-405B, and substantially ahead of Gemma2-27B (75.2) and Yi-1.5-34B (77.2) (Table 3). On BBH, 32B scores 84.5, exceeding Qwen2.5-72B's 86.3 only by a small margin, suggesting BBH performance may be saturating at this scale.

On MATH, Qwen2.5-32B reaches 57.7 (matching Qwen2.5-14B's 55.6, and within 4.4 points of Qwen2.5-72B's 62.1). GSM8K shows 32B at 92.9—the highest in its size class and competitive with larger models. On GPQA, 32B achieves 48.0, which is notably 2.1 points above Qwen2.5-72B (45.9)—the only benchmark where the 32B model meaningfully exceeds the 72B model.

Qwen2.5-14B scores 79.7 on MMLU and 78.2 on BBH, exceeding larger models like Yi-1.5-34B (77.2 MMLU) and Gemma2-27B (75.2 MMLU, 74.9 BBH). On MATH, 14B reaches 55.6—matching Llama-3-405B (53.8) and nearly matching Qwen2.5-32B (57.7), while being much smaller. This is one of the paper's strongest demonstrations of the "capability density" concept.

Qwen2.5-Turbo, with "significantly smaller" training and inference costs than 14B, achieves 79.5 MMLU (comparable to 14B's 79.7), 55.6 MMLU-Pro (above 32B's 55.1), and 41.4 GPQA (above 14B's 32.8). On MATH, Turbo scores 55.6—identical to 14B and matching the average of the 14B–32B range. Its coding performance (57.3 HumanEval, 76.2 MBPP) is comparable to 14B's.

Base Model Performance at the 7B Scale

Qwen2.5-7B demonstrates the most striking generational improvement. On MATH, it reaches 49.8—a 6.3-point improvement over Qwen2-7B (43.5) and more than double Llama3-8B's 20.5 and Mistral-7B's 10.2 (Table 4). This MATH score puts Qwen2.5-7B at roughly the same level as Qwen2-72B base (50.9) from the previous generation, despite being ~10× smaller.

On GSM8K, Qwen2.5-7B scores 85.4, exceeding Qwen2-7B (80.2), Llama3-8B (55.3), and Gemma2-9B (70.7). On MMLU, it reaches 74.2—improving over Qwen2-7B (70.3), Llama3-8B (66.6), Mistral-7B (64.2), and Gemma2-9B (71.3). Note that Gemma2-9B has 8.2B non-embedding parameters while Qwen2.5-7B has only 6.5B, yet Qwen2.5-7B outperforms it on 10 of 15 benchmarks in Table 4.

On HumanEval, Qwen2.5-7B achieves 57.9—a 6.7-point improvement over Qwen2-7B (51.2) and well ahead of Gemma2-9B (37.8). On MBPP, it reaches 74.9 vs. Qwen2-7B's 64.2 (+10.7 points). MultiPL-E shows 50.3 vs. Qwen2-7B's 41.0 (+9.3 points), indicating the coding improvements extend across multiple programming languages.

On BBH, Qwen2.5-7B scores 70.4, exceeding Qwen2-7B (62.3), Llama3-8B (57.7), and Gemma2-9B (68.2). GPQA reaches 36.4 vs. Qwen2-7B's 30.8. The only benchmarks where Qwen2.5-7B does not clearly lead are HellaSwag (80.2, marginally behind Gemma2-9B's 81.9 and Mistral-7B's 83.3) and Winogrande (75.9, below Gemma2-9B's 79.5).

Base Model Performance at the Smallest Scales (0.5B–3B)

Qwen2.5-3B achieves 65.6 on MMLU—outperforming Gemma2-2.6B (52.2) by 13.4 points and Qwen2-1.5B (55.9) by 9.7 points, while approaching Qwen2-7B's 70.3 (Table 5). On MATH, 3B reaches 42.6—more than double Gemma2-2.6B's 18.3 and substantially ahead of Qwen2-1.5B's 21.6. On GSM8K, 3B scores 79.1 vs. Gemma2-2.6B's 30.3—a 48.8-point gap that suggests the math pre-training data integration has outsized effects at small scales where every parameter's representational budget must be efficiently allocated.

Qwen2.5-1.5B shows dramatic improvements over Qwen2-1.5B: MMLU 60.9 vs. 55.9 (+5.0), MATH 35.0 vs. 21.6 (+13.4), GSM8K 68.5 vs. 46.9 (+21.6), HumanEval 37.2 vs. 34.8 (+2.4), MultiPL-E 33.1 vs. 27.9 (+5.2). The MATH and GSM8K gains are proportionally larger than those at the 7B level, consistent with the interpretation that data quality benefits are amplified when model capacity is constrained.

Qwen2.5-0.5B, the smallest model, achieves 47.5 MMLU (vs. Qwen2-0.5B's 44.3), 19.5 MATH (vs. 11.2), 41.6 GSM8K (vs. 36.4), and 30.5 HumanEval (vs. 22.6). Notably, on GSM8K, 0.5B (41.6) outperforms Gemma2-2.6B (30.3) despite having ~5× fewer parameters. On MATH, 0.5B (19.5) slightly exceeds Gemma2-2.6B (18.3). On HumanEval, 0.5B (30.5) substantially exceeds Gemma2-2.6B (19.5) and Qwen2-1.5B (34.8 is close).

A notable anomaly: on GPQA, Qwen2-0.5B (29.8) outperforms Qwen2.5-0.5B (24.8), Qwen2.5-1.5B (24.2), and Qwen2.5-3B (26.3). This is the only benchmark where a Qwen2 model clearly beats its Qwen2.5 successor, and the paper does not discuss this regression. Given that GPQA is a graduate-level science benchmark requiring extensive domain knowledge, it is possible that the increased emphasis on math and code in Qwen2.5's pre-training mixture came at the expense of science knowledge at very small scales, but this is speculative without ablation data.

Instruction-Tuned Model Performance: Open Benchmarks

70B+ Scale (Table 6)

Qwen2.5-72B-Instruct achieves 83.1 on MATH—a 14.1-point improvement over Qwen2-72B-Instruct (69.0) and 9.3 points above Llama-3.1-405B-Instruct (73.8). On Arena-Hard, it reaches 81.2 vs. Qwen2-72B-Instruct's 48.1 (+33.1 points, a 69% relative improvement) and Llama-3.1-405B-Instruct's 69.3 (+11.9 points). This is the single largest relative improvement in the instruction-tuned benchmarks and reflects the combined effect of SFT, DPO, and GRPO on human preference alignment.

On MMLU-Pro, Qwen2.5-72B-Instruct scores 71.1, exceeding Qwen2-72B-Instruct (64.4) and Llama-3.1-70B-Instruct (66.4), but trailing Llama-3.1-405B-Instruct (73.3). On MMLU-redux, it reaches 86.8, slightly ahead of Llama-3.1-405B-Instruct (86.2). On GPQA, it scores 49.0 vs. Qwen2-72B-Instruct's 42.4 and Llama-3.1-70B-Instruct's 46.7, but behind Llama-3.1-405B-Instruct's 51.1.

Coding benchmarks show strong gains: HumanEval 86.6 (vs. Qwen2-72B-Instruct's 86.0, essentially flat), MBPP 88.2 (vs. 80.2, +8.0 points), MultiPL-E 75.1 (vs. 69.2, +5.9 points), and LiveCodeBench 55.5 (vs. Qwen2-72B-Instruct's 32.2, +23.3 points, and Llama-3.1-405B-Instruct's 41.6, +13.9 points). LiveCodeBench's massive jump is particularly notable because it is designed as a contamination-free benchmark that tests on recent coding problems—suggesting the improvements are genuine capability gains rather than memorization.

On alignment benchmarks, MT-Bench reaches 9.35—the highest in the table, above Llama-3.1-405B-Instruct (9.08) and Qwen2-72B-Instruct (9.12). IFEval scores 84.1 vs. Llama-3.1-405B-Instruct's 86.0—one of the few benchmarks where the 405B model retains an advantage.

Qwen2.5-Plus achieves 72.5 MMLU-Pro (above 72B-Instruct's 71.1), 84.7 MATH (above 83.1), 87.8 HumanEval (above 86.6), 77.0 MultiPL-E (above 75.1), and 86.3 IFEval (above 84.1). On Arena-Hard, Plus scores 81.4—essentially tied with 72B-Instruct's 81.2. On LiveCodeBench, Plus scores 51.4 vs. 72B-Instruct's 55.5, suggesting the dense model may have an edge on recent coding problems.

14B–32B Scale (Table 7)

Qwen2.5-32B-Instruct scores 83.1 on MATH—equal to the 72B-Instruct model and exceeding GPT-4o-mini (70.2) and Gemma2-27B-IT (54.4). On Arena-Hard, it reaches 74.5 vs. 72B-Instruct's 81.2 (a 6.7-point gap), but well above GPT-4o-mini (74.9, essentially tied) and Qwen2-57BA14B-Instruct (17.8). On LiveCodeBench, 32B-Instruct achieves 51.2—trailing 72B-Instruct (55.5) but exceeding the 14B-Instruct (42.6) and GPT-4o-mini (40.7).

Qwen2.5-14B-Instruct reaches 80.0 on MATH—competitive with 32B-Instruct (83.1) and well above its predecessor Qwen2-57BA14B-Instruct (49.1). On MMLU-Pro, 14B-Instruct scores 63.7 vs. 32B-Instruct's 69.0. On Arena-Hard, 14B-Instruct achieves 68.3 vs. 32B-Instruct's 74.5. On GPQA, 14B-Instruct (45.5) is competitive with 32B-Instruct (49.5). Across all benchmarks, 14B-Instruct is positioned by the paper as "rivaling those of GPT-4o-mini"—a comparison that holds reasonably well: 14B-Instruct trails on MMLU-Pro (63.7 vs. 63.1), MMLU-redux (80.0 vs. 81.5), MATH (80.0 vs. 70.2, a clear win), and HumanEval (83.5 vs. 88.4, a clear loss).

Qwen2.5-Turbo achieves 64.5 MMLU-Pro (above 14B-Instruct's 63.7), 81.1 MATH (above 14B-Instruct's 80.0), 42.3 GPQA (below 14B-Instruct's 45.5), and 67.1 Arena-Hard (below 14B-Instruct's 68.3). The paper notes that Turbo "outperforms Qwen2.5-14B-Instruct on eight out of ten benchmarks" in this table despite significantly lower training and inference costs.

7B Scale (Table 8)

Qwen2.5-7B-Instruct reaches 75.5 on MATH—a 22.6-point improvement over Qwen2-7B-Instruct (52.9) and substantially above Gemma2-9B-IT (44.3) and Llama3.1-8B-Instruct (51.9). This MATH score exceeds Qwen2-72B base (50.9) and Qwen2-72B-Instruct (69.0), meaning a 7B model from the current generation outperforms the previous generation's 72B instruction-tuned flagship on mathematical reasoning.

On GSM8K, 7B-Instruct scores 91.6 vs. Qwen2-7B-Instruct's 85.7 (+5.9). On HumanEval, it reaches 84.8 vs. 79.9 (+4.9). On Arena-Hard, 7B-Instruct achieves 52.0—more than double Qwen2-7B-Instruct's 25.0 and well above Gemma2-9B-IT (41.6) and Llama3.1-8B-Instruct (27.8). On MMLU-Pro, it scores 56.3 vs. Qwen2-7B-Instruct's 44.1 (+12.2 points). On LiveBench 0831, it reaches 35.9 vs. 29.2 (+6.7).

The one benchmark where Qwen2.5-7B-Instruct does not lead is IFEval (71.2), where it trails Llama3.1-8B-Instruct (75.9) and Gemma2-9B-IT (70.1—essentially tied).

0.5B–3B Scale (Tables 9–10)

Qwen2.5-3B-Instruct achieves 65.9 on MATH—a dramatic result for a model with only 2.8B non-embedding parameters. This exceeds Phi3.5-Mini (3.6B, 48.5) by 17.4 points and MiniCPM3-4B (4.0B, 46.6) by 19.3 points. On GSM8K, 3B-Instruct scores 86.7, matching Phi3.5-Mini (86.2) and ahead of MiniCPM3-4B (81.1). On HumanEval, it reaches 74.4—essentially tied with Phi3.5-Mini (72.6) and MiniCPM3-4B (74.4). The coding advantage is most pronounced on MultiPL-E, where 3B-Instruct scores 60.2 vs. Phi3.5-Mini's 47.2 and MiniCPM3-4B's 49.1.

On MMLU-Pro, however, 3B-Instruct (43.7) trails Phi3.5-Mini (47.5), suggesting the math/code emphasis may come at some cost to general knowledge breadth at small scales. On MMLU-redux, 3B-Instruct (64.4) trails Phi3.5-Mini (67.7). On IFEval, 3B-Instruct (58.2) lags MiniCPM3-4B (68.4) significantly—instruction-following may require more capacity to internalize than the 3B model can provide.

Qwen2.5-1.5B-Instruct achieves 55.2 on MATH—a 29.9-point improvement over Qwen2-1.5B-Instruct (25.3), and exceeding Qwen2-7B base (43.5) and Qwen2-7B-Instruct (52.9). On GSM8K, it reaches 73.2 vs. Qwen2-1.5B-Instruct's 61.6 (+11.6). On HumanEval, it scores 61.6 vs. 42.1 (+19.5). These are proportionally the largest gains of any model tier across the Qwen2 → Qwen2.5 transition.

Qwen2.5-0.5B-Instruct reaches 34.4 on MATH—compared to Qwen2-0.5B-Instruct's 13.9 (+20.5 points, a 147% relative improvement). On GSM8K, 49.6 vs. 40.1 (+9.5). On LiveCodeBench, 5.1 vs. 1.6 (+3.5). The 0.5B instruct model outperforms Qwen2-1.5B-Instruct on several benchmarks (MATH: 34.4 vs. 25.3; LiveBench: 12.6 vs. 12.4; IFEval: 27.9 vs. 29.0), suggesting that data quality improvements can make a 0.5B model competitive with the previous generation's 1.5B model.

In-House Automatic Evaluation

English Evaluation (Table 11)

The in-house English benchmark covers six categories: Instruction Following, Knowledge, Comprehension, Coding, Math, and Reasoning. Qwen2.5-72B-Instruct achieves 82.65 on IF, 66.09 on Knowledge, 74.43 on Comprehension, 60.41 on Coding, 59.73 on Math, and 65.90 on Reasoning. Compared to GPT-4o-2024-08-06 (83.28 IF, 68.08 Knowledge, 76.51 Comprehension, 58.05 Coding, 52.36 Math, 66.45 Reasoning), Qwen2.5-72B-Instruct substantially leads on Math (+7.37 points) and Coding (+2.36), is competitive on Comprehension (−2.08) and Reasoning (−0.55), and trails on IF (−0.63) and Knowledge (−1.99).

Versus Claude3.5-sonnet-2024-10-22 (84.22 IF, 74.61 Knowledge, 79.02 Comprehension, 67.17 Coding, 48.67 Math, 70.20 Reasoning), Qwen2.5-72B-Instruct leads on Math (+11.06 points) but trails meaningfully on IF (−1.57), Knowledge (−8.52), Comprehension (−4.59), Coding (−6.76), and Reasoning (−4.30). This pattern—strong math, weaker knowledge and comprehension—is consistent with the open benchmark results.

Versus Llama-3.1-405B-Instruct (83.33 IF, 67.10 Knowledge, 75.55 Comprehension, 58.14 Coding, 47.09 Math, 64.74 Reasoning), Qwen2.5-72B-Instruct leads on Math (+12.64 points) and Coding (+2.27), is competitive on IF (−0.68) and Comprehension (−1.12), and slightly trails on Knowledge (−1.01) but leads on Reasoning (+1.16).

Qwen2.5-32B-Instruct (76.79 IF, 64.08 Knowledge, 71.28 Comprehension, 58.90 Coding, 60.97 Math, 65.49 Reasoning) remarkably exceeds Qwen2-72B-Instruct on every single metric—in some cases substantially: IF 76.79 vs. 76.08, Math 60.97 vs. 48.07 (+12.90 points), Reasoning 65.49 vs. 60.33 (+5.16). This means Qwen2.5-32B-Instruct is unambiguously better than the previous generation's 72B flagship.

The progressive improvement across the Qwen2.5 series is visible in the table: each step up in model size (0.5B → 1.5B → 3B → 7B → 14B → 32B → 72B) produces monotonic gains on all metrics. The 3B model (60.60 IF) substantially exceeds Qwen2-7B-Instruct (50.47 IF), and the 7B model (70.01 IF) approaches Qwen2-72B-Instruct (76.08 IF). This scaling trend is also visible in the Llama-3.1 series (70B → 405B) but the Qwen2.5 gains are steeper across comparable size intervals.

Qwen2.5-Plus achieves 83.18 IF—the only model in the table to exceed GPT-4o-2024-08-06 (83.28) on instruction following, and the highest Comprehension score (79.35) after Claude3.5-sonnet (79.02, which Plus exceeds). This is notable: a proprietary MoE model outperforms both GPT-4o and Claude3.5-sonnet on Comprehension in this in-house evaluation.

Chinese Evaluation (Table 12)

On the Chinese in-house benchmark, Qwen2.5-72B-Instruct scores 37.22 IF, 75.86 Knowledge, 78.85 Comprehension, 56.71 Coding, 68.39 Math, and 63.02 Reasoning. Compared to GPT-4o-2024-08-06 (42.50 IF, 68.55 Knowledge, 80.11 Comprehension, 61.53 Coding, 61.74 Math, 56.88 Reasoning), Qwen2.5-72B-Instruct leads on Knowledge (+7.31), Math (+6.65), and Reasoning (+6.14), but trails on IF (−5.28), Comprehension (−1.26), and Coding (−4.82).

Chinese instruction following is notably weaker for Qwen2.5-72B-Instruct (37.22) than English IF (82.65)—a pattern shared by all models in the comparison (GPT-4o-2024-08-06: 42.50 Chinese vs. 83.28 English). This likely reflects the Chinese IF benchmark being more challenging or differently calibrated. Qwen2.5-Plus substantially closes this gap, achieving 46.15 Chinese IF—the highest in the table after Claude3.5-sonnet (49.25), and above GPT-4o-2024-11-20 (42.71).

Qwen2.5-32B-Instruct again surpasses Qwen2-72B-Instruct on five of six metrics: Math 67.86 vs. 65.55 (+2.31), Reasoning 60.19 vs. 58.19 (+2.00), Knowledge 74.70 vs. 74.96 (−0.26), IF 32.64 vs. 31.98 (+0.66), Comprehension 79.46 vs. 75.49 (+3.97), Coding 54.45 vs. 41.57 (+12.88). The Coding improvement is particularly dramatic.

The scaling pattern is consistent: Qwen2.5-3B (16.50 IF, 57.18 Knowledge) roughly matches Qwen2-7B-Instruct (16.83 IF, 65.95 Knowledge on some metrics but not others), while Qwen2.5-7B-Instruct (26.64 IF) substantially exceeds Qwen2-72B-Instruct (31.98 IF) in trend but not yet in absolute terms.

Multilingual Evaluation (Tables 13–14)

For 70B+ instruction-tuned models (Table 13): Qwen2.5-72B-Instruct achieves 86.98 on multilingual IFEval—exceeding all listed competitors including Mistral-Large (82.69) and GPT-4o-mini (85.03). On the five MMLU-like knowledge benchmarks, it leads on AMMLU (72.44), JMMLU (80.56), KMMLU (61.96), IndoMMLU (69.25), and TurkishMMLU (76.12), with margins ranging from 1.52 to 6.89 points over the next best model. On okapi MMLU (translated), it reaches 79.97, above Mistral-Large (78.37) and Qwen2-72B (77.84). On MGSM8K extended, 72B-Instruct scores 88.16—slightly below Mistral-Large (89.01) but above GPT-4o-mini (87.36) and Qwen2-72B (82.72). On BLEnD (cultural nuances), it scores 32.48—trailing GPT-4o-mini (35.91) and Mistral-Large (33.47) but improving substantially over Qwen2-72B (25.90, +6.58 points).

For 7B–14B models (Table 14): Qwen2.5-7B-Instruct achieves 74.87 on multilingual IFEval—close to Gemma2-9B (77.47) and Qwen2.5-14B-Instruct (77.08), and well above Qwen2-7B (51.43) and Llama3.1-8B (60.68). On the five knowledge benchmarks, Qwen2.5-14B-Instruct consistently leads, with margins of 4–16 points over Qwen2.5-7B-Instruct. On MGSM8K extended, 14B-Instruct scores 82.27—above Gemma2-9B (78.37) and substantially above Qwen2-7B (56.13) and Llama3.1-8B (66.05). BLEnD scores remain modest: 23.66 for 7B, 26.99 for 14B, above Qwen2-7B (22.49) but below Gemma2-9B (28.31).

Reward Model Evaluation (Table 15)

Qwen2.5-RM-72B achieves strongest results on PPE (69.85 Objective-Avg, the highest in the table) and Human-Preference-Chinese (61.27 accuracy, also the highest), while performing competitively on Reward Bench (91.59 Score, only behind Llama-3.1-Nemotron-70B-Reward at 94.10 and Nemotron-4-340B-Reward at 92.00) and RMB (68.71 Overall, second to Athene-RM-70B at 73.98).

On Reward Bench breakdowns: Qwen2.5-RM-72B scores 97.21 on Chat (second to Athene-RM-70B at 98.32), 78.73 on Chat Hard (second to Nemotron-4-340B-Reward at 87.10), 92.71 on Safety (third), and 97.65 on Reasoning (second to Llama-3.1-Nemotron-70B-Reward at 98.10). On PPE objective benchmarks, Qwen2.5-RM-72B leads on MATH (81.48 vs. Athene-RM-70B's 79.14), IFEval (67.97 vs. Athene-RM-70B's 62.15), and is competitive on GPQA (59.80 vs. 59.26) and MMLU-Pro (75.66 vs. 76.95).

The paper notes that "Llama-3.1-Nemotron-70B-Reward excels on the Reward Bench" and "Athene-RM-70B performs best on the RMB benchmark"—the Qwen2.5 reward model does not dominate any single benchmark but achieves strong average performance. More critically, the paper observes that "current reward model evaluation benchmarks do not accurately predict the performance of the RL models trained under their guidance"—a higher score on RM benchmarks does not correlate with superior downstream RL model performance. This insight is presented as a caution against optimizing reward models for benchmark scores.

Long-Context Capabilities (Tables 16–17, Figures 2–3)

RULER (Table 16): Qwen2.5-72B-Instruct achieves 95.1 average RULER score, with performance of 97.7 (4K), 97.2 (8K), 97.7 (16K), 96.5 (32K), 93.0 (64K), and 88.4 (128K). This exceeds all listed models including GPT-4 (91.6 avg, 81.2 at 128K), GPT-4o-mini (87.3 avg, 65.8 at 128K), and Llama-3.1-70B-Instruct (89.6 avg, 66.6 at 128K). The performance is notably flat from 4K to 32K—the model maintains >96 across this range—before declining gradually to 88.4 at 128K.

Without YARN+DCA, Qwen2.5-72B-Instruct's 128K RULER score drops to 67.0 (from 88.4), and at lower lengths the scores are unchanged (within 32K, YARN+DCA "does not change the model behavior"). The degradation is more severe for smaller models: 7B-Instruct drops from 55.1 to 31.4 at 128K without YARN+DCA; 32B-Instruct drops from 82.0 to 57.7. The gap between with and without YARN+DCA widens with sequence length, consistent with these being inference-time length extrapolation techniques.

Qwen2.5-Turbo achieves 93.1 average RULER with scores of 97.5 (4K), 95.7 (8K), 95.5 (16K), 94.8 (32K), 90.8 (64K), and 84.5 (128K)—comparable to Qwen2.5-72B-Instruct but not exceeding it. The Turbo model's rated context length is 1M tokens (with progressive training to 262K plus 4× extrapolation), but RULER is only evaluated to 128K for Turbo.

LV-Eval and LongBench-Chat (Table 17): On LV-Eval, Qwen2.5-72B-Instruct scores 60.4 (16K), 57.5 (32K), 53.9 (64K), 50.9 (128K), and 45.2 (256K)—the highest scores at all lengths among listed models. Without YARN+DCA, 128K drops to 27.0 and 256K drops to 2.4, demonstrating that the training curriculum alone is insufficient for ultra-long context. Qwen2.5-Turbo achieves 53.4 (16K), 50.0 (32K), 45.4 (64K), 43.9 (128K), and 38.0 (256K)—competitive with 72B-Instruct but slightly below at all lengths.

On LongBench-Chat: Qwen2.5-72B-Instruct scores 8.72, 32B-Instruct 8.70, Turbo 8.34, 14B-Instruct 8.04—all above GPT-4o-mini (8.48) and below GPT-4 (not reported). The scores are tightly clustered (7.42–8.72), suggesting LongBench-Chat may have limited discriminative power at this performance level.

Passkey Retrieval (Figure 2): Qwen2.5-Turbo achieves 100% accuracy on the passkey retrieval task at 1M tokens across all document depths (top to bottom of document). This is a needle-in-a-haystack test: the model must retrieve a hidden number from irrelevant context spanning 1M tokens. Perfect accuracy at all depths is a strong demonstration of the progressive context expansion and YARN+DCA combination.

Inference Speed (Figure 3): Qwen2.5-Turbo with the MInference-based sparse attention achieves TTFT speedups of 3.2–4.3× compared to full attention, depending on hardware configuration. The speedup varies with GPU count: on a single GPU, 4.3× speedup is reported; on multiple GPUs, speedups range from 2.4× to 3.2×. Qwen2.5-7B shows smaller but still substantial speedups (1.4–3.1×). These speedups are critical for practical deployment at 1M-token context lengths.

Ablation Studies and Robustness Checks

YARN + DCA for long-context inference: Without these techniques, performance at 128K–256K context degrades substantially across all model sizes. For Qwen2.5-72B-Instruct on RULER at 128K: 88.4 with YARN+DCA vs. 67.0 without (Table 16). On LV-Eval at 256K: 45.2 with vs. 2.4 without (Table 17). The techniques do not affect performance within the native 32K training context, confirming they function purely as length extrapolation methods rather than general quality improvements. This ablation demonstrates that the pre-training context extension curriculum (to 32K or 262K) is necessary but insufficient—inference-time techniques multiply the effective context length.

DCA + YARN impact by model size: The degradation without YARN+DCA is more severe for smaller models. At 128K RULER: 72B drops 21.4 points (88.4 → 67.0), 32B drops 24.3 points (82.0 → 57.7), 14B drops 25.1 points (78.1 → 53.0), 7B drops 23.7 points (55.1 → 31.4). The proportional degradation is worse for smaller models because their base long-context capability is already lower—a 23-point drop from 55.1 represents a 43% relative decline vs. 24% for 72B.

Progressive context expansion for Turbo: The four-stage expansion (32K → 65K → 131K → 262K) with 40/60 long/short mixture at each stage is validated by Turbo's performance: 100% passkey retrieval at 1M tokens (Figure 2), competitive RULER scores at 128K (84.5, Table 16), and strong LV-Eval scores at 256K (38.0, Table 17). The paper does not ablate the number of stages or the 40/60 ratio, so we cannot determine whether these specific values are optimal or simply sufficient.

SFT data quality filtering: The paper's "flawless by all scoring systems" retention criterion is an ablation in spirit—only data passing multiple independent quality checks is retained. The paper does not report what fraction of SFT data is filtered out or provide a controlled comparison with a less strict filtering threshold. This is a missing ablation that would clarify the marginal value of aggressive filtering.

Two-stage RL (offline DPO + online GRPO): The paper does not provide a direct ablation comparing single-stage vs. two-stage RL. The closest available comparison is Qwen2-72B-Instruct vs. Qwen2.5-72B-Instruct on alignment benchmarks (Table 6): Arena-Hard improves from 48.1 to 81.2 (+33.1 points). However, this conflates the effect of two-stage RL with improvements in SFT data, DPO data quality, and the base model's capabilities. Without a controlled experiment where only the RL stage design varies, the specific contribution of the two-stage decomposition cannot be isolated.

DPO training data verification: The DPO stage uses both human and automated review processes to verify training pairs (Section 4.2). The paper does not ablate human review vs. automated review alone, so the marginal value of human verification in the offline RL pipeline is unknown.

GRPO query prioritization by score variance: The paper states that queries with higher response score variance are prioritized during GRPO training. No ablation comparing variance-based prioritization to random ordering is provided, so the benefit of this curriculum strategy is unquantified.

ReST^EM training for revision models: While not an explicit ablation for Qwen2.5 (this is from the reference example paper), the Qwen2.5 report does not include any experiments with self-improvement loops (e.g., STaR, ReST^EM) applied to the revision or reasoning capabilities. Section 8 of the Qwen paper (Conclusion) mentions future work on "strategic scaling of inference compute resources" for reasoning enhancement, but no experiments are reported.

MoE architecture choices: The paper mentions fine-grained expert segmentation and shared expert routing as architectural innovations carried from Qwen1.5-MoE, but provides no ablation comparing MoE configurations or justifying the specific expert count, top-K routing, or shared expert ratio. The MoE models are treated as cost-effective alternatives to dense models rather than as subjects of architectural analysis.

Synthetic data generation and filtering: The dual-filter approach (general reward model + specialized Qwen2-Math-RM-72B) for synthetic data is described but not ablated. We cannot determine the marginal contribution of each filter or the false positive/negative rates at different filtering thresholds.

Data mixture rebalancing: The paper describes down-sampling overrepresented domains and up-sampling underrepresented ones, but provides no quantitative information about the original vs. rebalanced domain proportions, the number of domain categories, or the performance impact of rebalancing vs. a uniform sample from the raw distribution. This is a significant missing ablation given that data mixture is one of the paper's central claimed innovations.

Critical Assessment

Do the Benchmark Results Actually Demonstrate Improved Reasoning Capabilities, or Do They Reflect Data Contamination or Narrow Specialization?

The paper's headline claim is that Qwen2.5 achieves competitive or superior performance to much larger models through improved training data quality and methodology. The benchmark evidence for this claim is extensive and largely consistent across model sizes and task categories, but several interpretive cautions are necessary.

First, the MATH benchmark improvements are the most dramatic and central to the paper's narrative. Qwen2.5-72B base scores 62.1 vs. Qwen2-72B's 50.9; Qwen2.5-7B base scores 49.8 vs. Llama3-8B's 20.5; Qwen2.5-72B-Instruct scores 83.1 vs. Qwen2-72B-Instruct's 69.0. However, the paper explicitly states that it incorporated Qwen2.5-Math training data into pre-training (Section 3.1). MATH performance improvements may therefore reflect domain-specific training rather than a general improvement in reasoning capability that transfers to novel mathematical problems outside the training distribution. The paper's reliance on MATH (which has been publicly available for years) makes it impossible to distinguish between genuine mathematical reasoning improvement and effective memorization of problem patterns. The LiveCodeBench results (55.5 for 72B-Instruct vs. 32.2 for Qwen2-72B-Instruct, Table 6) partially address this concern because LiveCodeBench uses recent coding problems designed to be contamination-free, but no equivalent "contamination-free math" benchmark is used.

Second, the HumanEval regression on base models (Qwen2.5-72B: 59.1 vs. Qwen2-72B: 64.6, Table 2) is unexplained and potentially concerning. If the enhanced math/code pre-training data improved MATH but degraded HumanEval, it suggests a tradeoff in the data mixture that the paper does not acknowledge. The instruction-tuned model recovers HumanEval performance (86.6 vs. 86.0, Table 6), but this recovery likely comes from SFT coding data rather than pre-training—raising questions about whether the pre-training coding data mixture is actually better, or merely different.

Third, the paper provides no benchmark measuring out-of-distribution generalization. All evaluations use standard benchmarks whose formats, domains, and difficulty distributions are well-known in the community. A model trained on 18T tokens that include deliberately enriched math and code content may overperform on in-distribution benchmarks without corresponding improvements in genuinely novel reasoning contexts. The paper's own in-house benchmarks (Tables 11–12) partially address this by being proprietary, but they are not described in sufficient detail to assess their distributional properties.

Fourth, test data contamination is addressed only through n-gram matching (≥13 tokens with ≥60% overlap). This is a relatively permissive criterion—a problem that is semantically identical to a training example but paraphrased would not be detected. With 18T training tokens, the probability of near-duplicates that evade n-gram detection is non-trivial. The paper does not report what fraction of training data was removed by the n-gram filter or provide contamination analysis beyond the filtering description.

Is the Claim That Qwen2.5-72B-Instruct "Matches" Llama-3-405B-Instruct Supported?

The paper states that the flagship Qwen2.5-72B-Instruct "demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger." Looking at Table 6:

Benchmarks where Qwen2.5-72B-Instruct clearly leads: Arena-Hard (81.2 vs. 69.3, +11.9 points), MATH (83.1 vs. 73.8, +9.3 points), MBPP (88.2 vs. 84.5, +3.7 points), LiveCodeBench (55.5 vs. 41.6, +13.9 points), MT-Bench (9.35 vs. 9.08, +0.27)
Benchmarks where they are roughly comparable (within 2–3 points): MMLU-redux (86.8 vs. 86.2, +0.6), HumanEval (86.6 vs. 89.0, −2.4), MultiPL-E (75.1 vs. 73.5, +1.6), IFEval (84.1 vs. 86.0, −1.9)
Benchmarks where Llama-3.1-405B-Instruct clearly leads: MMLU-Pro (71.1 vs. 73.3, −2.2), GPQA (49.0 vs. 51.1, −2.1), LiveBench 0831 (52.3 vs. 53.2, −0.9)

The claim of "competitive performance" is well-supported: Qwen2.5-72B-Instruct leads on 6 of 13 benchmarks and is close on the remaining 7. However, the benchmarks favor different capability profiles—Qwen2.5's strengths are concentrated in math, coding, and instruction-following alignment, while Llama-3.1-405B's strengths are in general knowledge (MMLU-Pro, GPQA). The paper's narrative emphasizes the benchmarks where Qwen2.5 wins, but a user whose workload is primarily knowledge-intensive (rather than math/code-heavy) might prefer the larger model. The "5× smaller" framing is accurate but the performance equivalence is domain-dependent.

The in-house English evaluation (Table 11) provides a more nuanced picture: Qwen2.5-72B-Instruct leads Llama-3.1-405B-Instruct on Math (+12.64 points) and Coding (+2.27), but trails on IF (−0.68) and Knowledge (−1.01). The Chinese evaluation (Table 12) shows larger gaps favoring Qwen2.5: IF (+6.83), Knowledge (+12.07), Comprehension (+6.58), Math (+22.34), Reasoning (+7.14)—but this likely reflects Llama-3.1's primarily English training rather than Qwen2.5's general superiority.

Small Model Claims: Genuine Capability or Favorable Benchmark Selection?

The paper makes strong claims about small models: Qwen2.5-3B-Instruct achieves 65.9 MATH (Table 9), Qwen2.5-1.5B-Instruct achieves 55.2 MATH (Table 10), and Qwen2.5-0.5B base outperforms Gemma2-2.6B on several benchmarks (Table 5). These are genuinely impressive numbers that suggest the data quality and training methodology investments have disproportionately large effects at small scales.

However, the benchmark selection may favor Qwen2.5's training emphasis. The small-model comparisons heavily feature MATH and GSM8K—exactly the domains where Qwen2.5's pre-training data was enriched. On MMLU-Pro (which tests broader knowledge), Qwen2.5-3B-Instruct (43.7) trails Phi3.5-Mini (47.5) by 3.8 points (Table 9). On IFEval (instruction following), Qwen2.5-3B-Instruct (58.2) trails MiniCPM3-4B (68.4) by 10.2 points. The small models excel at math and coding but are less dominant at general knowledge and instruction following—exactly the pattern expected if math/code data enrichment during pre-training is the primary driver of improvements.

Additionally, the paper does not compare its small models against distilled versions of larger models (e.g., Llama-3.2-1B/3B, which were released around the same time). A distilled model might achieve competitive math performance while retaining stronger general knowledge, challenging the "capability density" narrative.

Missing Experiments That Would Strengthen the Paper

Controlled data mixture ablation. The paper attributes much of Qwen2.5's improvement to better data filtering, domain rebalancing, and math/code integration. A controlled experiment training models at a fixed scale (e.g., 7B) on (a) Qwen2's 7T-token dataset without modifications, (b) the same 7T tokens with Qwen2.5's filtering and rebalancing, and (c) the full 18T-token Qwen2.5 dataset would isolate the contributions of filtering, mixture, and scale. Without this, we cannot determine whether the gains come from more data, better data, or both.

Two-stage RL ablation. The paper presents the offline DPO + online GRPO decomposition as a key innovation, but provides no experiment comparing this to a single-stage approach with the same total RL compute budget. Training a Qwen2.5 variant with only DPO or only GRPO (matched for total training steps) would quantify the marginal contribution of each stage and the interaction effect. This is particularly important because the offline/online RL decomposition is one of the paper's few genuinely novel methodological claims.

Scaling law validation. The paper develops scaling laws from 44M–14B parameter experiments to predict hyperparameters for larger models, but does not validate these predictions. Training a 14B model with the scaling-law-predicted hyperparameters and with alternative hyperparameters, then comparing final loss, would provide evidence that the scaling laws generalize. Without this, the hyperparameter choices for 32B and 72B models rest on extrapolations with unknown error.

SFT data filtering threshold analysis. The "flawless by all scoring systems" retention criterion is extremely conservative. An experiment comparing models trained on SFT data at different filtering strictness levels (e.g., accepted by 1/3, 2/3, or 3/3 scoring systems) would reveal whether the aggressive filtering is necessary or whether a larger dataset with slightly lower quality would perform equivalently.

Inference-time compute scaling analysis. The paper mentions inference-time compute scaling (citing o1) in the introduction and conclusion as future work, but provides no experiments on how Qwen2.5 models perform with additional test-time compute (e.g., majority voting, best-of-N, chain-of-thought, or tree search). Given the paper's emphasis on reasoning capabilities, understanding whether Qwen2.5's MATH improvements are "one-shot" or can be further amplified by inference-time techniques would be valuable.

Long-context RL transfer validation. The paper claims that RL on short instructions transfers to long-context behavior for Turbo, but provides no ablation comparing Turbo with and without the RL stage evaluated on long-context tasks. The claim is plausible but unverified.

Where the Claims Hold Conditionally

"Competitive with Llama-3-405B-Instruct" holds most strongly on math, coding, and alignment benchmarks. On knowledge-intensive benchmarks (MMLU-Pro, GPQA), the gap favors the larger model, and on Chinese-language tasks, Qwen2.5's advantage reflects training language distribution rather than general capability superiority. The claim should be understood as "competitive or superior on reasoning-heavy and alignment benchmarks, competitive or slightly behind on knowledge benchmarks."

"4× context length expansion through YARN+DCA" holds for the specific RULER and LV-Eval benchmarks at 128K tokens. The ablation (Tables 16–17) shows that without YARN+DCA, long-context performance collapses, confirming these techniques are load-bearing. However, the claim applies to inference-time extension beyond the training context length—the models were trained to 32K (or 262K for Turbo), and YARN+DCA extends this 4× further. The extent to which this extension generalizes to tasks unlike RULER/LV-Eval is not tested.

"Data quality engineering as primary performance driver" is supported by the consistent generational improvements across all model sizes and benchmarks, but the specific contributions of filtering, mixture, synthetic data, and math/code integration are not disentangled. The claim is a reasonable interpretation of the aggregate results but not a causally verified hypothesis.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Is Prohibitively Expensive and Not Accounted for in Efficiency Claims

The entire difficulty-conditioned compute-optimal framework in the reference example paper rests on estimating prompt difficulty before allocating the inference budget. The Qwen2.5 paper takes a different approach—it makes no attempt at difficulty-conditioned inference allocation—but the evaluation methodology shares a structurally similar problem: the paper reports benchmark performance without accounting for the computational cost of achieving that performance, making direct efficiency comparisons impossible.

The assumption or constraint. The paper compares Qwen2.5 models against baselines purely on benchmark accuracy, with no normalization for training compute, inference cost, or model size. While the paper frequently references cost-effectiveness (e.g., Qwen2.5-Turbo has "significantly lower training and inference costs" than Qwen2.5-14B, Section 5.1), it never quantifies these costs. The paper does not report total pre-training FLOPs, SFT compute, RL training compute, or inference FLOPs per benchmark evaluation. The only cost-related figure is the TTFT speedup for long-context inference (Figure 3, reporting 3.2–4.3× speedups with sparse attention), but this is isolated to one model variant on one task type.

The consequence. A practitioner deciding whether to deploy Qwen2.5-72B-Instruct versus Llama-3.1-70B-Instruct cannot determine which model is more cost-effective for their use case. The paper shows Qwen2.5-72B-Instruct outperforms Llama-3.1-70B-Instruct on Arena-Hard (81.2 vs. 55.7, Table 6), but if the Qwen model required 3× more training compute or has 3× higher inference latency, the accuracy advantage might not justify the cost. Similarly, the paper emphasizes that Qwen2.5-7B base achieves 49.8 on MATH versus Llama3-8B's 20.5 (Table 4), but the Qwen model was trained on 18T tokens versus Llama3's 15T—a 20% data advantage that makes the comparison not purely about "capability density." Without FLOPs-matched comparisons, the paper's central narrative—that data quality engineering substitutes for parameter count—cannot be verified against the alternative hypothesis that Qwen2.5 simply invested more total compute (through larger datasets and more extensive post-training) to achieve its results.

The 18T token pre-training corpus itself represents a massive computational investment. Training a 72B model on 18T tokens requires approximately 6 × 72B × 18T ≈ 7.8 × 10^24 FLOPs for pre-training alone, not counting SFT, DPO, GRPO, or the scaling law experiments (44M to 14B models across 0.8B to 600B tokens each). If this total training budget exceeds what Llama-3-405B consumed (which trained on 15T tokens at 405B parameters: 6 × 405B × 15T ≈ 3.6 × 10^25 FLOPs), then the "5× smaller" framing is misleading—Qwen2.5-72B may have consumed comparable or greater training compute through data scaling.

What evidence exists in the paper. The paper provides no FLOPs accounting, no training cost estimates, and no inference latency measurements (except Figure 3 for long-context sparse attention). The evaluation tables (Tables 2–14) report only accuracy, with model size noted in the table captions but no compute normalization. The paper states that MoE models have "significantly lower training and inference costs" (Sections 5.1, 5.2) without quantifying these differences.

Mitigation status. Not addressed. The paper treats accuracy at a given parameter count as the relevant metric, implicitly assuming that parameter count is a reasonable proxy for total cost. This assumption is increasingly untenable as training data quantities diverge across model families (Qwen2.5: 18T tokens; Llama-3: 15T; prior Qwen2: 7T) and as post-training pipelines grow in complexity (Qwen2.5's SFT + DPO + GRPO vs. simpler RLHF approaches). The paper's future work section (Section 6) does not mention cost normalization as a priority.

Limitation 2: No Ablation Studies Isolate the Contribution of Individual Techniques, Making the Central Claim—That Coordinated Optimization Drives Gains—Untestable

The paper's core argument is that the combination of improved data filtering, domain rebalancing, math/code pre-training integration, synthetic data generation, staged pre-training, decomposable RL, and targeted SFT data construction produces outsized performance gains. However, the paper provides no controlled experiments that vary one component while holding others constant, making it impossible to determine which techniques actually matter and whether their combination is truly synergistic or merely additive.

The assumption or constraint. The paper treats the full training pipeline as an integrated system and evaluates only the final output—a series of models at different sizes produced by the complete pipeline. All comparisons are between Qwen2.5 and prior models (Qwen2, Qwen1.5) or competing models (Llama-3, Gemma2), which differ along multiple dimensions simultaneously (data quantity, data mixture, training recipe, architecture details). The paper does not train a single ablation model—e.g., a 7B model on Qwen2's 7T-token dataset but with Qwen2.5's improved filtering, or a 7B model on the full 18T tokens but with Qwen2's SFT recipe—that would isolate the contribution of any individual technique.

The consequence. The paper's conclusions about which techniques matter are speculative. Section 3.1 claims that "better data filtering" provides "more nuanced quality assessment" and "more effective filtering of low-quality samples," but without comparing Qwen2.5 models trained with and without Qwen2-Instruct filtering, we cannot know whether the filtering actually improves performance or whether the 2.6× increase in token count (7T → 18T) accounts for all observed gains. Similarly, the claim that math/code data integration during pre-training "proves highly effective" (Section 3.1) is supported only by end-to-end MATH benchmark improvements—but those improvements could equally be attributed to the SFT math data (Qwen2.5-Math chain-of-thought examples), the DPO math training pairs, or the general quality improvements from better filtering. The two-stage RL decomposition (offline DPO + online GRPO) is presented as a key innovation, but no experiment compares single-stage vs. two-stage RL, so we cannot determine whether the decomposition provides any benefit over simply running more online RL or more DPO.

This creates a fundamental interpretability problem: the paper demonstrates that Qwen2.5 is better than Qwen2, but cannot explain why in any causally validated sense. A practitioner seeking to replicate the approach on their own model family would not know which of the many described techniques to prioritize or whether the full complexity of the pipeline is necessary.

What evidence exists in the paper. None—there are zero ablation experiments. The paper provides no controlled comparisons varying individual components of the training pipeline.

Mitigation status. Not addressed. The paper does not acknowledge the absence of ablations as a limitation. The presentation implicitly treats the aggregate benchmark improvements as sufficient evidence for the effectiveness of the described techniques, but this conflates correlation (Qwen2.5 uses technique X and performs better) with causation (technique X causes the improvement).

Limitation 3: All Evaluations Are on In-Distribution Benchmarks with Contamination Protection That May Be Insufficient at 18T Token Scale

The paper evaluates Qwen2.5 exclusively on standard academic benchmarks (MMLU, MATH, HumanEval, etc.) and proprietary in-house datasets. At 18 trillion pre-training tokens—more than 200× the size of the English Wikipedia—the probability that benchmark test examples appear in the training data in some form is substantial, even with n-gram deduplication. The paper's contamination detection method may not adequately protect against memorization, and the absence of out-of-distribution evaluation makes it impossible to determine whether benchmark improvements reflect genuine capability gains or effective pattern matching on familiar problem types.

The assumption or constraint. The paper uses n-gram matching for contamination detection: a training sequence is removed if it shares a longest common subsequence with a test sequence that is both ≥13 tokens long and ≥60% of the minimum sequence length (Section 5). This approach catches near-verbatim copies but would miss semantically identical problems expressed with different wording, paraphrased solutions, or translated versions. At 18T tokens, even a 1-in-10^9 contamination rate would introduce ~18,000 contaminated examples—potentially covering entire benchmark test sets that typically contain hundreds to thousands of questions.

More fundamentally, the paper evaluates only on benchmarks whose formats and distributions are public and well-known. The in-house benchmarks (Tables 11–14) are proprietary but described only at the category level (Instruction Following, Knowledge, Comprehension, etc.) with no information about their construction, difficulty distribution, or whether they contain problems that are genuinely novel versus re-formulations of standard problem types. Without out-of-distribution evaluation—on benchmarks constructed after the training data cutoff, on tasks requiring novel problem-solving strategies, or on held-out problem sources not available during training—the paper cannot distinguish between generalization and memorization.

The consequence. The MATH benchmark improvements are particularly susceptible to contamination concerns. MATH (Hendrycks et al., 2021b) has been publicly available since 2021 and its 12,500 problems are widely used in training datasets. Qwen2.5's pre-training data includes "Qwen2.5-Math training data" (Section 3.1) and the SFT data includes "chain-of-thought data of Qwen2.5-Math" spanning "public datasets, K-12 problem collections, and synthetic problems" (Section 4.1). If any of the MATH test problems or near-variants appeared in these data sources, the 83.1 MATH score for Qwen2.5-72B-Instruct may reflect memorization of solution patterns rather than mathematical reasoning capability.

The HumanEval regression in base models (Qwen2.5-72B: 59.1 vs. Qwen2-72B: 64.6, Table 2) provides circumstantial evidence for domain-specific optimization: if the pre-training data mixture was genuinely "better" for code, we would expect improvements on all coding benchmarks, not a decline on HumanEval coupled with a large gain on MBPP (84.7 vs. 76.9). This pattern—regression on one benchmark, gain on another in the same domain—is consistent with training data composition changes that favor certain problem types over others, which is a form of dataset bias rather than general capability improvement.

What evidence exists in the paper. The contamination filtering description is brief (Section 5, one sentence) and provides no statistics on what fraction of training data was removed, what benchmarks triggered the most removals, or whether the filtering parameters (13 tokens, 60%) were validated. The paper does not report performance on any benchmark designed to be immune to contamination (e.g., LiveBench is used for instruction-tuned models in Table 6 but not for base models). LiveCodeBench (Tables 6–10) is the only benchmark explicitly described as "contamination-free," and it does show large Qwen2.5 gains (e.g., 72B-Instruct: 55.5 vs. Qwen2-72B-Instruct: 32.2 in Table 6), providing partial evidence for genuine improvement—but only in coding, not in math or general reasoning.

Mitigation status. Partially addressed through the n-gram filter and the use of LiveCodeBench for coding evaluation. The paper acknowledges the risk through its contamination filtering description but does not discuss the limitations of n-gram matching at the 18T token scale or the absence of out-of-distribution testing. Section 6 (Conclusion) does not mention contamination robustness as an area for improvement.

Limitation 4: The Reward Model Evaluation Reveals a Fundamental Disconnect Between Reward Benchmarks and Downstream RL Performance That the Paper Cannot Resolve

The paper's online RL stage depends critically on the quality of the reward model, and Section 5.2.3 provides an unusually candid acknowledgment: "current reward model evaluation benchmarks do not accurately predict the performance of the RL models trained under their guidance." This is not a minor caveat—it undermines the paper's ability to validate a central component of the post-training pipeline and raises questions about whether the online GRPO stage is reliably improving the model rather than exploiting reward model weaknesses.

The assumption or constraint. The paper trains a reward model (Qwen2.5-RM-72B) and evaluates it on Reward Bench, RMB, PPE, and an internal Chinese benchmark (Table 15). The model performs competitively but not dominantly—it leads on PPE (69.85 Objective-Avg) and Human-Preference-Chinese (61.27) but trails Llama-3.1-Nemotron-70B-Reward on Reward Bench (91.59 vs. 94.10) and Athene-RM-70B on RMB (68.71 vs. 73.98). The paper then states that "a higher score on RM benchmarks does not necessarily correlate with superior performance of the resulting RL model" and that "over-optimization on a specific benchmark may trigger Goodhart's law, resulting in degraded performance on other benchmarks and potentially impacting downstream alignment performance."

This creates an evaluation gap: the paper cannot validate whether Qwen2.5-RM-72B is actually better for RL training than alternative reward models, because the standard validation method (RM benchmarks) is acknowledged to be unreliable. The downstream model performance (Arena-Hard, MT-Bench, IFEval in Table 6) reflects the combined effect of base model quality, SFT data, DPO, and GRPO—it cannot isolate the contribution of the reward model specifically.

The consequence. The online RL stage is essentially unvalidated. The paper cannot determine whether GRPO with Qwen2.5-RM-72B improves the model more than GRPO with an off-the-shelf reward model would, or whether the RL stage introduces subtle degradations that are not captured by the evaluation benchmarks. The paper's own observation about Goodhart's law suggests that optimizing against a specific reward model during RL can cause the policy to exploit reward model idiosyncrasies—producing outputs that score highly under the reward model but are not actually better by human standards. Without a method for detecting such exploitation, the online RL stage could be reducing true output quality while appearing to improve it on proxy metrics.

The two-stage RL decomposition (offline DPO for verifiable capabilities, online GRPO for subjective qualities) assumes that the reward model is competent at evaluating the dimensions it's tasked with (truthfulness, helpfulness, conciseness, relevance, harmlessness, debiasing). If the reward model's evaluations on these dimensions are unreliable—and the paper provides evidence that RM benchmarks do not predict RL outcomes—then the entire decomposition's rationale collapses. Offline DPO would be doing the heavy lifting for alignment as well as reasoning, and online GRPO might be contributing noise or harm.

What evidence exists in the paper. Table 15 shows Qwen2.5-RM-72B's performance across multiple RM benchmarks, demonstrating that it is competitive but not state-of-the-art on any single benchmark. Section 5.2.3 explicitly discusses the disconnect between RM benchmarks and downstream RL performance. The instruction-tuned model results (Table 6) show large improvements on alignment benchmarks (Arena-Hard: 81.2; MT-Bench: 9.35) but cannot attribute these to the reward model specifically versus SFT, DPO, or base model improvements.

Mitigation status. Partially acknowledged but not resolved. The paper states that "this insight underscores the need for further research into more predictive evaluation methods for reward models" (Section 5.2.3), treating it as a field-wide problem rather than a Qwen2.5-specific limitation. The paper does not propose or implement any alternative reward model validation method. The use of multiple RM benchmarks is a partial mitigation—a model that performs well across diverse benchmarks is less likely to be overfit to any single one—but this does not address the core problem that RM benchmark performance may not correlate with downstream RL performance on any benchmark.

Limitation 5: Long-Context Evaluation Is Limited to Synthetic Benchmarks with Unknown Real-World Generalization

The paper demonstrates impressive long-context capabilities—Qwen2.5-Turbo achieves 100% passkey retrieval at 1M tokens (Figure 2) and 84.5 RULER at 128K (Table 16)—but the evaluation relies entirely on synthetic benchmarks (RULER, LV-Eval, LongBench-Chat, passkey retrieval) that may not represent realistic long-context use cases. Real-world long-context tasks (document summarization over hundreds of pages, multi-document synthesis, long-term dialogue, repository-level code understanding) impose demands—cross-document reasoning, temporal ordering, inconsistent information resolution—that are absent from synthetic benchmarks testing primarily retrieval and local attention.

The assumption or constraint. The long-context evaluation suite (Section 5.2.4) consists of RULER (synthetic tasks like variable tracking, keyword frequency, and question answering over long synthetic contexts), LV-Eval (keyword recall in long documents), LongBench-Chat (a small set of long-context QA tasks), and passkey retrieval (finding a hidden number in irrelevant filler text). These benchmarks primarily test the model's ability to attend to and retrieve information from specific positions in long sequences—they do not test complex reasoning over long contexts, multi-step inference spanning distant parts of a document, or the ability to synthesize information across multiple long documents.

The paper does not evaluate on realistic long-context tasks such as: (a) book-length summarization requiring thematic understanding across hundreds of pages, (b) multi-document legal or scientific analysis where relevant information is scattered across thousands of pages of text with cross-references, (c) long-term conversational agents that must maintain coherent personality and factual consistency over thousands of turns, or (d) repository-level software engineering tasks requiring understanding of code across hundreds of files with complex dependencies. The paper's own long-context training methodology—40% max-length sequences mixed with 60% shorter sequences, using back-translation for long-response SFT data (Section 4.1)—suggests that the model was trained primarily to handle long inputs and generate long outputs, not necessarily to perform complex reasoning over long contexts.

The consequence. A practitioner deploying Qwen2.5-Turbo for a 1M-token document analysis task may find that while the model can retrieve facts from arbitrary positions in the document (as demonstrated by passkey retrieval), it cannot integrate information across hundreds of pages to produce a coherent analysis or detect subtle contradictions between widely separated sections. The 100% passkey retrieval result demonstrates that the attention mechanism can physically reach all positions in a 1M-token sequence, but it does not demonstrate that the model can reason over that span. This is analogous to having perfect eyesight but limited working memory—you can see every word on every page, but you may not be able to hold enough information in working memory to synthesize a comprehensive understanding.

The YARN+DCA ablation (Tables 16–17) shows that without inference-time length extrapolation, long-context performance collapses (e.g., Qwen2.5-72B-Instruct LV-Eval at 256K drops from 45.2 to 2.4 without YARN+DCA). This means the paper's long-context capability depends on techniques that modify attention patterns to be sparser or interpolated at long range—techniques that might preserve retrieval accuracy while degrading the model's ability to perform attention patterns that require dense cross-context integration. The synthetic benchmarks are designed to be solvable with sparse attention (finding a keyword or a number), so they may not detect degradations in complex cross-context reasoning that real-world tasks require.

What evidence exists in the paper. Tables 16–17 report long-context benchmarks across multiple length scales; Figure 2 shows passkey retrieval at 1M tokens; Figure 3 shows inference speedups. None of these evaluate complex reasoning over long contexts. The paper does not report performance on any realistic long-context task.

Mitigation status. Not addressed. The paper treats performance on RULER, LV-Eval, and LongBench-Chat as sufficient evidence for long-context capability, without discussing the gap between synthetic benchmarks and real-world long-context reasoning. The conclusion (Section 6) does not mention this as a limitation or area for future work.

Limitation 6: The Paper Provides No Analysis of Failure Modes, Safety Properties, or Bias Beyond High-Level Alignment Benchmarks

The paper evaluates Qwen2.5 extensively on capability benchmarks (MMLU, MATH, HumanEval, etc.) and provides alignment metrics through Arena-Hard, MT-Bench, and IFEval (Tables 6–10). However, it provides no systematic analysis of model failures, safety risks, bias patterns, or behavioral boundaries. For a model family being released publicly in sizes from 0.5B to 72B with open-weight licenses (Apache 2.0 for most variants, Table 1), this is a significant omission—practitioners deploying these models in production need to understand not just average-case accuracy but worst-case behavior, safety properties, and demographic bias patterns.

The assumption or constraint. The paper's post-training methodology includes safety-relevant design choices: the reward model is trained with explicit harmlessness and debiasing criteria ("avoiding any content that could lead to illegal, immoral, or harmful behavior"; "producing responses that are free from bias, including but not limited to gender, race, nationality, and politics," Section 4.3), and the DPO training includes human review for training signal reliability (Section 4.2). However, the paper does not evaluate whether these design choices actually produce safe, unbiased behavior.

The consequence. The alignment benchmarks used (Arena-Hard, MT-Bench, IFEval) primarily measure instruction-following and helpfulness, not safety or bias. A model can score highly on these benchmarks while still producing harmful content when prompted adversarially, exhibiting demographic bias in its responses, or failing to refuse inappropriate requests. The Qwen2.5 reward model evaluation (Table 15) includes a Safety score on Reward Bench (92.71), but this measures the reward model's ability to rank response safety, not the instruction-tuned model's actual safety behavior. Without red-teaming results, bias audits, or toxicity evaluations, practitioners cannot make informed decisions about deployment risks.

The open-weight release amplifies this concern: models from 0.5B to 72B are available for download and fine-tuning, meaning downstream users can modify these models in ways that may undermine any safety properties the post-training pipeline attempted to instill. The paper provides no guidance on which safety properties are robust to fine-tuning, which failure modes are most likely at different model sizes, or how the safety profile varies across the model family. The small models (0.5B, 1.5B, 3B) are particularly concerning for deployment in resource-constrained edge devices where they may interact directly with users without safety filters—yet the paper provides no safety evaluation specific to these smaller variants.

What evidence exists in the paper. The paper lists harmlessness and debiasing as reward model training criteria (Section 4.3) and includes a Safety score in the RM evaluation (Table 15). It does not report any direct safety evaluation of the instruction-tuned models—no toxicity benchmarks (e.g., RealToxicityPrompts), no bias benchmarks (e.g., BBQ, WinoBias), no red-teaming results, and no analysis of refusal rates for harmful queries.

Mitigation status. Minimally addressed. The paper mentions safety in the context of reward model training criteria but does not evaluate safety outcomes. The open-weight release includes a license (Apache 2.0 for most models, "Qwen Research" for 3B, "Qwen" for 72B, Table 1) but no accompanying safety documentation, model cards with bias analysis, or deployment guidelines. This is a significant gap for models being positioned as infrastructure for downstream applications.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a single algorithmic breakthrough or architectural innovation. It is an engineering systems paper that demonstrates what becomes possible when data quality engineering and post-training methodology are elevated from operational hygiene to first-class research disciplines. The conceptual shift—if one takes the paper's results at face value—is that the open-weight community's focus on parameter scaling as the primary capability driver may be misallocated. Qwen2.5-7B base scores 49.8 on MATH while Llama3-8B scores 20.5, and Qwen2.5-72B-Instruct achieves 81.2 on Arena-Hard while Llama-3.1-405B-Instruct achieves 69.3—these are not marginal gains that could be dismissed as benchmark variance. They represent a fundamentally different point on the capability-per-parameter Pareto frontier, and they were achieved without architectural novelty.

The paper's most important reframing is the implicit argument that data quality and training methodology are multiplicative, not additive, factors. The 2.6× increase in pre-training tokens (7T → 18T) would be expected to yield modest improvements under standard scaling law assumptions—perhaps a few points on MMLU, not the 11+ point MATH gains observed. The paper's claim, supported by the deliberate integration of Qwen2.5-Math and Qwen2.5-Coder data into pre-training, is that which tokens you train on matters at least as much as how many tokens, and that this dimension has been systematically underexplored. If this claim generalizes—if other model families could achieve comparable efficiency gains through similar data engineering investments—then the field's implicit assumption that scaling parameter count is the most reliable path to capability improvement needs substantial revision.

The paper also contributes a diagnostic framework for post-training design, even if it does not fully validate it. The decomposition into offline DPO for verifiable capabilities (reasoning, factuality) and online GRPO for subjective qualities (style, safety, conciseness) is not experimentally isolated, but it articulates a testable hypothesis: that different capabilities have different "learnability surfaces" with respect to reward signals, and a single-phase RL approach conflates capabilities with incompatible training requirements. Whether or not Qwen2.5's specific instantiation of this idea is optimal, the framework provides a vocabulary and a research program for the field. Future post-training work should measure not just aggregate alignment scores but which capabilities improve under which training signals—a level of diagnostic granularity that prior RLHF papers rarely achieved.

The paper reconciles a latent tension in the open-weight community: the assumption that smaller models are inherently less capable in specialized domains. Qwen2.5-3B-Instruct scoring 65.9 on MATH—exceeding larger models like Phi3.5-Mini (3.6B, 48.5) and MiniCPM3-4B (4.0B, 46.6) by 17–19 points—demonstrates that model size is not destiny for reasoning capabilities. The mechanism appears to be data quality: when every parameter in a small model is trained on curated, high-value tokens rather than web-scale noise, the model extracts more capability per parameter. This has implications for the "emergent abilities" debate (Wei et al., 2022): if small models trained on equivalently curated data exhibit capabilities previously thought to require scale, then some "emergence" may reflect data quality differences between large and small model training pipelines rather than fundamental scaling properties.

The research directions that become more attractive after this work include: systematic comparison of data quality interventions (filtering, rebalancing, synthetic augmentation) at fixed compute budgets to establish "data quality scaling laws"; characterization of which capabilities benefit from pre-training vs. post-training data exposure (since the paper's largest gains came from moving math/code into pre-training); and development of more predictive reward model evaluation methods (given the paper's candid observation that RM benchmarks don't predict downstream RL performance).

The research directions that become less attractive include: architectural innovations pursued without commensurate investment in data engineering (the paper's gains came from data and training improvements on a standard Transformer architecture, not from novel attention mechanisms or expert configurations); and reward model optimization for benchmark scores (the paper explicitly warns that this triggers Goodhart's law without improving downstream RL outcomes).

Follow-Up Research This Work Enables

Data quality scaling laws with controlled ablations. The paper's central claim—that data quality engineering drives the observed performance gains—is supported only by end-to-end comparisons between Qwen2.5 and Qwen2, which differ along multiple dimensions simultaneously. A controlled experiment would train a fixed model architecture (e.g., 7B) on a fixed token budget (e.g., 5T tokens) while varying only the data curation pipeline: (a) raw web-scale data with minimal filtering, (b) Qwen2-Instruct filtered data only, (c) filtered + domain-rebalanced data, (d) filtered + rebalanced + math/code enriched data. Measuring the performance delta at each step would isolate the marginal contribution of each data quality intervention and establish whether the techniques compose additively or synergistically. This experiment is newly tractable because Qwen2.5 provides a detailed recipe for each data curation step (Section 3.1) that can be independently varied. A strong result would show that domain rebalancing alone accounts for the majority of MATH improvement, or alternatively that filtering and rebalancing interact non-additively.

Controlled ablation of the two-stage RL decomposition. The paper presents offline DPO + online GRPO as a key post-training innovation but provides zero experimental evidence that the decomposition matters. A direct test would train three variants of a Qwen2.5 model from the same SFT checkpoint: (a) offline DPO only, (b) online GRPO only, and (c) DPO followed by GRPO (the paper's approach), all matched for total RL training steps. If the two-stage model outperforms both single-stage variants on reasoning benchmarks (suggesting DPO preserves reasoning while GRPO adds alignment) while matching or exceeding the GRPO-only variant on alignment benchmarks, the decomposition hypothesis is supported. If DPO-only matches or exceeds the two-stage model on all benchmarks, the GRPO stage is unnecessary. If GRPO-only matches the two-stage model, the DPO stage is unnecessary. This experiment is critical because the paper's theoretical framework for post-training depends entirely on this decomposition, yet it is the least empirically validated component of the pipeline.

Out-of-distribution evaluation of mathematical reasoning. The paper's MATH benchmark improvements are the most dramatic but also the most susceptible to contamination concerns at 18T token scale. A rigorous test would evaluate Qwen2.5 models on a mathematics benchmark constructed after the training data cutoff—ideally one with problems that require the same underlying reasoning skills as MATH but with novel problem structures, notation, or domain framing that cannot be solved by pattern matching against training examples. The recently introduced FrontierMath benchmark, or a held-out set of competition math problems from 2024, would serve this purpose. If Qwen2.5 maintains its MATH-level performance on such a benchmark (e.g., Qwen2.5-72B-Instruct scoring ~80+ on genuinely novel problems), the paper's claims of improved mathematical reasoning are validated. If performance drops substantially (e.g., to levels comparable with models that did not emphasize math in pre-training), the MATH gains may primarily reflect effective memorization of publicly available problem patterns rather than genuine reasoning improvement. This experiment would also test the paper's claim that integrating math data into pre-training produces transferable reasoning skills rather than narrow specialization.

Long-context reasoning evaluation beyond retrieval benchmarks. The paper demonstrates 100% passkey retrieval at 1M tokens and strong RULER/LV-Eval scores, but these benchmarks primarily test whether the model can attend to information at arbitrary positions—not whether it can reason over information distributed across the full context span. A meaningful stress test would evaluate Qwen2.5-Turbo on a task that requires synthesizing information from multiple distant segments of a 500K+ token document to answer questions that cannot be answered from any single segment. For example: given a 500K-token legal document with contract terms scattered across hundreds of pages, can the model identify conflicts between clauses in sections 50,000 tokens apart? Given a 300K-token scientific review article, can it identify methodological inconsistencies between studies described in different chapters? These tasks require not just attention to distant positions but working memory and multi-hop reasoning over those positions—capabilities that YARN+DCA's sparse attention patterns may degrade even while preserving retrieval accuracy. If performance on such tasks is substantially below what retrieval benchmarks predict, it would reveal a capability gap that the current long-context evaluation methodology systematically masks.

Small-model capability ceiling as a function of data quality investment. The paper's small model results are among its most striking: Qwen2.5-0.5B base outperforms Gemma2-2.6B on several benchmarks despite 5× fewer parameters. This suggests that small models may benefit disproportionately from data quality investments, but the paper does not explore the shape of this relationship. A systematic study would train models at multiple sizes (0.5B, 1.5B, 3B, 7B) on datasets of identical composition but varying quality (e.g., unfiltered web data, filtered data, filtered + rebalanced data, filtered + rebalanced + synthetic data) and measure the performance improvement at each size. If the improvement is proportionally larger at small sizes (as the paper's results hint), it would establish that data quality is not just a capability amplifier but a capability equalizer—reducing the gap between small and large models. Conversely, if the improvement is proportional or smaller at small sizes (suggesting small models saturate earlier regardless of data quality), the paper's small-model results may reflect peculiarities of the specific benchmarks where they excel (MATH, GSM8K) rather than a general principle.

Reward model predictive validity for downstream RL. The paper's most methodologically candid moment is its observation that "current reward model evaluation benchmarks do not accurately predict the performance of the RL models trained under their guidance" (Section 5.2.3). This insight—that RM benchmark scores and downstream RL model quality can diverge—has major implications for the entire RLHF literature, but the paper does not empirically characterize the divergence. A follow-up study would train multiple reward models (varying architecture, training data, and training recipe) that achieve different scores on standard RM benchmarks, use each to train an RL policy from the same SFT checkpoint, and measure the correlation (or lack thereof) between RM benchmark ranking and downstream policy quality. If the correlation is weak or negative—if the highest-scoring RM on Reward Bench produces the worst downstream policy—then the field's current approach to reward model development is broken in a specific and measurable way. The Qwen team is uniquely positioned to conduct this study given their access to the Qwen2.5 training infrastructure and their stated recognition of the problem.

Practical Applications and Downstream Use Cases

On-device mathematical assistants using small Qwen2.5 variants. Qwen2.5-3B-Instruct achieves 65.9 on MATH and 86.7 on GSM8K (Table 9)—performance that exceeds most 7B-class models from the previous generation on mathematical reasoning. With only 2.8B non-embedding parameters, this model can run on consumer-grade hardware (phones, laptops) without quantization, enabling fully local mathematical tutoring, homework assistance, or STEM problem-solving that does not require network connectivity or API calls. The practical benefit is clear: a student in a low-connectivity environment can receive step-by-step mathematical reasoning guidance from a model running entirely on-device, with the Qwen2.5-3B-Instruct providing not just answer retrieval but chain-of-thought reasoning (trained via Qwen2.5-Math SFT data, Section 4.1). The 0.5B and 1.5B variants, while less capable, could handle simpler mathematical tasks (arithmetic, basic algebra) with even lower hardware requirements. The open-weight availability under Apache 2.0 license for most variants (Table 1) removes licensing barriers to commercial deployment.

Cost-effective coding assistants using mid-size models. Qwen2.5-7B-Instruct achieves 84.8 on HumanEval and 28.7 on LiveCodeBench (Table 8)—competitive with much larger models—while Qwen2.5-14B-Instruct reaches 83.5 HumanEval and 42.6 LiveCodeBench (Table 7), approaching GPT-4o-mini levels (88.4 HumanEval, 40.7 LiveCodeBench). For organizations deploying coding assistants at scale, the 7B or 14B models offer a compelling cost-performance tradeoff: they can run on a single GPU (the 7B model requires ~14 GB at FP16, the 14B ~28 GB), enabling private deployment of coding assistance without sending proprietary code to external APIs. The multi-language code generation capability (MultiPL-E scores of 70.4 for 7B and 72.8 for 14B, spanning Python, C++, Java, and other languages) makes these models suitable for polyglot development environments. The LiveCodeBench scores—which use contamination-free recent coding problems—provide some assurance that the coding capability is not purely memorized from training data.

Multilingual deployment in non-English-speaking markets. Qwen2.5 substantially improves multilingual performance over Qwen2, with the 72B base model achieving 89.6 on Multi-Understanding (vs. Qwen2-72B's 80.7) and 78.7 on Multi-Exam (Table 2), and the instruction-tuned models showing strong performance on language-specific knowledge benchmarks (JMMLU: 80.56, KMMLU: 61.96, AMMLU: 72.44 for 72B-Instruct, Table 13). For organizations deploying LLM-powered applications in markets where English-only models perform poorly—particularly East Asian and Middle Eastern languages—Qwen2.5 provides a strong off-the-shelf option. The cross-lingual transfer methodology (Section 4.1, using translation models to convert high-resource language instructions into low-resource languages) means the instruction-following capabilities are not limited to English and Chinese. A customer service chatbot deployed in Arabic or Japanese can leverage the same instruction-tuning quality that the English model achieves, without requiring separate fine-tuning pipelines for each language.

Long-document processing for legal and academic applications. Qwen2.5-Turbo's 1M-token context window with 100% passkey retrieval accuracy (Figure 2) and 3.2–4.3× TTFT speedup via sparse attention (Figure 3) makes it practically feasible to process entire legal contracts, academic theses, or regulatory filings in a single inference pass. The key operational benefit is the elimination of chunking and aggregation pipelines: a 200,000-word legal document (~260K tokens) can be processed atomically, avoiding the information loss and inconsistency that arises when documents are split into overlapping chunks for separate processing. The maintained short-context performance (Turbo scores 97.5 on RULER at 4K, Table 16, and is competitive with the 14B model on standard benchmarks, Table 7) means the same deployment can handle both quick factual queries and multi-hundred-page document analysis without switching models. The limitation—that complex cross-document reasoning may not match retrieval accuracy—should be tested before deployment in high-stakes settings.

When to Prefer This Method

The paper does not articulate a clear decision framework comparing Qwen2.5 against specific named alternatives with explicit tradeoff conditions. It positions Qwen2.5 as a general-purpose model family spanning sizes from 0.5B to 72B, with each size competing against the best available model at that scale, rather than proposing a method that is preferable under specific conditions and inferior under others. The closest the paper comes to a conditional recommendation is the comparison between MoE variants (Turbo, Plus) and dense models: the paper states that MoE models offer "superior cost-effectiveness" and "a great balance among accuracy, latency, and cost" (Section 1), suggesting that when inference cost or latency is the binding constraint, the MoE models may be preferable to comparably capable dense models. However, the paper does not quantify these tradeoffs with specific latency or cost numbers, making the recommendation directional rather than operational. No decision matrix is provided, and constructing one would require inferring tradeoffs the paper does not explicitly articulate.