Qwen2 Technical Report

🎯 Pitch

Qwen2-72B rivals proprietary giants like GPT-4o on key benchmarks, yet even a 0.5B-parameter version outperforms much larger open-weight models—showing that data quality trumps sheer model size.

1. Executive Summary

This technical report introduces the Qwen2 series, a suite of foundational and instruction-tuned language models spanning 0.5B to 72B parameters in both dense and Mixture-of-Experts architectures, and analyzes their performance against prior open-weight and proprietary models across a wide range of benchmarks including MMLU, HumanEval, GSM8K, and multilingual evaluations. The series incorporates targeted training data enhancements—notably increased high-quality code and mathematics data alongside long-context training via Dual Chunk Attention with YARN—and post-training alignment through supervised fine-tuning and Direct Preference Optimization (e.g., applying DPO to offline preference pairs followed by online reward-model-guided refinement). The flagship Qwen2-72B base model achieves 84.2 on MMLU, 64.6 on HumanEval, and 89.5 on GSM8K, while Qwen2-72B-Instruct reaches 9.1 on MT-Bench and 48.1 on Arena-Hard, with the smaller models demonstrating that data scaling remains effective even in sub-billion-parameter regimes, establishing that enhanced pre-training data quality and quantity yields competitive performance gains across model scales, though near-zero improvement persists for the hardest reasoning tasks regardless of scale.

2. Context and Motivation

The Core Problem: Systematic Comparison and Open Release of LLMs at Scale

This paper addresses a fundamental gap in the large language model landscape: the lack of comprehensive, well-documented open-weight model families that span multiple scales and architectural paradigms while providing transparent evaluation against both prior open-source models and proprietary systems. Specifically, the Qwen2 series fills three concurrent needs: (1) a rigorously evaluated suite of models from 0.5B to 72B parameters that researchers can study, fine-tune, and deploy without API restrictions; (2) a Mixture-of-Experts variant that demonstrates how sparse architectures can achieve dense-model performance with fewer activated parameters; and (3) a detailed technical report that documents training data composition, architectural decisions, and post-training methodology—information that proprietary model developers typically withhold.

This problem matters for several practical and theoretical reasons the paper implicitly addresses throughout Sections 1–4:

Democratization of model access. The release of ChatGPT (OpenAI, 2022) and subsequent proprietary models like GPT-4o (OpenAI, 2024) and Claude-3 Opus (Anthropic, 2024) established new performance ceilings, but these systems remain accessible only through paid APIs with usage restrictions. Open-weight alternatives—the Llama series (Touvron et al., 2023), Mistral (Jiang et al., 2023a), Gemma (Mesnard et al., 2024), and the original Qwen (Bai et al., 2023a)—have progressively narrowed this gap, but each prior release left specific capability gaps. Llama-3 (AI@Meta, 2024) achieved GPT-4-level recognition but offered only two model sizes (8B and 70B) and limited multilingual coverage. Mistral released strong 7B models but no larger variants until Mixtral. Gemma provided small models (2B and 7B) but lacked the larger scales needed for maximum performance. The Qwen2 series explicitly targets these gaps by releasing four dense scales plus one MoE variant, covering the full range from on-device deployment to datacenter-scale inference.

The training data bottleneck. A recurring theme in the LLM literature is that model architecture—while important—has been secondary to data quality and quantity in driving recent performance gains. The paper positions itself within this understanding by devoting substantial attention to data composition (Section 3.1). The Qwen1.5 series was trained on 3 trillion tokens. Qwen2 expands this to 7 trillion tokens with specific enhancements: more high-quality code data, more mathematics content, and broader multilingual coverage spanning approximately 30 languages. This is not merely "more data"—it is a targeted data mix designed to strengthen reasoning capabilities that the authors identify as the primary differentiator among contemporary models. The paper explicitly states:

"Compared to previous editions of Qwen, Qwen2 includes a broader spectrum of linguistic data, enhancing the quantity and quality of code and mathematics content. This enrichment is hypothesized to improve reasoning abilities of LLMs."

This hypothesis—that code and mathematics data serve as proxies for reasoning training—is increasingly prevalent in the field but requires empirical validation at scale, which this paper provides.

The efficiency imperative: matching performance with fewer parameters. The paper introduces Qwen2-57B-A14B, a Mixture-of-Experts model that activates only 14 billion parameters per token while maintaining the performance of a ~30B dense model. This addresses a practical deployment problem: the largest dense models (70B+) require substantial GPU memory and are expensive to serve, while smaller dense models sacrifice capability. The MoE approach—using fine-grained experts with shared and routed components—promises to break this tradeoff. But prior MoE models (e.g., Mixtral-8x7B) activated roughly 12B parameters and underperformed similarly-sized dense alternatives on several benchmarks. The paper's contribution is demonstrating that with careful expert initialization, routing design, and sufficient pre-training (4.5T additional tokens beyond the base 7T), an MoE model can genuinely match dense models with ~2× more activated parameters—a practical efficiency breakthrough that the tables in Section 5.1.1 quantify.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior open-weight models that Qwen2 is designed to address:

Limited scale diversity. Llama-3 released only 8B and 70B variants (AI@Meta, 2024). Gemma offered 2B and 7B (Mesnard et al., 2024). Qwen1.5 provided a broader range (0.5B through 110B) but the smaller models showed significant performance degradation on reasoning tasks compared to larger scales. The Qwen2 series explicitly targets this by showing that even the 0.5B and 1.5B models benefit substantially from the improved data mix—Qwen2-0.5B achieves 45.4 on MMLU (Table 5) with only 0.3B non-embedding parameters, and Qwen2-1.5B reaches 56.5, significantly outperforming Qwen1.5-1.8B's 46.8. This demonstrates that data quality improvements propagate to the smallest scales, making on-device deployment more viable.

Weak multilingual capabilities in English-centric models. Despite claims of multilingual support, many open-weight models show sharp performance cliffs when evaluated on non-English benchmarks. The Llama series, for instance, was primarily trained on English data. Mixtral-8x22B scores only 53.4 on Chinese CMMLU and 54.6 on C-Eval (Table 2). The paper addresses this gap directly by including multilingual evaluations across four categories (Exam, Understanding, Mathematics, Translation) in ~30 languages, and by conducting human-evaluated multilingual assessments (Table 13) that position Qwen2-72B-Instruct as competitive with GPT-4-Turbo across Arabic, French, Spanish, Japanese, Korean, and others—averaging 3.93/5.0 versus GPT-4-Turbo's 3.98.

Insufficient long-context capabilities. Prior to Qwen2, many open-weight models were limited to 4K–8K context windows at release, requiring community-developed extensions (RoPE scaling, YaRN) to stretch further—often with performance degradation. The paper addresses this through native long-context training (Section 3.2): extending context length from 4,096 to 32,768 tokens during pre-training, modifying RoPE base frequency from 10,000 to 1,000,000, and incorporating Dual Chunk Attention (DCA) with YaRN to push effective context to 131,072 tokens. The Needle in a Haystack tests (Figure 1) and the NeedleBench/LV-Eval results (Table 12) validate that this approach maintains retrieval accuracy across the full 128K range for the larger models, rather than degrading at extended lengths.

Post-training methodology opacity. Most technical reports provide limited detail on how instruction-tuned variants are produced—what data was used, how it was annotated, how RLHF was applied. This paper provides unusually detailed documentation of the post-training pipeline (Section 4), including:

Collaborative data annotation using InsTag for automatic ontology extraction from instruction corpora, instruction selection based on tag diversity and complexity, self-evolution to increase instruction complexity, and human ranking of multiple responses to create preference pairs.
Automated data synthesis through rejection sampling (for math: generate multiple reasoning paths, keep those yielding correct answers), execution feedback (for code: generate solutions and test cases, evaluate by compilation and execution), data repurposing (aggregate high-quality literary works and Wikipedia character profiles, then use LLMs to generate corresponding instructions), and constitutional feedback (define safety/value principles, generate aligned and deviating responses for preference data).
Two-stage RLHF: offline DPO on pre-compiled preference data followed by online iterative DPO with reward model feedback, using an Online Merging Optimizer to mitigate alignment tax.

This level of detail enables the community to replicate and build upon the post-training approach, addressing a common criticism of prior model releases.

Unclear data contamination boundaries. Many model evaluations are potentially inflated by test-set leakage into training data, but few reports provide rigorous contamination analysis. The paper implements a two-stage decontamination approach (Section 5.2.6): first excluding training sequences with 13-gram overlap AND ≥60% longest common subsequence (LCS) matching to any test sequence during data construction, then post-hoc identifying remaining contamination using strict 13-gram overlap (without LCS constraint). The results (Table 15) show that performance differences between original and strictly decontaminated test sets are small—typically <1% for knowledge benchmarks and <2% for coding benchmarks—with the exception of MATH where Qwen2-72B-Instruct improves by 5.6 percentage points on the non-contaminated subset (suggesting that the contaminated samples were actually harder than average, a finding consistent with spurious matches on formulaic mathematical expressions). This transparency is rare and valuable for interpreting reported scores.

How This Paper Positions Itself

The paper positions Qwen2 as a direct successor to Qwen1.5 that achieves its gains primarily through data improvements rather than architectural novelty. The architectural changes are incremental and well-motivated:

Grouped Query Attention (GQA) replaces conventional multi-head attention (MHA) to reduce KV cache memory during inference—a practical optimization, not a theoretical breakthrough. The specific KV head configurations (Table 1) show aggressive grouping: the 72B model uses 64 query heads but only 8 KV heads, yielding an 8:1 ratio that significantly reduces memory footprint.
Dual Chunk Attention (DCA) with YaRN for long contexts extends existing techniques rather than introducing new ones.
SwiGLU activations, RoPE, RMSNorm, and pre-normalization are all adopted from prior work and explicitly credited.
The MoE architecture uses fine-grained experts (Dai et al., 2024) and shared+routed expert routing (Rajbhandari et al., 2022), both established approaches, with novel elements in the expert initialization procedure (shuffling parameters along the intermediate dimension and randomly reinitializing 50% of each expert's parameters to promote diversity).

This is a deliberate positioning choice: the paper is not claiming to have invented fundamentally new architectures. Instead, it argues that careful engineering of the data pipeline, methodical scaling across model sizes, and transparent evaluation constitute the primary contribution. The abstract frames this explicitly: "Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks." The contribution is the comprehensive suite and its demonstrated performance, not any single technique.

The paper also positions itself within the broader trend toward open-weight models achieving parity with proprietary systems. The benchmark tables consistently compare against both open-weight baselines (Llama-3, Mixtral, Gemma, Yi-1.5) and proprietary models (GPT-4, GPT-4o, Claude-3 Opus), with the results showing that the gap has narrowed substantially. On MMLU, Qwen2-72B (84.2) is competitive with GPT-4's reported ~86.4. On HumanEval, Qwen2-72B (64.6) approaches proprietary model territory. On MT-Bench, Qwen2-72B-Instruct (9.12) surpasses GPT-4's score of 8.99 as reported in the original MT-Bench paper (Zheng et al., 2023). The paper's existence as a detailed technical report—complete with training data descriptions, hyperparameter tables, and contamination analysis—is itself a statement about the value of transparency in AI development.

Finally, the paper positions its multi-scale release strategy as enabling a different kind of research and deployment than single-scale releases. The 0.5B and 1.5B models are explicitly described as:

"designed for easy deployment on portable devices such as smartphones, earphones, and smart glasses"

This is not merely about achieving the highest possible benchmark scores—it is about covering the full deployment spectrum from edge to cloud, which requires careful data scaling to ensure that even the smallest models benefit from the improved training corpus. The results in Table 5 showing Qwen2-0.5B (0.3B non-embedding parameters) matching or exceeding Gemma-2B on several benchmarks validate this strategy.

Relationship to the Executive Summary's Findings

The Context and Motivation section provides the why behind the benchmark results described in the Executive Summary. Where the Summary states that Qwen2-72B achieves 84.2 on MMLU, this section explains that this represents a 4.7-point improvement over Llama-3-70B (79.5) and a 6.7-point improvement over Qwen1.5-72B (77.5)—gains that the paper attributes primarily to the enhanced 7-trillion-token pre-training dataset with increased code and mathematics content. Where the Summary notes the 0.5B model's competitive performance, this section contextualizes that as evidence for the data-scaling-is-effective-even-at-sub-billion-parameter-regimes thesis. The Summary's observation about "near-zero improvement for the hardest reasoning tasks regardless of scale" finds its motivation here: the paper explicitly focuses on data quality and coverage improvements, not on novel reasoning architectures, and thus the hardest problems—those requiring capabilities fundamentally absent from even the enhanced training distribution—remain unsolved, a limitation the paper does not attempt to address.

3. Technical Approach

3.1 Reader Orientation

The Qwen2 series is a family of five Transformer-based language models—four dense (0.5B, 1.5B, 7B, 72B parameters) and one Mixture-of-Experts (57B total, 14B activated)—that are pre-trained on 7 trillion tokens of curated multilingual data and then post-trained through supervised fine-tuning and Direct Preference Optimization to follow instructions. What is being built is not a single model but a complete deployment ecosystem spanning from smartphone-scale (0.5B) to datacenter-scale (72B) inference, with the core technical challenge being how to make every scale benefit from the same underlying data quality improvements. The shape of the solution is a two-phase pipeline: first, a pre-training phase that focuses on data composition, quality filtering, and long-context adaptation rather than architectural novelty; second, a post-training phase that uses collaborative human annotation and automated synthesis to produce instruction-following variants, with the key insight being that targeted data enrichment (code, mathematics, multilingual content) drives performance gains more effectively than architectural changes alone.

3.2 Big-Picture Architecture (Diagram in Words)

The Qwen2 system has four major processing stages:

Tokenizer — A byte-level byte-pair encoding (BPE) tokenizer with 151,643 regular tokens and 3 control tokens, inherited from the original Qwen (Bai et al., 2023a). It converts raw text into token IDs and exhibits better compression rates than alternatives for multilingual text, which directly reduces the number of tokens the model must process.
Dense Model Core — A stack of Transformer layers using Grouped Query Attention (GQA) with causal masking, SwiGLU activations, Rotary Positional Embeddings (RoPE), RMSNorm pre-normalization, and QKV bias. The number of layers and hidden dimensions vary by model scale (24 layers/896 dim for 0.5B up to 80 layers/8192 dim for 72B). This component produces next-token probability distributions from input sequences.
Mixture-of-Experts Extension — Applied specifically to the 57B-A14B model, this replaces dense feed-forward networks with MoE layers containing 64 fine-grained routed experts plus 8 shared experts, with a gating network selecting 8 experts per token. It enables the model to have 57B total parameters while only activating 14B per forward pass.
Post-Training Pipeline — A sequence of Supervised Fine-Tuning (SFT) on 500K+ demonstrations followed by two-stage Direct Preference Optimization (offline DPO on pre-compiled preference pairs, then online DPO with reward model feedback), producing instruction-tuned variants from the base models.

Information flows: raw text → tokenizer → token IDs → embedding layer → stacked Transformer/MoE layers → logits → next-token prediction (during pre-training) OR response generation (during inference). For instruction-tuned models, an additional SFT step interposes instruction-response pairs, and DPO adjusts the model's output distribution to prefer human-approved responses over rejected ones.

3.3 Roadmap for the Deep Dive

First, the tokenizer, because it is the entry point for all text and its vocabulary size and compression characteristics affect everything downstream—training efficiency, multilingual support, and maximum context utilization.
Second, the dense Transformer architecture in detail, including GQA, DCA, RoPE adjustments, and the full model configuration table—this is the computational engine shared across all five model scales.
Third, the MoE architecture and its three design innovations (fine-grained experts, shared+routed routing, and diversity-promoting initialization), since it is a structural departure from the dense models and involves the most architectural complexity.
Fourth, the pre-training data pipeline—quality filtering, data expansion targets, distribution optimization, and the 7T vs. 12T token decision—because the paper's central claim is that data improvements drive the performance gains.
Fifth, long-context training (RoPE frequency adjustment, YARN, DCA), as it is a distinct phase at the end of pre-training with its own technical decisions.
Sixth, the post-training methodology (SFT and DPO) and the data construction techniques that feed it (ontology extraction, rejection sampling, execution feedback, constitutional AI), since instruction-tuning transforms base models into usable products.
Seventh, the architectural configuration table and its implications for memory, inference cost, and deployment across scales—connecting the design choices to practical constraints.

3.4 Detailed, Sentence-Based Technical Breakdown

This is fundamentally a systems and engineering paper whose core idea is that systematic improvements to data quality, quantity, and composition—combined with incremental architecture optimizations and transparent post-training methodology—produce a model family that matches or exceeds proprietary systems across diverse benchmarks. There are no novel training algorithms or theoretical breakthroughs. Instead, the contribution is in the careful integration of known techniques applied at scale with rigorous evaluation.

Tokenizer: Byte-Level BPE with Multilingual Compression

The tokenizer is the first component that processes any input text, and its design directly affects training efficiency (fewer tokens means shorter sequences for the same semantic content) and multilingual capability (a tokenizer biased toward English will fragment non-English text into many subword pieces, degrading performance).

The paper follows the identical tokenizer from the original Qwen (Bai et al., 2023a). It is a byte-level byte-pair encoding (BPE) tokenizer. Byte-level BPE means the base vocabulary consists of individual bytes (256 possible values), and merges are learned on top of these to form subword units. This has the advantage that every possible Unicode character can be represented without <unk> tokens, which is essential for multilingual text that may contain rare characters.

Vocabulary composition:

The vocabulary contains 151,643 regular tokens plus 3 control tokens, for a total of 151,646 entries. Control tokens are special markers not representing text content—typically used for things like beginning-of-sequence, end-of-sequence, or padding indicators. The paper notes that "owing to considerations in distributed training, the effective size for the embeddings is larger" than the vocabulary size, which is a common practice: padding the embedding matrix to a multiple of the GPU's tensor core tile size (often 128 or 256) improves hardware utilization during the embedding lookup operation.

Compression rate advantage:

The paper claims this tokenizer "exhibits high encoding efficiency, as evidenced by its better compression rate relative to alternatives, facilitating the multilingual capabilities of Qwen2." Compression rate in this context means the ratio of raw characters (or bytes) to tokens—a higher compression rate means fewer tokens to represent the same text. This matters because Transformer computational cost scales quadratically with sequence length in the attention mechanism, so a 10% better compression rate translates to roughly 19% fewer attention operations for the same content. Multilingual text tends to have lower compression rates under English-optimized tokenizers because subword merges learned on predominantly English data don't match character sequences in other languages. The Qwen2 tokenizer was designed with multilingual data from the start, so merges reflect patterns across many languages rather than just English.

Token embedding tying:

For the two smallest models (0.5B and 1.5B), embedding tying is enabled (Table 1), meaning the input embedding matrix and the output projection matrix share the same weights. For the larger models (7B, 57B-A14B, 72B), embedding tying is disabled. This is a parameter-efficiency decision: in small models, the embedding matrix represents a significant fraction of total parameters (with a vocabulary of 151,646 and hidden dimensions of 896 or 1536, the embedding layer alone accounts for ~136M or ~233M parameters), and tying saves memory with minimal performance impact. In larger models with hidden dimensions of 3584 or 8192, the embedding matrix is a proportionally smaller fraction of total parameters, and untying gives the model more flexibility at minimal relative cost.

Why this design over alternatives?

The paper doesn't train a new tokenizer for Qwen2 but reuses the Qwen tokenizer. This is a pragmatic choice: changing the tokenizer would require retraining the embedding layer from scratch, would make the model incompatible with Qwen1.5 checkpoints (preventing weight inheritance and comparison), and would invalidate the tokenizer-specific optimizations in the data processing pipeline. The design choice prioritizes continuity and comparability over potential tokenizer improvements.

Dense Transformer Architecture: GQA, DCA, and Standardized Components

All Qwen2 dense models share the same architectural template—a decoder-only Transformer with causal self-attention—differentiated only by the depth, width, and attention head counts specified in Table 1. The architecture uses several modern Transformer techniques, each addressing a specific limitation of the original Vaswani et al. (2017) design.

Grouped Query Attention (GQA):

Conventional multi-head attention (MHA) maintains separate key (K) and value (V) projections for each attention head. At inference time, every token's K and V vectors must be cached (the "KV cache") to avoid recomputing them for autoregressive generation. With H query heads, the KV cache stores 2 × H × hidden_dim numbers per token per layer.

GQA (Ainslie et al., 2023) reduces this by sharing K and V projections across groups of query heads. Specifically:

Qwen2-0.5B: 14 query heads, 2 KV heads → 7 query heads share each KV head (7:1 ratio)
Qwen2-1.5B: 12 query heads, 2 KV heads → 6:1 ratio
Qwen2-7B: 28 query heads, 4 KV heads → 7:1 ratio
Qwen2-72B: 64 query heads, 8 KV heads → 8:1 ratio
Qwen2-57B-A14B: 28 query heads, 4 KV heads → 7:1 ratio

The KV cache memory per token per layer is reduced from 2 × H × head_dim to 2 × num_KV_heads × head_dim. For Qwen2-72B, this is an 8× reduction in KV cache size compared to full MHA—from 64 K heads + 64 V heads to 8 K heads + 8 V heads. At a head dimension of 128, each token per layer requires 2 × 64 × 128 = 16,384 values under MHA versus 2 × 8 × 128 = 2,048 values under GQA. Over 80 layers, this accumulates to a substantial memory reduction, enabling larger batch sizes or longer sequences during inference.

Why GQA over alternatives?

Multi-Query Attention (MQA) pushes this further by using a single KV head for all query heads. GQA is a middle ground: it provides most of the memory savings of MQA while retaining enough K/V diversity to avoid the quality degradation that MQA sometimes exhibits. The specific KV head counts (2, 2, 4, 8) were chosen empirically based on "detailed KV head configurations for various model sizes" (Section 2.2.1), though the paper does not provide ablation experiments on this parameter.

Dual Chunk Attention (DCA) with YARN:

DCA (An et al., 2024) is the mechanism that enables Qwen2 to handle sequences beyond its training-time context window. The core idea: when a sequence is short enough to fit within one "chunk" (a manageable subsequence length), DCA produces mathematically identical output to standard causal attention. When the sequence exceeds one chunk, DCA splits it into chunks and computes attention in a two-level scheme:

Intra-chunk attention: Standard causal attention within each chunk, capturing local dependencies.
Inter-chunk attention: A mechanism to propagate information between chunks, "facilitating effective capture of relative positional information between tokens within and across chunks" (Section 2.2.1).

This is a training-free extension method: the model was trained on 32,768-token sequences, but with DCA it can process sequences up to 131,072 tokens (4× the training length) while "maintaining high performance, as evidenced by minimal perplexity degradation in preliminary experiments" (Section 3.2).

YARN (Peng et al., 2023) is applied alongside DCA "to rescale the attention weights for better length extrapolation." YARN (Yet another RoPE extensioN method) modifies how Rotary Positional Embeddings behave at positions beyond the training range. Specifically, RoPE encodes position $m$ as a rotation by angle $m\theta$ where $\theta$ is a base frequency. At positions beyond training, these rotations enter regions the model has never seen, causing degradation. YARN applies a scaling factor that compresses the effective position indices for longer sequences, keeping them within the model's familiar range while preserving the local attention pattern that RoPE is designed to capture.

Rotary Positional Embeddings (RoPE):

RoPE (Su et al., 2024) encodes positional information by rotating the query and key vectors in the attention computation. For a token at position $m$ , the query or key vector $\mathbf{x} \in \mathbb{R}^d$ is treated as $d/2$ two-dimensional vectors and rotated by angles $m\theta_i$ where $\theta_i$ are preset frequencies. The attention score between positions $m$ and $n$ depends only on their relative distance $m-n$ through the rotation property:

$\langle \text{RoPE}(\mathbf{q}_m, m), \text{RoPE}(\mathbf{k}_n, n) \rangle = f(\mathbf{q}_m, \mathbf{k}_n, m-n)$

where $\mathbf{q}_m$ and $\mathbf{k}_n$ are the query and key vectors, $m$ and $n$ are positional indices, and $f$ is a function that depends only on the relative position $m-n$ .

What this computes: the standard dot-product attention score but with the query and key vectors rotated proportionally to their position indices. The rotation angle increases with position, so the model learns to attend based on relative distance rather than absolute position—key for length generalization.

Why this form: absolute positional encodings (learned or sinusoidal added to embeddings) make it difficult for the model to handle sequences longer than training because positions beyond training have never been seen. RoPE's relative encoding property means the model only needs to have seen a range of relative distances during training, not all absolute positions, enabling length extrapolation (with YARN's help for positions far beyond training).

RoPE base frequency adjustment for long contexts:

Section 3.2 notes that during long-context training, the base frequency of RoPE was modified from the standard 10,000 to 1,000,000 (following Xiong et al., 2023). The base frequency determines the rotation rate: higher base frequency means position encodings change more slowly with position, making the model less sensitive to very large position differences and better able to handle extended sequences without the positional attention pattern breaking down.

QKV Bias:

The paper mentions "QKV bias (Su, 2023) for attention." Standard attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

where $Q$ , $K$ , and $V$ are linear projections of the input. Adding bias terms to these projections ( $Q = xW_Q + b_Q$ , $K = xW_K + b_K$ , $V = xW_V + b_V$ ) gives the model learnable offsets. Su (2023) demonstrated that adding bias to the Q, K, and V projections before applying RoPE improves length extrapolation, possibly because the biases allow the model to center its representations before rotation.

SwiGLU Activation:

The feed-forward networks use SwiGLU (Dauphin et al., 2017) instead of the original Transformer's ReLU. SwiGLU is a gated activation:

$\text{SwiGLU}(x) = (xW_1 \odot \text{Swish}(xW_2))W_3$

where $W_1$ , $W_2$ , and $W_3$ are learned weight matrices, $\text{Swish}(z) = z \cdot \sigma(z)$ is the Swish activation (with $\sigma$ being the sigmoid function $1/(1+e^{-z})$ ), and $\odot$ is element-wise multiplication.

What it computes: a gated linear unit where one linear projection provides the "signal" ( $xW_1$ ) and another provides a learned "gate" that selectively passes the signal ( $\text{Swish}(xW_2)$ ). The gated result is then projected back to the model dimension.

Why this form: SwiGLU consistently outperforms ReLU and GELU in language modeling benchmarks. The gating mechanism allows the network to learn which dimensions to amplify or suppress based on the input, providing more expressive per-position computation. The Swish nonlinearity on the gate is smoother than a sigmoid or hard gate, which improves gradient flow during training.

RMSNorm and Pre-Normalization:

RMSNorm (Jiang et al., 2023b) normalizes a vector by its root-mean-square statistic rather than by its mean and variance (as in LayerNorm). For a vector $\mathbf{x}$ of dimension $d$ :

$\text{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}} \odot \gamma$

where $\gamma$ is a learned scale parameter and $\epsilon$ is a small constant for numerical stability.

What it computes: each hidden vector is divided by its RMS value (the square root of the mean of squared elements), then scaled element-wise by learned parameters. Unlike LayerNorm, there is no mean subtraction, only variance normalization.

Why this form: RMSNorm is computationally cheaper than LayerNorm (no mean computation, no subtract-then-add pattern) while performing similarly in practice. Pre-normalization (applying normalization before the attention and FFN sublayers, rather than after as in the original Transformer) is now standard practice because it improves training stability—gradients flow more cleanly through the residual connections when the normalization is at the input rather than the output of each sublayer.

Mixture-of-Experts Architecture: Fine-Grained Experts with Shared Routing

Qwen2-57B-A14B is the sole MoE model in the series. Its architecture mirrors Qwen1.5-MoE-A2.7B (Qwen Team, 2024c) but upscaled from Qwen2-7B as the base dense model. The MoE mechanism replaces every feed-forward network with a gated collection of multiple FFNs ("experts"), each specializing in different types of computation.

Core mechanism:

For each token, the MoE layer computes:

$p = \text{softmax}(G(x))$

$y = \sum_{i \in \text{topk}(p)} p_i E_i(x)$

where:

$x \in \mathbb{R}^{d_{\text{model}}}$ is the input hidden state for a single token,
$G(\cdot)$ is the gating network (a learned linear transformation that projects from $d_{\text{model}}$ to $n$ dimensions, where $n$ is the number of routed experts),
$\text{softmax}$ converts the gate outputs into a probability distribution over experts,
$\text{topk}(p)$ selects the indices of the $k$ largest probabilities (with $k = 8$ for Qwen2-57B-A14B),
$E_i(\cdot)$ is the feed-forward computation for expert $i$ ,
$p_i$ is the softmax probability for expert $i$ (used as a weight in the summation),
$y$ is the output, a weighted sum of the top- $k$ expert outputs.

What it computes: a sparse weighted combination of expert computations. Only $k$ out of $n$ experts process each token—this is the key efficiency property. The softmax gating learns to route different types of tokens to different experts, allowing specialization.

Why this form over dense FFNs: a dense FFN applies the same computation (with the same parameters) to every token. An MoE layer can learn different computations for different token types—for example, some experts might specialize in mathematical notation, others in code syntax, others in natural language grammar. The sparsity (only 8 of 64 routed experts active per token) means the total parameter count can be much larger than a dense model (57B vs. ~30B for comparable-quality dense) while the per-token computation remains manageable (14B activated parameters). The top-k selection with $\text{softmax}$ weights is a standard MoE gating mechanism that provides a differentiable approximation to hard expert selection.

Expert granularity (fine-grained experts):

A key design decision is the granularity of experts. The standard approach (used by Mixtral-8x7B, Jiang et al., 2024) is to make each expert the same size as a full FFN from the corresponding dense model. For a 7B-scale model, this might mean 8 experts each with the same intermediate dimension as the dense FFN. Mixtral activates 2 of 8 such experts per token.

Qwen2 uses fine-grained experts (Dai et al., 2024): the 64 routed experts each have an intermediate size of only 2,560 (Table 1), much smaller than the dense FFN's 18,944 for the Qwen2-7B base. The model activates 8 of these 64 experts per token. Given the same total expert parameters (64 × 2,560 = 163,840) and the same number of activated parameters (8 × 2,560 = 20,480), fine-grained experts offer more combinatorial flexibility: with 8 experts chosen from 64, there are $\binom{64}{8}$ possible expert combinations per token versus $\binom{8}{2}$ for the coarse-grained approach. The paper claims this "facilitates more diverse and dynamic expert utilization, thereby enhancing overall performance and adaptability."

The shared experts (8 of them, with 2,560 intermediate size each) are always active for every token, providing a common computational foundation that all tokens receive regardless of routing decisions. This decouples general-purpose computation (shared) from specialized computation (routed).

Expert initialization (upcycling with diversity):

The MoE model is not trained from scratch. Instead, it uses upcycling (Komatsuzaki et al., 2023): starting from a pretrained dense checkpoint (Qwen2-7B) and converting its FFN layers into MoE layers. The initialization procedure has several steps designed to promote expert diversity:

Replication: The dense FFN is replicated $\lceil n \times h_E / h_{\text{FFN}} \rceil$ times, where $n=64$ is the number of desired experts, $h_E=2560$ is the expert intermediate size, and $h_{\text{FFN}} = 18944$ is the original dense FFN intermediate size. For Qwen2-7B: $\lceil 64 \times 2560 / 18944 \rceil = \lceil 8.65 \rceil = 9$ copies of the dense FFN.
Parameter shuffling within each copy: "Parameters are shuffled along the intermediate dimension. This guarantees that each fine-grained expert exhibits unique characteristics, even across different FFN copies." Shuffling means permuting the rows of the weight matrices along the intermediate (hidden) dimension, so each copy has the same parameters but in a different arrangement—different subsets of the original dense FFN's neurons map to different experts.
Expert extraction: From the shuffled copies, individual experts of size $h_E = 2560$ are extracted by slicing along the intermediate dimension, and the remaining dimensions are discarded. If the copies don't exactly fill the required number of experts, the excess is discarded.
Random reinitialization: For each fine-grained expert, 50% of its parameters are randomly reinitialized. "This process introduces additional stochasticity into expert initialization, potentially enhancing the model's capacity for exploration during training." The reinitialization breaks the deterministic relationship between experts that would exist if they were merely sliced and shuffled copies of the same weights—the experts start with some shared foundation (the 50% retained weights) but have room to diversify (the 50% random weights).

Why this initialization over alternatives? Training an MoE from scratch is expensive and may result in poor expert specialization early in training. Upcycling from a dense checkpoint provides a strong initialization because the dense model has already learned useful representations. The shuffling and partial reinitialization address the main risk of naive upcycling: that all experts would start too similar and fail to specialize. The 50% reinitialization fraction is a hyperparameter choice—too little and experts remain similar; too much and the benefit of the dense pretraining is lost. The paper does not provide ablations on this fraction.

Shared vs. routed experts:

The inclusion of shared experts (always active) alongside routed experts (selectively activated) follows a trend in recent MoE designs (Rajbhandari et al., 2022; Dai et al., 2024). Shared experts handle computation that is useful for all tokens—basic syntactic processing, common-sense knowledge—while routed experts specialize. This prevents the router from having to "waste" expert capacity on common patterns that every token needs, and ensures that even tokens routed to suboptimal experts still get a solid baseline computation from the shared experts.

Model Configuration Table: The Five Scale Points

Table 1 provides the complete architectural specification. The key patterns across scales:

Scaling laws in architecture:

Hidden size roughly doubles with each major scale jump (896 → 1,536 → 3,584 → 8,192 and 3,584 for the MoE). The MoE uses the same hidden dimension as the 7B model since it is upcycled from it.
Number of layers increases with scale but sub-linearly (24 → 28 → 28 → 80). The jump from 28 to 80 layers for the 72B model is the most aggressive scaling dimension—the model becomes much deeper rather than much wider, likely because very wide models encounter memory bandwidth bottlenecks for individual matrix multiplications.
Head size is fixed at 128 for all models ≥1.5B; only the 0.5B uses 64. This is a standard value that balances head expressivity (smaller heads capture finer-grained patterns) with computational efficiency (larger heads amortize the attention softmax overhead).
Intermediate size (FFN hidden dimension) is consistently larger than the model hidden size, following the standard practice of having the FFN be wider than the attention projection to provide more per-position computation capacity.

Parameter counts and training tokens:

Dense models from 1.5B upward were trained on 7 trillion tokens. Qwen2-0.5B, counter-intuitively, was trained on 12 trillion tokens—more than the larger models. The paper explains: "An attempt to further relax the quality threshold resulted in a 12 trillion token dataset. However, the model trained on this dataset did not show a significant performance improvement over the 7 trillion token model." The 0.5B was trained on this larger dataset, and its competitive performance (45.4 MMLU) demonstrates that data scaling helps even the smallest models.
The MoE model received 4.5 trillion additional tokens of pre-training beyond the 7T base, "in line with the principle of upcycling." This additional training allows the expert router to learn meaningful specialization and the individual experts to diversify from their shuffled, partially-randomized initialization.

KV cache implications:

The "Embedding Tying" row shows that only the two smallest models share input/output embeddings. The "Vocabulary Size" is consistently 151,646 across all models. The paper emphasizes that "Qwen2 models demonstrate a substantially lower Key-Value (KV) size per token relative to Qwen1.5 models" due to GQA, translating into "a reduced memory footprint, particularly advantageous in long-context inference tasks."

For Qwen2-72B specifically: 80 layers × 8 KV heads × 128 head dimension × 2 (K+V) = 163,840 values per token in the KV cache (at 16-bit precision, ~328 KB per token). For a 128K-token context, the KV cache alone requires ~42 GB, which is significant but substantially less than the ~336 GB that full MHA (64 KV heads) would require at this scale.

Pre-Training Data: Quality, Quantity, and Distribution

The pre-training data pipeline is the primary differentiator between Qwen2 and its predecessor Qwen1.5. The paper describes three axes of improvement:

Quality Enhancement:

The paper applies "additional heuristic and model-based methods, including the use of the Qwen models to filter out low-quality data." This means a Qwen model (likely a previously trained version) is used as a classifier or scorer to assess data quality—for example, flagging documents with poor grammar, incoherent content, or boilerplate text. Additionally, Qwen models "are utilized to synthesize high-quality pre-training data," meaning the models generate training examples in domains where natural data is scarce or low-quality.

Data Expansion:

The pre-training data volume increased from 3 trillion tokens (Qwen1.5) to 7 trillion tokens (Qwen2 dense models). The expansion is specifically targeted: "a significantly larger volume of high-quality code, mathematics, and multilingual data." The paper supports approximately 30 languages explicitly: English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, Vietnamese, and others.

The code and mathematics data expansion is non-trivial because these domains have different characteristics from natural language. Code has strict syntax constraints, long-range dependencies (function definitions used much later), and requires exact reproduction of API calls and variable names. Mathematics requires multi-step logical chains, specialized notation, and abstract reasoning patterns that don't appear in general text. By increasing the proportion of these data types, the model sees more examples of structured reasoning during pre-training.

Distribution Improvement:

The paper mentions experiments on "scaled-down models to optimize the mixing of data from various sources and domains." This means training small proxy models (perhaps at the 0.1B–0.5B scale) on different data mixtures and evaluating which mixture produces the best performance, then applying that optimal mixture at full scale. This is a common technique to avoid expensive full-scale data mixture sweeps.

The 12T vs. 7T decision:

"We attempted to further relax the quality threshold resulting in a 12 trillion token dataset. However, the model trained on this dataset did not show a significant performance improvement over the 7 trillion token model." This is a crucial empirical finding: beyond a certain point, adding more data—if that data is of lower average quality—does not help and may hurt. The paper hypothesizes that "increasing the volume of data does not necessarily benefit model pre-training." Instead, they opted for the higher-quality 7T dataset for all models except the 0.5B. The 0.5B was trained on 12T tokens, perhaps as a further test of the "data scaling helps at small sizes" hypothesis, and its strong performance relative to other sub-billion models supports this.

Multi-task instruction data during pre-training:

"Similar to previous Qwen models, high-quality multi-task instruction data is integrated into the Qwen2 pre-training process to enhance in-context learning and instruction-following abilities." This means that alongside the next-token prediction objective on raw text, the model also sees instruction-response pairs during pre-training. This blurs the boundary between pre-training and fine-tuning and has become increasingly common (e.g., Llama-3 also incorporates instruction data). The benefit: the base model develops some instruction-following capability even before SFT, making the post-training phase more effective.

Long-Context Training: RoPE Re-tuning and Length Extension

Section 3.2 describes a targeted intervention at the end of pre-training to extend the context window.

Baseline context length:

The model was initially trained with a context length of 4,096 tokens. This is a standard and relatively conservative starting point, consistent with Qwen1.5 and many prior models.

Extension to 32,768 tokens:

During "the concluding phase of pre-training," the context length is increased to 32,768 tokens. This is accompanied by "the introduction of a significantly increased volume of high-quality, lengthy data." The model is not simply asked to extrapolate—it is trained on actual long sequences, so the attention patterns at long range are learned rather than inferred.

RoPE base frequency modification:

The base frequency of RoPE is changed from 10,000 to 1,000,000 (following Xiong et al., 2023). Recall that RoPE encodes position $m$ as rotation by $m \cdot \theta_i$ where $\theta_i = 10000^{-2i/d}$ in the original formulation (with base 10,000). Changing the base to 1,000,000 means the rotation angles grow much more slowly with position. At position 32,768 with base 10,000, the highest-frequency component completes many cycles (potentially causing aliasing and pattern confusion). With base 1,000,000, the same position sees much smaller rotation angles, staying within a range the model can process cleanly.

YARN integration:

YARN (Peng et al., 2023) is applied during inference to push the effective context further to 131,072 tokens—exactly 4× the 32,768 training length. YARN works by scaling the RoPE frequencies differently for different frequency bands: low frequencies (which control long-range attention) are scaled less aggressively than high frequencies (which control local attention), preserving the model's local attention patterns while enabling long-range attention to generalize.

DCA integration:

Dual Chunk Attention (An et al., 2024) provides the computational mechanism for handling sequences that exceed the attention window. Rather than computing full $O(n^2)$ attention over 128K tokens, DCA splits the sequence into chunks (likely of size close to 32K, the training length) and computes attention hierarchically. This keeps computation manageable while still allowing information to flow across chunk boundaries.

Validation:

The paper states these strategies "enable the model to process sequences of up to 131,072 tokens while maintaining high performance, as evidenced by minimal perplexity degradation in preliminary experiments." Perplexity degradation refers to the increase in next-token prediction loss when moving from within-training-range sequences to extended-range sequences. "Minimal" means the loss curve stays smooth as sequence length increases, rather than showing a sharp inflection point at 32K that would indicate the model is confused by positions it has not seen during training.

Post-Training Data Construction: Collaborative Annotation and Automated Synthesis

Section 4.1 provides unusually detailed documentation of how the instruction-tuning data was created. The data serves two downstream processes: Demonstration data $D = {(x_i, y_i)}$ (instruction $x_i$ , satisfactory response $y_i$ ) for SFT, and Preference data $P = {(x_i, y_i^+, y_i^-)}$ (instruction $x_i$ , preferred response $y_i^+$ , dispreferred response $y_i^-$ ) for DPO.

Collaborative Data Annotation (Section 4.1.1):

This is the human-in-the-loop component, consisting of four stages:

Automatic Ontology Extraction: InsTag (Lu et al., 2024c), an "open-set fine-grained tagger," processes a large-scale instruction dataset and extracts an ontology—a structured taxonomy of instruction types (e.g., "summarization," "code generation with constraints," "multilingual translation with cultural adaptation"). The ontology is manually refined by human annotators to ensure accuracy.
Instruction Selection: Each instruction is evaluated for "tag diversity, semantic richness, complexity, and intent completeness." Instructions that score high on these dimensions are selected as representative examples for annotation. This is a filtering step that ensures the human annotation budget is spent on high-quality, diverse instructions rather than repetitive or trivial ones.
Instruction Evolution: "A self-evolution strategy (Zhao et al., 2024) is employed, prompting the Qwen models to add constraints or requirements to existing instructions, thereby increasing their complexity." For example, a base instruction "Write a Python function to sort a list" might evolve to "Write a Python function to sort a list of dictionaries by a nested key, handling missing keys gracefully, with type hints and docstrings, in O(n log n) time." This creates a range of difficulty levels from the same seed instructions.
Human Annotation: Multiple responses to each instruction are generated using "diverse generation strategies and Qwen models of different scales." Human annotators rank the responses and ensure the best response meets quality criteria. This yields both the $y_i$ demonstrations and the $(y_i^+, y_i^-)$ preference pairs—the highest-ranked response becomes $y_i^+$ and the lowest-ranked becomes $y_i^-$ .

Automated Data Synthesis (Section 4.1.2):

For domains where human annotation is too expensive, slow, or requires specialized expertise, the paper describes four automated strategies:

Rejection Sampling (for mathematics and similar tasks):
- LLMs generate multiple reasoning paths for each instruction.
- Paths that yield the correct final answer (as determined by a verifier or exact match) are kept as demonstration data.
- Correct paths are paired with incorrect paths to create preference data.
- The "reasonable by the model" check means the model self-evaluates whether the reasoning path is logically sound, not just whether the final answer matches—a path could reach the right answer through flawed reasoning, and such paths are filtered out.
Execution Feedback (for coding and instruction following):
- For coding: LLMs generate solutions AND associated test cases. Solutions are compiled and executed against test cases; those passing become demonstrations; passing vs. failing solutions form preference pairs.
- For instruction following: The LLM generates a Python verification function for each instruction with constraints (e.g., length limits, format requirements). The verifier checks whether the response meets the constraints. This is a self-contained automated evaluation loop that doesn't require human judgment.
Data Repurposing (for literary writing and roleplay):
- High-quality literary works are aggregated from the public domain.
- LLMs generate instructions of varying detail that could have produced those works.
- The instruction-work pairs become demonstration data.
- For roleplay: Character profiles are sourced from knowledge repositories like Wikipedia. LLMs generate instructions and responses based on these profiles, similar to a reading comprehension task where the profile must be accurately reflected in the response. This approach leverages existing high-quality text (literary works, encyclopedia entries) that humans have already created, rather than trying to synthesize high-quality creative text from scratch.
Constitutional Feedback (for safety and values):
- A "constitution dataset" is compiled—a set of principles specifying desirable and undesirable behaviors (e.g., "do not generate hate speech," "refuse to provide instructions for illegal activities," "be respectful and helpful").
- LLMs are instructed to generate responses that either align with or deviate from these principles.
- Aligned vs. deviating responses become preference pairs.
- This follows the Constitutional AI approach (Bai et al., 2022) where the model learns to internalize safety guidelines through preference optimization rather than through explicit rules.

Supervised Fine-Tuning (SFT): Training Details

Section 4.2 describes the SFT phase where the pre-trained base model is fine-tuned on instruction-response pairs.

Dataset: More than 500,000 examples covering "instruction following, coding, mathematics, logical reasoning, role-playing, multilingualism, and safety."

Training configuration:

Sequence length: 32,768 tokens—the full extended context window, so the model learns to follow instructions that may involve long documents or multi-turn conversations.
Epochs: 2—a small number to avoid overfitting to the instruction distribution at the cost of general knowledge.
Learning rate schedule: Decreased from $7 \times 10^{-6}$ to $7 \times 10^{-7}$ (a factor of 10 reduction). The paper does not specify the decay schedule (linear, cosine, etc.).
Weight decay: 0.1—moderate regularization to prevent overfitting.
Gradient clipping: Maximum value of 1.0—prevents individual gradient updates from being too large, which is important when the instruction dataset may contain outliers (very long sequences, unusual formatting).

Reinforcement Learning from Human Feedback: Two-Stage DPO with Online Merging

Section 4.3 describes the alignment phase that follows SFT. The method is Direct Preference Optimization (DPO, Rafailov et al., 2023) rather than the more complex PPO-based RLHF.

Why DPO over PPO-based RLHF?

DPO reframes preference learning as a direct policy optimization problem, eliminating the need for a separate reward model. Given a preference pair $(y^+, y^-)$ , DPO optimizes:

$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y^+, y^-)}\left[\log \sigma\left(\beta \cdot \left[\log\frac{\pi_\theta(y^+|x)}{\pi_{\text{ref}}(y^+|x)} - \log\frac{\pi_\theta(y^-|x)}{\pi_{\text{ref}}(y^-|x)}\right]\right)\right]$

where:

$\pi_\theta$ is the policy being optimized (the instruction-tuned model),
$\pi_{\text{ref}}$ is a reference policy (typically the SFT model, frozen),
$\beta$ is a temperature parameter controlling how strongly preferences are enforced,
$\sigma$ is the sigmoid function,
The expectation is over the preference dataset $P$ .

What it computes: for each preference pair, DPO increases the log-probability ratio of the preferred response relative to the reference model, while decreasing it for the dispreferred response. The $\beta \cdot \log$[ratio] term is analogous to an implicit reward: if the model assigns higher probability to $y^+$ than the reference model does (relative to $y^-$ ), the loss decreases.

Why this form: it avoids training and maintaining a separate reward model, which would itself require careful calibration and could introduce reward hacking. By optimizing the policy directly against preference data, DPO achieves similar alignment effects with simpler implementation and fewer hyperparameters.

Two-stage training:

Offline DPO: Uses the pre-compiled preference dataset $P$ constructed in Section 4.1. The model sees all preference pairs and learns to distinguish preferred from dispreferred responses. This is "offline" because the preference data is fixed and not updated during training.
Online DPO: The model generates multiple responses to instructions from its current policy $\pi_\theta$ . A reward model (separately trained, despite DPO's standard reward-free formulation) "selects the most and the least preferred responses, forming preference pairs that are used for DPO in each episode." This is "online" because the preference pairs come from the model's own current output distribution, making the training on-policy. The advantage: as the model improves, it generates better responses that become the new "dispreferred" baselines, creating a self-improving curriculum. The disadvantage: it requires a reward model and introduces the complexity of online sampling.

Online Merging Optimizer:

The paper mentions using "Online Merging Optimizer (Lu et al., 2024a) to mitigate the alignment tax," where alignment tax refers to the performance degradation on standard benchmarks that often accompanies RLHF fine-tuning (the model becomes better at following instructions but worse at factual recall or reasoning). The Online Merging Optimizer periodically merges the current policy weights with the reference policy weights, keeping the model from drifting too far from its pre-trained knowledge distribution while still learning alignment. The paper provides no details on the merging frequency, ratio, or methodology, citing Lu et al. (2024a).

Contamination Control: Two-Stage Decontamination

Section 5.2.6 describes how the training data was cleaned to prevent test set leakage, and how remaining contamination was assessed.

During data construction (removal phase):

The procedure uses two overlapping criteria:

N-gram matching: Potentially contaminated sequences are identified, but the paper notes this leads to "a high false negative rate, because there could be commonly used expressions, especially in mathematical and coding data." False negatives mean the n-gram approach misses true contamination because common code snippets or mathematical expressions appear verbatim in both training and test sets without actually conferring an unfair advantage.
Longest Common Subsequence (LCS) constraint: For a training sequence $s_t$ and a test sequence $s_e$ , the sequence is removed if:
- $|\text{LCS}(s_t, s_e)| \geq 13$ (the longest common subsequence has at least 13 elements), AND
- $|\text{LCS}(s_t, s_e)| \geq 0.6 \times \min(|s_t|, |s_e|)$ (the LCS covers at least 60% of the shorter sequence).

Before computing LCS, all symbols and punctuation are removed from both sequences "to focus on the substantive content rather than formatting similarities."

What this computes: the LCS is the longest sequence of tokens that appears in both sequences in the same order (but not necessarily contiguously). The 13-element minimum prevents flagging very short common subsequences (which are inevitable for common phrases). The 60% relative length threshold prevents flagging sequences that share a small fraction of their content—a math problem and solution might share variable names throughout, but if the overlap is only 30%, it's likely coincidental rather than contamination.

Post-hoc contamination analysis (assessment phase):

After training, a stricter criterion is applied to identify any remaining contamination: any test sample with a 13-gram overlap with pre-training or post-training data (without the LCS constraint) is flagged as potentially contaminated. This is stricter because it catches exact text matches even if they're a small fraction of the total sequence. The results (Table 15) show that many flagged samples are "false positives, primarily stemming from the mathematics and coding datasets" where "certain code snippets and mathematical equations are so common that they do not provide any meaningful advantage."

The key validation: performance on the original test set vs. the non-contaminated subset (with all flagged samples removed) shows small differences, mostly within ±2 percentage points. The notable exception is MATH where Qwen2-72B-Instruct scores 69.0 on the original but 74.6 on the non-contaminated subset, suggesting that the flagged MATH samples were actually harder than average (contamination is negatively correlated with performance, which is the opposite of what you'd expect if contamination artificially inflated scores).

4. Key Insights and Innovations

Innovation 1: Reframing LLM Progress as Data Engineering Rather Than Architectural Novelty

The most intellectually distinctive contribution of this paper is not any single technical mechanism but rather the implicit reframing of what constitutes meaningful progress in LLM development. The Qwen2 series achieves its substantial performance gains over Qwen1.5—often 10–17 percentage points on benchmarks like GSM8K and MATH (Table 2)—while making only incremental, well-credited architectural changes (GQA from Ainslie et al., 2023; DCA from An et al., 2024; YARN from Peng et al., 2023; SwiGLU from Dauphin et al., 2017). There is no novel attention mechanism, no new training objective, no theoretical breakthrough in optimization.

Instead, the paper makes its case through the engineering of the data pipeline: expanding the pre-training corpus from 3T to 7T tokens with targeted increases in code, mathematics, and multilingual content; using Qwen models themselves as quality filters and data synthesizers; running scaled-down proxy experiments to optimize data mixture ratios; and carefully documenting the post-training data construction process including ontology extraction, instruction evolution, rejection sampling, execution feedback, and constitutional AI-based preference generation. The paper's central empirical finding—that the 12T-token lower-quality dataset produced no significant improvement over the 7T higher-quality version (Section 3.1)—is a negative result with real implications: it suggests that for models at this scale and training duration, data quality bottlenecks have replaced data quantity bottlenecks, and the field's attention should shift accordingly.

Why this is distinctive: Prior major LLM releases have typically emphasized architectural innovations (the original Transformer, sparse attention patterns, mixture-of-experts routing algorithms) or training methodology breakthroughs (RLHF, constitutional AI). Even when data was acknowledged as important, it was rarely the central narrative. Llama-3 (AI@Meta, 2024) emphasized scale and careful data filtering. Mistral (Jiang et al., 2023a) highlighted sliding window attention. Gemma (Mesnard et al., 2024) focused on distillation from larger models. The Qwen2 paper, by foregrounding data composition, quality filtering, and the 7T-vs-12T decision as its primary technical storyline, makes an implicit argument: in the current era, the hardest engineering problems in LLM development—and the largest sources of performance variance—are in data, not in architecture.

The contamination analysis (Section 5.2.6, Table 15) reinforces this framing. By implementing a two-stage decontamination process (n-gram + LCS during construction; strict n-gram for post-hoc assessment) and transparently reporting the results—including the counterintuitive finding that some benchmarks improve on non-contaminated subsets—the paper treats evaluation integrity as a first-class technical problem, not an afterthought. This level of rigor in data hygiene is rare in technical reports and represents a conceptual contribution to evaluation methodology, even though it produces no direct performance gain.

Significance beyond performance: This reframing has practical consequences for how research teams allocate effort. If data quality, composition, and hygiene are the primary levers for improvement at current scales, then investments in better filtering algorithms, synthetic data generation, multilingual corpus collection, and contamination detection yield higher returns than investments in novel attention patterns or activation functions. The paper provides concrete evidence for this thesis across all five model scales, from 0.5B to 72B—the gains are remarkably consistent regardless of architecture size, suggesting the data effect dominates.

Limitation of this framing: The paper does not prove that architecture is unimportant—it simply demonstrates that data improvements alone can produce substantial gains. A model with both Qwen2's data pipeline and a genuinely novel architecture might outperform Qwen2 by margins that would make data-only improvements look modest. The paper's implicit argument is about where marginal research effort is best spent given current capabilities, not about the absolute ceiling of architectural innovation.

Innovation 2: Making Mixture-of-Experts Efficiency Pragmatic Through Fine-Grained Experts and Upcycling with Diversity-Promoting Initialization

The Qwen2-57B-A14B model represents a significant practical advance in the deployment economics of large language models: it matches the performance of ~30B dense models (Table 3) while activating only 14B parameters per token, achieving roughly 2× efficiency in activated parameters. What makes this contribution intellectually distinctive is not the MoE architecture itself—fine-grained experts come from Dai et al. (2024), shared+routed expert routing from Rajbhandari et al. (2022), upcycling from Komatsuzaki et al. (2023)—but rather the specific combination of design decisions and the validation that they work together at meaningful scale.

The three design choices that jointly enable this efficiency are:

Fine-grained experts (64 experts of 2,560 intermediate size each, 8 activated per token) rather than coarse-grained (8 experts of ~18,944 each, 2 activated): This combinatorial argument—that selecting 8 from 64 provides vastly more expert combinations than selecting 2 from 8—is straightforward mathematically, but the empirical question is whether the router can learn meaningful specialization across so many small experts without collapsing to a few dominant ones. The paper's strong benchmark results (76.5 MMLU, 80.7 GSM8K, 53.0 HumanEval in Table 3) suggest it can.

Shared experts alongside routed experts: This decouples universal computation from specialized computation, ensuring that even poorly-routed tokens receive a competent baseline. Prior MoE models sometimes exhibited "expert collapse" where the router sent most tokens to a single expert regardless of content. The shared expert design provides insurance against this failure mode.

The upcycling initialization with shuffling and 50% random reinitialization: This is the most underappreciated innovation in the paper. Naive upcycling—simply copying dense FFN weights into each expert—produces identical experts that provide no benefit over the original dense model and may never diversify during training because all experts compute the same function and receive the same gradients (up to routing noise). The shuffling step creates initial diversity by permuting the intermediate dimension, ensuring each expert starts with a different subset of the dense model's neurons. The 50% random reinitialization introduces stochasticity that gives the router a reason to differentiate between experts from the first training step: two experts may process the same token type, but one happens to have random weights that produce a slightly better output, so the router learns to prefer it for that token type, and the expert specializes further from there.

Why this matters beyond raw performance: The economics of LLM deployment are dominated by inference costs. A model that achieves 30B-dense-model quality with 14B activated parameters cuts serving costs by roughly 50% for the same quality level (assuming costs scale roughly with activated parameters). For organizations deploying LLMs at scale—handling millions of queries per day—this is a multi-million-dollar annual savings. The paper's contribution is demonstrating that this efficiency is achievable without sacrificing quality, contra earlier MoE results like Mixtral-8x7B (Jiang et al., 2024) which showed some benchmark gaps relative to comparably-sized dense models.

A critical nuance: Table 7 shows that Qwen2-57B-A14B-Instruct scores below Yi-1.5-34B-Chat on GSM8K (85.3 vs. 90.2) and MATH (49.1 vs. 50.1), and below Qwen1.5-32B-Chat on some English knowledge benchmarks. The "matches ~30B dense" claim is approximately true on average but the MoE model is not strictly dominating—there are capability tradeoffs that the paper does not analyze in detail. The authors acknowledge this implicitly by noting the MoE model "is designed to match the performance of 30 billion parameter dense models" (Table 3 caption), not exceed it.

The initialization method is a conceptual contribution in its own right: The idea that you can create diverse MoE experts from a single dense checkpoint by shuffling and partially randomizing dimensions is elegant and likely generalizes beyond this specific model. It solves a fundamental bootstrapping problem in MoE training: how to get experts to specialize when they all start identical. The 50% reinitialization fraction is a hyperparameter likely tuned empirically (the paper provides no ablation), and the method's sensitivity to this fraction is an open question. But the concept—using structured permutation plus partial randomization to seed expert diversity—is a genuine design insight.

Innovation 3: Transparent Post-Training Methodology as a Contribution to Reproducibility and Community Practice

This paper's detailed documentation of its post-training pipeline—spanning collaborative data annotation (ontology extraction, instruction selection, instruction evolution, human annotation) and automated data synthesis (rejection sampling, execution feedback, data repurposing, constitutional feedback)—represents an intellectual contribution to the practice of LLM development rather than to the theory of language modeling. The contribution is not any single technique (all are drawn from prior work with explicit citations) but the comprehensive, end-to-end documentation of how these techniques are combined into a production pipeline, including design decisions that are typically omitted from technical reports.

Several specific documentation choices stand out as contributions:

The instruction evolution strategy (Section 4.1.1): Using the Qwen model itself to add constraints to existing instructions creates a curriculum of difficulty levels from a single seed set. This is simple—prompt the model with "make this instruction more complex by adding constraints"—but the paper documents it as an explicit step in the pipeline, making it reproducible rather than a tribal-knowledge trick.

The dual use of execution feedback (Section 4.1.2): For code, the standard approach is generating test cases and evaluating solutions. The paper extends this same paradigm to instruction following—have the model generate a Python verification function that checks whether a response meets specified constraints (length limits, format requirements, content restrictions). This unifies two seemingly different evaluation problems under a single automated framework, which is conceptually elegant and practically important for scaling instruction-tuning data without human annotators verifying every constraint.

The data repurposing strategy for creative domains (Section 4.1.2): Using existing high-quality text (literary works, Wikipedia) and having the model work backwards to generate the instructions that could have produced that text is a clever inversion of the standard instruction→response paradigm. It leverages the fact that generating plausible instructions given excellent text is easier than generating excellent text given instructions, sidestepping the hardest part of creative data synthesis.

The Online Merging Optimizer to mitigate alignment tax (Section 4.3): The paper acknowledges a well-known failure mode of RLHF—that aligning models with human preferences can degrade their factual knowledge and reasoning capabilities—and names a specific technique for addressing it. While the paper provides no details on the merging mechanism (all deferred to Lu et al., 2024a), the identification of the problem and the explicit claim of a solution changes the conversation from "alignment tax is an unsolved issue" to "alignment tax is a solvable engineering problem with known approaches."

Why this matters: The LLM research community has a reproducibility crisis driven by the opacity of post-training. Most technical reports describe post-training in vague terms ("we used RLHF," "we collected human preference data") without specifying the data construction pipeline, the balance of human vs. synthetic data, the filtering criteria, or the iteration strategy. This makes it impossible to determine whether a model's performance comes from better pre-training, better post-training data, better alignment techniques, or some combination. By documenting the post-training pipeline in unusual detail—including the specific tools used (InsTag for ontology extraction), the data sources (Wikipedia for character profiles, public domain literature for writing), and the quality control mechanisms (execution feedback for code, rejection sampling for math)—the paper enables the community to conduct meaningful ablation studies on post-training methodology, even if only conceptually.

A limitation: The paper does not provide ablation experiments isolating the contribution of each post-training technique. We do not know, for example, whether instruction evolution contributes more or less to final performance than rejection sampling, or whether constitutional feedback improves safety at the cost of reduced helpfulness. The claim is not that each individual technique is necessary or optimal, but that the documented combination works. This is a contribution to engineering practice, not to scientific understanding of why specific techniques work.

Innovation 4: Normalizing Rigorous Contamination Analysis as an Expected Component of Model Releases

The contamination analysis in Section 5.2.6 may appear to be a minor evaluation detail, but it represents a methodological contribution to how the field should report and interpret benchmark results. The paper goes substantially beyond the standard "we deduplicated against test sets" disclaimer that has become boilerplate in LLM technical reports, and in doing so establishes a template that future releases should follow.

The key methodological moves are:

Two-stage decontamination with different stringency levels: During data construction, the paper removes training sequences using a combined criterion: 13-gram overlap AND LCS length ≥ 60% of the shorter sequence. This is designed to have high precision—removing genuine contamination while minimizing false positives. After training, a stricter criterion (13-gram overlap alone, without LCS threshold) is applied to identify potentially contaminated test samples for analysis. This separation of removal (where false positives incur real cost by reducing training data) from assessment (where false positives only affect the reported statistics, not the model) is conceptually clean and practically sensible.

The LCS-without-punctuation trick: Before computing the longest common subsequence, the paper strips all symbols and punctuation. This is a domain-specific insight: in mathematics and code, symbol sequences like "x = 5; y = 10; return x + y" are so universal that exact n-gram matches are inevitable and meaningless. By focusing on the structural content (the variable names, the algorithmic structure) rather than the syntax, the LCS criterion better captures genuine information leakage.

Reporting performance on the non-contaminated subset: Table 15 shows the performance delta between the full test set and the decontaminated subset. This is the critical contribution: it allows readers to assess whether reported scores are inflated by data leakage. The finding that most deltas are small (within ±2 percentage points) and that some benchmarks improve on the non-contaminated subset (MATH jumps from 69.0 to 74.6 for Qwen2-72B-Instruct) is informative in both directions: it suggests contamination is not a major confound for this release, and it highlights the curious phenomenon where contaminated samples can be harder than average (possibly because formulaic matches on common mathematical expressions flag easy problems as "contaminated" while hard problems with novel expressions pass the filter).

Why this is a conceptual contribution: Prior to papers like this one, contamination analysis in LLM releases was often perfunctory—a sentence or two about n-gram deduplication with no quantitative assessment of remaining contamination or its impact on scores. The field lacked a shared standard for what constitutes adequate contamination reporting. By demonstrating a rigorous, multi-stage methodology and reporting the results transparently, this paper sets a de facto standard that raises expectations for future releases. It makes it harder for subsequent papers to claim "we deduplicated" without providing the kind of quantitative evidence Table 15 provides.

A critical nuance: The paper is honest about the limitations of its own analysis. It acknowledges that the strict 13-gram criterion produces "false positives, primarily stemming from the mathematics and coding datasets" where common expressions are flagged as contamination. This transparency is itself part of the contribution—acknowledging that contamination detection is imperfect prevents readers from over-interpreting the reported deltas as precise measures of contamination impact.

The MATH result deserves special attention: The largest delta in Table 15 is for MATH, where Qwen2-72B-Instruct scores 69.0 on original vs. 74.6 on non-contaminated—a 5.6-point improvement after removing flagged samples. The paper's explanation—that contaminated samples were harder than average, not easier—is plausible but not proven. An alternative interpretation is that the decontamination during training was too aggressive on MATH data, removing training examples that would have helped with the harder test problems, and the flagged MATH samples are simply the ones where the model was undertrained. The paper does not investigate this alternative, and it represents an open question about the interaction between data filtering and model capability on specific benchmarks.

The broader implication: As LLM evaluation becomes increasingly high-stakes (model releases are accompanied by claims of "GPT-4-level performance" or "state-of-the-art on benchmark X"), the credibility of those claims depends on the rigor of contamination control. This paper's approach—transparent, multi-stage, quantitatively assessed—provides a template for how to earn that credibility, and its publication as part of a major model release normalizes the expectation that such analysis is required, not optional.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses a wide array of established benchmark datasets. For base models, these include MMLU (Hendrycks et al., 2021a) (Massive Multitask Language Understanding, 5-shot), MMLU-Pro (Wang et al., 2024) (5-shot), GPQA (Rein et al., 2023) (Graduate-Level Google-Proof Q&A, 5-shot), Theorem QA (Chen et al., 2023a) (5-shot), BBH (Suzgun et al., 2023) (BIG-Bench Hard, 3-shot), HellaSwag (Zellers et al., 2019) (10-shot), Winogrande (Sakaguchi et al., 2021) (5-shot), TruthfulQA (Lin et al., 2022a) (0-shot), ARC-C (Clark et al., 2018) (AI2 Reasoning Challenge, 25-shot), HumanEval (Chen et al., 2021) (0-shot), MBPP (Austin et al., 2021) (0-shot), EvalPlus (Liu et al., 2023a) (0-shot), MultiPL-E (Cassano et al., 2023) (0-shot on Python, C++, Java, PHP, TypeScript, C#, Bash, JavaScript), GSM8K (Cobbe et al., 2021) (5-shot), MATH (Hendrycks et al., 2021b) (4-shot), C-Eval (Huang et al., 2023) (5-shot), and CMMLU (Li et al., 2023) (5-shot). Multilingual evaluation is grouped into four categories: Exam (M3Exam 5-shot, IndoMMLU 3-shot, ruMMLU 5-shot, translated MMLU 5-shot), Understanding (BELEBELE 5-shot, XCOPA 5-shot, XWinograd 5-shot, XStoryCloze 0-shot, PAWS-X 5-shot), Mathematics (MGSM 8-shot CoT), and Translation (Flores-101 5-shot). For instruction-tuned models, additional benchmarks include MT-Bench (Zheng et al., 2023), Arena-Hard (Li et al., 2024), AlignBench (Liu et al., 2023b), MixEval (Ni et al., 2024), IFEval (Zhou et al., 2023) (strict-prompt subset), and LiveCodeBench v1 (Jain et al., 2024). Long-context evaluation uses Needle in a Haystack (Kamradt, 2023), NeedleBench (OpenCompass Contributors, 2023), and LV-Eval (Yuan et al., 2024). The specific test split sizes for most benchmarks are not reported in the paper, though MATH is known to use the Lightman et al. (2022) split of 500 test questions, and the in-house Chinese and English automatic evaluation benchmarks are of unspecified size.
Base model(s). The paper evaluates five models from the Qwen2 family: Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B (a Mixture-of-Experts model with 57B total parameters and 14B activated per token), and Qwen2-72B, along with their instruction-tuned counterparts (Qwen2-0.5B-Instruct, Qwen2-1.5B-Instruct, Qwen2-7B-Instruct, Qwen2-57B-A14B-Instruct, Qwen2-72B-Instruct). These models are chosen to span the full range from on-device deployment scales (0.5B, suitable for smartphones and wearables) to datacenter-scale (72B), with the MoE variant included to demonstrate sparse architecture efficiency. The models are pre-trained on a 7-trillion-token dataset (12T for the 0.5B model) and post-trained via supervised fine-tuning and Direct Preference Optimization.
Metrics. The primary metric across all benchmark evaluations is accuracy — the fraction of test examples where the model's generated answer matches the ground truth. For coding benchmarks (HumanEval, MBPP, EvalPlus, MultiPL-E), accuracy is typically computed as pass@1 (the fraction of problems solved correctly on the first attempt). For alignment benchmarks, MT-Bench uses a 1–10 scale scored by GPT-4 as a judge, Arena-Hard reports a win rate against a reference model, AlignBench reports a score derived from LLM-as-judge evaluation, MixEval approximates Chatbot Arena rankings, and IFEval reports the fraction of responses satisfying all specified constraints (strict-prompt subset). For multilingual human evaluation (Table 13), annotators assign scores from 1 to 5 based on response quality. For safety evaluation (Table 14), the metric is the proportion of harmful responses generated (lower is better). For long-context evaluation, the Needle in a Haystack test reports retrieval accuracy as a function of context length and fact position, while NeedleBench and LV-Eval report accuracy under their respective scoring protocols (LV-Eval uses keyword recall to address false negatives in its original metric).
Baselines. Base model baselines include Mixtral-8x22B (Jiang et al., 2024), Llama-3-70B and Llama-3-8B (AI@Meta, 2024), Qwen1.5-72B and Qwen1.5-110B (Qwen Team, 2024a,b), Mixtral-8x7B (Jiang et al., 2024), Jamba (Lieber et al., 2024), Yi-1.5-34B (Young et al., 2024), Qwen1.5-32B (Qwen Team, 2024a), Mistral-7B-v0.2 (Jiang et al., 2023a), Gemma-7B and Gemma-2B (Mesnard et al., 2024), Qwen1.5-7B, Phi-2 (Abdin et al., 2024), and Qwen1.5-1.8B. Instruction-tuned baselines include Mixtral-8x22B-Instruct, Llama-3-70B-Instruct, Qwen1.5-72B-Chat, Qwen1.5-110B-Chat, Mixtral-8x7B-Instruct, Yi-1.5-34B-Chat, Qwen1.5-32B-Chat, Llama-3-8B-Instruct, Yi-1.5-9B-Chat, GLM-4-9B-Chat (Zeng et al., 2024), Qwen1.5-7B-Chat, Qwen1.5-0.5B-Chat, and Qwen1.5-1.8B-Chat. Proprietary models used as baselines in specific evaluations include GPT-4, GPT-4-Turbo-0409, GPT-4o-2024-05-13, GPT-3.5-Turbo-1106, Claude-3-Opus-0229, and Qwen-Max-0428. Long-context baselines include ChatGLM4-9B-1M (Zeng et al., 2024). For in-house evaluations (Tables 10–11), the primary baselines are Qwen1.5 models of comparable size and Llama-3 models.
Generation budget / compute accounting. The paper does not use "generation budget" as a unit of test-time compute in the manner of inference-scaling studies. Instead, models are compared at fixed parameter scales (e.g., all ~7B models, all ~70B models) with evaluations conducted using the standard few-shot or zero-shot prompting protocols specified for each benchmark. The FLOPs-matched comparison that characterizes inference-time compute scaling studies is not part of this paper's experimental methodology — the evaluation focuses on comparing models of similar scale trained with similar or larger data budgets, with the key independent variables being pre-training data quantity (3T vs. 7T vs. 12T tokens) and quality (targeted code and math enrichment), plus post-training methodology (SFT + DPO). The training compute is not explicitly quantified in FLOPs, though the paper reports the number of pre-training tokens for each model (Table 1): 12T for Qwen2-0.5B, 7T for Qwen2-1.5B, 7B, and 72B, and 4.5T additional tokens for Qwen2-57B-A14B (beyond the 7T base).
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for the benchmark evaluations. The evaluation protocol relies on standard few-shot and zero-shot prompting with established benchmarks, and results are reported as single accuracy values per benchmark per model. For the contamination analysis (Section 5.2.6), the paper computes performance on both the original test set and a non-contaminated subset (constructed by excluding any sample with a 13-gram overlap with pre-training or post-training data), and reports the delta between these two values for each benchmark (Table 15). No confidence intervals, standard errors, or statistical tests are reported for any of the benchmark results. For the in-house Chinese and English evaluations (Tables 10–11), the aggregation methodology is not specified beyond averaging across categories. For the multilingual human evaluation (Table 13), one professional annotator per language evaluates responses on a 1–5 scale, with results reported as averages — no inter-annotator agreement statistics are provided, and the use of a single annotator per language means there is no measure of annotation reliability.

Main Quantitative Results

Base Language Model Core Capabilities (Section 5.1.1)

Qwen2-72B vs. 70B+ baselines (Table 2). Qwen2-72B achieves 84.2 on MMLU (5-shot), representing a 4.7-point improvement over Llama-3-70B (79.5) and a 6.7-point improvement over Qwen1.5-72B (77.5). On MMLU-Pro, Qwen2-72B scores 55.6 versus Llama-3-70B's 52.8 (+2.8) and Qwen1.5-110B's 49.4 (+6.2). On GPQA, Qwen2-72B achieves 37.9, edging out Llama-3-70B (36.3) by 1.6 points. On Theorem QA, Qwen2-72B scores 43.1, substantially ahead of Llama-3-70B (32.3) by 10.8 points and Mixtral-8x22B (35.9) by 7.2 points. On BBH (3-shot), Qwen2-72B achieves 82.4 versus Llama-3-70B's 81.0 (+1.4) and Qwen1.5-72B's 65.5 (+16.9), the latter indicating a major improvement in reasoning capabilities over the previous generation.

In coding, Qwen2-72B scores 64.6 on HumanEval (0-shot) compared to Llama-3-70B's 48.2 (+16.4), Qwen1.5-110B's 54.3 (+10.3), and Mixtral-8x22B's 46.3 (+18.3). On MBPP, Qwen2-72B achieves 76.9 versus Llama-3-70B's 70.4 (+6.5) and Qwen1.5-72B's 66.9 (+10.0). On MultiPL-E (averaged across 8 programming languages), Qwen2-72B scores 59.6 versus Llama-3-70B's 46.3 (+13.3) and Qwen1.5-72B's 41.8 (+17.8).

In mathematics, Qwen2-72B achieves 89.5 on GSM8K (5-shot), surpassing Llama-3-70B's 83.0 (+6.5), Qwen1.5-110B's 85.4 (+4.1), and Mixtral-8x22B's 83.7 (+5.8). On MATH (4-shot), Qwen2-72B scores 51.1, ahead of Llama-3-70B's 42.5 (+8.6) and Qwen1.5-72B's 34.1 (+17.0).

In Chinese language understanding, Qwen2-72B achieves 91.0 on C-Eval versus Llama-3-70B's 65.2 (+25.8) and 90.1 on CMMLU versus Llama-3-70B's 67.2 (+22.9). The contrast with English-centric models is stark: Mixtral-8x22B scores only 54.6 on C-Eval and 53.4 on CMMLU, both less than Qwen2-72B by over 36 points.

In multilingual evaluation (aggregated across Exam, Understanding, Mathematics, and Translation categories), Qwen2-72B shows advantages over all baselines. On the Multilingual Exam aggregate, Qwen2-72B scores 76.6 versus Llama-3-70B's 70.0 (+6.6) and Qwen1.5-110B's 75.6 (+1.0). On Multilingual Understanding, Qwen2-72B scores 80.7 versus Llama-3-70B's 79.9 (+0.8). On Multilingual Mathematics, Qwen2-72B achieves 76.0, substantially ahead of Llama-3-70B's 67.1 (+8.9) and Qwen1.5-110B's 64.4 (+11.6).

Qwen2-57B-A14B vs. 30B+ baselines (Table 3). The MoE model with 14B activated parameters achieves 76.5 on MMLU, outperforming Mixtral-8x7B (71.8, also MoE with 12B activated) by 4.7 points but trailing Yi-1.5-34B (77.1, dense 32B) by 0.6 points. On HumanEval, Qwen2-57B-A14B scores 53.0 versus Mixtral-8x7B's 37.2 (+15.8) and Yi-1.5-34B's 46.3 (+6.7). On GSM8K, Qwen2-57B-A14B achieves 80.7, surpassing Mixtral-8x7B's 62.5 (+18.2) but falling behind Yi-1.5-34B's 82.7 (−2.0). On MATH, Qwen2-57B-A14B scores 43.0, ahead of Mixtral-8x7B's 30.8 (+12.2) and Qwen1.5-32B's 36.1 (+6.9), and marginally ahead of Yi-1.5-34B's 41.7 (+1.3). In Chinese, Qwen2-57B-A14B achieves 87.7 on C-Eval and 88.5 on CMMLU, comparable to Qwen2-72B (91.0 and 90.1) despite having roughly one-quarter the activated parameters. Across multilingual categories, Qwen2-57B-A14B scores 65.5 on Exam, 77.0 on Understanding, 62.3 on Mathematics, and 34.5 on Translation, consistently outperforming Qwen1.5-32B (61.6, 76.5, 56.1, 33.5 respectively) and Yi-1.5-34B (58.3, 73.9, 49.3, 30.0).

Qwen2-7B vs. 7B+ baselines (Table 4). Qwen2-7B achieves 70.3 on MMLU, surpassing Llama-3-8B's 66.6 (+3.7), Mistral-7B-v0.2's 64.2 (+6.1), Gemma-7B's 64.6 (+5.7), and Qwen1.5-7B's 61.0 (+9.3). On HumanEval, Qwen2-7B scores 51.2, well ahead of Llama-3-8B's 33.5 (+17.7) and Gemma-7B's 37.2 (+14.0). On GSM8K, Qwen2-7B achieves 79.9, substantially above Llama-3-8B's 56.0 (+23.9), Mistral-7B's 52.2 (+27.7), and Qwen1.5-7B's 62.5 (+17.4). On MATH, Qwen2-7B scores 44.2, more than doubling Llama-3-8B's 20.5 (+23.7) and Gemma-7B's 24.3 (+19.9). In Chinese, Qwen2-7B achieves 83.2 on C-Eval and 83.9 on CMMLU, dramatically above Llama-3-8B's 49.5 (+33.7) and 50.8 (+33.1) respectively. On BBH, Qwen2-7B scores 62.6 versus Llama-3-8B's 57.7 (+4.9). However, Qwen2-7B trails Llama-3-8B on HellaSwag (80.7 vs. 82.2, −1.5) and slightly on Winogrande (77.0 vs. 77.4, −0.4).

Qwen2-1.5B and Qwen2-0.5B vs. small model baselines (Table 5). Qwen2-1.5B (1.2B non-embedding parameters) achieves 56.5 on MMLU, outperforming Phi-2 (2.5B non-embedding params, 52.7) by 3.8 points and Qwen1.5-1.8B (1.2B non-embedding, 46.8) by 9.7 points. Qwen2-0.5B (0.3B non-embedding parameters) scores 45.4 on MMLU, exceeding Gemma-2B (2.0B non-embedding, 42.3) by 3.1 points. On GSM8K, Qwen2-1.5B achieves 58.5, edging out Phi-2's 57.2 (+1.3) and substantially ahead of Qwen1.5-1.8B's 38.4 (+20.1). Qwen2-0.5B scores 36.5 on GSM8K, more than doubling Gemma-2B's 17.7. On HumanEval, Qwen2-1.5B scores 31.1, trailing Phi-2's 47.6 (−16.5) but ahead of Gemma-2B's 22.0 (+9.1). Qwen2-0.5B achieves 22.0 on HumanEval, matching Gemma-2B exactly and exceeding Qwen1.5-1.8B's 20.1 (+1.9). On BBH, both Qwen2 models trail Phi-2: Qwen2-1.5B scores 37.2 versus Phi-2's 43.4 (−6.2). On C-Eval, Qwen2-1.5B achieves 70.6 versus Phi-2's 23.4 (+47.2), and Qwen2-0.5B scores 58.2 versus Phi-2's 23.4 (+34.8), highlighting the extreme Chinese language performance gap between models with and without Chinese-centric pre-training.

Instruction-Tuned Model Evaluation (Section 5.2)

Qwen2-72B-Instruct vs. 70B+ instruction-tuned baselines (Table 6). On core capability benchmarks, Qwen2-72B-Instruct scores 82.3 on MMLU, matching Llama-3-70B-Instruct's 82.0 (+0.3) and substantially above Qwen1.5-72B-Chat's 75.6 (+6.7). On MMLU-Pro, Qwen2-72B-Instruct achieves 64.4, ahead of Llama-3-70B-Instruct's 56.2 (+8.2). On HumanEval, Qwen2-72B-Instruct scores 86.0 versus Llama-3-70B-Instruct's 81.7 (+4.3). On GSM8K, Qwen2-72B-Instruct achieves 93.2, nearly matching Llama-3-70B-Instruct's 93.0 (+0.2). On MATH, Qwen2-72B-Instruct scores 69.0, substantially above Llama-3-70B-Instruct's 50.4 (+18.6). On LiveCodeBench v1, Qwen2-72B-Instruct achieves 35.7 versus Llama-3-70B-Instruct's 29.3 (+6.4).

On alignment benchmarks, Qwen2-72B-Instruct scores 9.12 on MT-Bench, surpassing Llama-3-70B-Instruct's 8.95 (+0.17) and Mixtral-8x22B-Instruct's 8.66 (+0.46). On Arena-Hard, Qwen2-72B-Instruct achieves 48.1 versus Llama-3-70B-Instruct's 41.1 (+7.0). On IFEval (strict-prompt), Qwen2-72B-Instruct scores 77.6, essentially tied with Llama-3-70B-Instruct's 77.3 (+0.3) and well above Qwen1.5-72B-Chat's 55.8 (+21.8). On AlignBench, Qwen2-72B-Instruct scores 8.27 versus Llama-3-70B-Instruct's 7.42 (+0.85). On MixEval, Qwen2-72B-Instruct achieves 86.7 versus Llama-3-70B-Instruct's 84.0 (+2.7). Notably, Qwen2-72B-Instruct scores lower than Llama-3-70B-Instruct on GPQA (42.4 vs. 41.9, a 0.5-point disadvantage for Qwen2, though the baseline scores 41.9, not 42.4 — actually Qwen2 is slightly ahead at 42.4 vs. 41.9, +0.5) and on MBPP (80.2 vs. 82.3, −2.1).

Qwen2-57B-A14B-Instruct vs. 30B+ instruction-tuned baselines (Table 7). Qwen2-57B-A14B-Instruct achieves 75.4 on MMLU, slightly below Yi-1.5-34B-Chat's 76.8 (−1.4) but above Qwen1.5-32B-Chat's 74.8 (+0.6). On HumanEval, Qwen2-57B-A14B-Instruct scores 79.9, ahead of Yi-1.5-34B-Chat's 75.2 (+4.7). On GSM8K, Qwen2-57B-A14B-Instruct achieves 85.3, trailing Yi-1.5-34B-Chat's 90.2 (−4.9). On MATH, Qwen2-57B-A14B-Instruct scores 49.1, slightly below Yi-1.5-34B-Chat's 50.1 (−1.0). On MT-Bench, Qwen2-57B-A14B-Instruct achieves 8.55, edging out Yi-1.5-34B-Chat's 8.50 (+0.05). On MixEval, Qwen2-57B-A14B-Instruct scores 82.3 versus Yi-1.5-34B-Chat's 81.7 (+0.6). On IFEval, Qwen2-57B-A14B-Instruct achieves 59.9, far ahead of Qwen1.5-32B-Chat's 50.3 (+9.6).

Qwen2-7B-Instruct vs. 7B+ instruction-tuned baselines (Table 8). Qwen2-7B-Instruct achieves 70.5 on MMLU, ahead of Llama-3-8B-Instruct's 68.4 (+2.1) and Qwen1.5-7B-Chat's 59.5 (+11.0). On HumanEval, Qwen2-7B-Instruct scores 79.9, substantially above Llama-3-8B-Instruct's 62.2 (+17.7). On GSM8K, Qwen2-7B-Instruct achieves 85.7, ahead of Llama-3-8B-Instruct's 79.6 (+6.1). On MATH, Qwen2-7B-Instruct scores 52.9 versus Llama-3-8B-Instruct's 30.0 (+22.9). On MT-Bench, Qwen2-7B-Instruct achieves 8.41, above Llama-3-8B-Instruct's 8.05 (+0.36). On LiveCodeBench v1, Qwen2-7B-Instruct scores 26.6 versus Llama-3-8B-Instruct's 17.3 (+9.3). However, on IFEval (strict-prompt), Qwen2-7B-Instruct scores only 54.7, substantially below Llama-3-8B-Instruct's 72.1 (−17.4), representing a major weakness in instruction following that the paper explicitly acknowledges: "in terms of instruction following, Qwen2-7B-Instruct greatly falls behind the competitor."

Qwen2-1.5B-Instruct and Qwen2-0.5B-Instruct vs. small instruction-tuned baselines (Table 9). Qwen2-1.5B-Instruct achieves 52.4 on MMLU, above Qwen1.5-1.8B-Chat's 43.7 (+8.7). On HumanEval, Qwen2-1.5B-Instruct scores 47.0 versus Qwen1.5-1.8B-Chat's 27.4 (+19.6). On GSM8K, Qwen2-1.5B-Instruct achieves 61.6 versus Qwen1.5-1.8B-Chat's 35.3 (+26.3). On IFEval, Qwen2-1.5B-Instruct scores 29.0 versus Qwen1.5-1.8B-Chat's 16.8 (+12.2). Qwen2-0.5B-Instruct achieves 37.9 on MMLU (vs. Qwen1.5-0.5B-Chat's 35.0, +2.9), 29.9 on HumanEval (vs. 10.4, +19.5), 40.1 on GSM8K (vs. 11.3, +28.8), and 20.0 on IFEval (vs. 14.6, +5.4). The GSM8K jump from 11.3 to 40.1 for the 0.5B model is particularly notable — nearly a 4× improvement — suggesting that the enhanced mathematics pre-training data has disproportionate benefits at the smallest scales.

In-House Automatic Evaluation (Section 5.2.2)

Chinese evaluation (Table 10). Qwen2-72B-Instruct achieves an average score of 69.58 across six categories (Knowledge, Exam, Comprehension, Coding, Math, Reasoning), surpassing GPT-4o-2024-05-13's 68.87 (+0.71) and Qwen1.5-110B-Chat's 65.86 (+3.72). The largest category-level advantages over Qwen1.5-110B-Chat are in Coding (49.53 vs. 44.25, +5.28) and Math (70.80 vs. 64.92, +5.88). Qwen2-7B-Instruct achieves 56.96 average, above Qwen1.5-7B-Chat's 47.61 (+9.35). Qwen2-1.5B-Instruct scores 36.11 versus Qwen1.5-1.8B-Chat's 31.90 (+4.21). Qwen2-57B-A14B-Instruct achieves 61.63 average, above Qwen1.5-32B-Chat's 59.94 (+1.69), though it trails on Knowledge (64.15 vs. 68.63, −4.48) — the paper attributes this to "a short of pre-training tokens" for the MoE model.

English evaluation (Table 11). Qwen2-72B-Instruct achieves an average of 72.94 across four categories (Knowledge, Comprehension, Coding, Math), below GPT-4o-2024-05-13's 76.11 (−3.17) and Llama-3-70B-Instruct's 74.06 (−1.12). The gap with Llama-3-70B-Instruct is concentrated in Knowledge (83.00 vs. 83.06, −0.06) and Comprehension (73.58 vs. 76.31, −2.73), while Qwen2-72B-Instruct leads in Math (82.15 vs. 79.70, +2.45) and trails in Coding (53.03 vs. 57.18, −4.15). Qwen2-7B-Instruct achieves 62.23, ahead of Llama-3-8B-Instruct's 61.03 (+1.20). The paper attributes the English performance gap to "both the amount of English tokens for pre-training and the quantity and diversity of data for post-training."

Long Context Capabilities (Section 5.2.3)

Needle in a Haystack (Figure 1). The heatmap visualization shows Qwen2-72B-Instruct achieving near-perfect retrieval accuracy (100%, indicated by the color scale) across the full 128K context length and at all document depths (from 0% to 100%). For longer contexts (beyond 32K), YARN is applied. Qwen2-7B-Instruct also demonstrates high accuracy up to 128K tokens, though with slightly more variability at extreme depths. Qwen2-57B-A14B-Instruct performs well up to 64K tokens. Qwen2-1.5B-Instruct and Qwen2-0.5B-Instruct handle contexts up to 32K tokens proficiently. The paper states that "all models that supports context lengths above 32k tokens integrates the YARN mechanism."

NeedleBench and LV-Eval (Table 12). Qwen2-72B-Instruct, with YARN and DCA, achieves 90.27 on NeedleBench at 32K (vs. 92.01 without YARN+DCA), 85.21 at 128K (vs. 73.05 without, +12.16), 53.03 at 256K (vs. 17.13 without, +35.9). On LV-Eval, Qwen2-72B-Instruct with YARN+DCA scores 48.83 at 64K (vs. 42.92 without, +5.91), 42.35 at 128K (vs. 31.79 without, +10.56), and degrades to 2.88 at 256K even with YARN+DCA. Qwen2-7B-Instruct with YARN+DCA achieves 66.32 on NeedleBench at 32K (vs. 73.64 without, −7.32), 60.71 at 128K (vs. 38.77 without, +21.94), and 42.14 at 256K (vs. 2.92 without, +39.22). The negative delta at 32K (−7.32) for Qwen2-7B-Instruct suggests YARN+DCA slightly degrades performance at the native context length while providing substantial benefits at extended lengths. The comparison with ChatGLM4-9B-1M shows Qwen2-7B-Instruct with YARN+DCA outperforming at 8K (87.07 vs. 56.61) and 32K (73.64 vs. 49.15) but converging at 128K (38.77 vs. 44.30).

Multilingual Human Evaluation (Section 5.2.4)

Table 13 results. Qwen2-72B-Instruct achieves an average score of 3.93 out of 5 across 10 languages, substantially above GPT-3.5-Turbo-1106's 3.16 (+0.77) and competitive with GPT-4-Turbo-0409's 3.98 (−0.05). By language, Qwen2-72B-Instruct scores highest in Russian (4.15) and Korean (4.14), and lowest in Japanese (3.63) and Thai (3.75). Compared to Claude-3-Opus-0229 (average 4.15), Qwen2-72B-Instruct trails by 0.22 points on average, with the largest gaps in Arabic (3.86 vs. 4.15, −0.29) and Indonesian (3.83 vs. 4.40, −0.57). Compared to GPT-4o-0513 (average 4.09), Qwen2-72B-Instruct trails by 0.16 points. The paper notes that Qwen2-72B-Instruct "significantly outperforms GPT-3.5-Turbo and is competitive with GPT-4-Turbo," summarizing the finding that it approaches but does not consistently match the strongest proprietary models released in the preceding six months.

Safety & Responsibility (Section 5.2.5)

Table 14 results. The safety evaluation measures the proportion of harmful responses generated in response to jail-breaking prompts across four risk categories. Qwen2-72B-Instruct generates harmful responses at rates of 0.00% for Illegal (matching GPT-4's 0.00%), 2.41% for Fraud (vs. GPT-4's 3.40% and Mixtral-8x22B-Instruct's 8.49%), 22.91% for Pornography (vs. GPT-4's 23.63% and Mixtral-8x22B-Instruct's 33.82%), and 2.47% for Privacy (vs. GPT-4's 3.37% and Mixtral-8x22B-Instruct's 15.03%). The paper notes that pornography remains "a conventionally difficult category to differentiate even for humans" and that "there is still much room for our model to improve."

Contamination Analysis (Section 5.2.6)

Table 15 results. Under the strict 13-gram overlap criterion, the percentage of contaminated samples varies widely by benchmark: 75.0% of HumanEval samples are flagged, 37.7% of MultiPL-E, 31.7% of MATH, 29.6% of MBPP, 11.6% of MMLU-Pro, 11.2% of MMLU, and less than 1% for GPQA (1.0%), GSM8K (0.7%), and IFEval (0.9%). For Qwen2-72B-Instruct, the performance difference (∆) between original and non-contaminated test sets is: +0.9 on MMLU, +1.2 on MMLU-Pro, −0.6 on GPQA, +1.0 on HumanEval, −0.5 on MBPP, 0.0 on MultiPL-E, +0.4 on GSM8K, +5.6 on MATH, +0.2 on IFEval. For Qwen2-7B-Instruct, the deltas are: +0.8 on MMLU, +2.4 on MMLU-Pro, −0.2 on GPQA, +7.9 on HumanEval, +1.8 on MBPP, −0.2 on MultiPL-E, −0.1 on GSM8K, +4.7 on MATH, −1.0 on IFEval. The paper interprets these small deltas (with the notable exception of MATH) as evidence that "the potential issue of data contamination does not significantly impact the model's performance."

Ablation Studies and Robustness Checks

This paper is primarily a technical report documenting a model release rather than a research paper presenting controlled experiments. Consequently, there are no traditional ablation studies in the sense of systematically removing components and measuring performance impact. The paper does not isolate the contribution of individual architectural decisions (GQA vs. MHA, DCA vs. standard attention, fine-grained vs. coarse-grained experts), individual pre-training data components (code data vs. math data vs. multilingual data), or individual post-training techniques (rejection sampling vs. execution feedback vs. constitutional feedback). However, several analyses function as implicit ablations or robustness checks:

Data quantity scaling (7T vs. 12T): Section 3.1 reports that "an attempt to further relax the quality threshold resulted in a 12 trillion token dataset. However, the model trained on this dataset did not show a significant performance improvement over the 7 trillion token model." This is presented as a finding rather than a formal ablation with reported numbers, and no head-to-head benchmark results are provided comparing a model trained on 7T vs. 12T at the same scale. The only concrete consequence is that Qwen2-0.5B was trained on 12T tokens while all other dense models used 7T, making any performance comparison between the 0.5B and 1.5B models confounded by both model scale and data quantity.
Long-context method comparison (YARN+DCA vs. baseline): Table 12 provides results with and without YARN+DCA for Qwen2-72B-Instruct and Qwen2-7B-Instruct on NeedleBench and LV-Eval at multiple context lengths. This functions as an ablation of the long-context extension methods, showing that YARN+DCA dramatically improves performance at lengths beyond the 32K training window (e.g., +35.9 points at 256K for Qwen2-72B-Instruct on NeedleBench) while sometimes slightly degrading performance at 32K (e.g., −7.32 for Qwen2-7B-Instruct on NeedleBench at 32K).
Contamination impact assessment (original vs. non-contaminated): Table 15 compares performance on the full test set vs. a non-contaminated subset, functioning as an ablation of data contamination effects. The finding that deltas are mostly within ±2 percentage points (except MATH) suggests benchmark scores are not meaningfully inflated by data leakage.
MoE model vs. dense equivalent: The comparison between Qwen2-57B-A14B (Table 3) and dense models of similar capability (Yi-1.5-34B, Qwen1.5-32B) functions as an implicit ablation of the MoE architecture itself — the question being whether sparse activation achieves comparable performance to dense models with ~2× more activated parameters. The answer is a qualified yes: Qwen2-57B-A14B is competitive on average but trails on specific benchmarks (GSM8K: 80.7 vs. Yi-1.5-34B's 82.7; MMLU: 76.5 vs. Yi-1.5-34B's 77.1).
Model scale progression: The performance progression across model sizes (0.5B → 1.5B → 7B → 57B-A14B → 72B) observable across Tables 2–5 functions as an implicit scaling law demonstration, showing that performance improves with scale but with diminishing returns on some benchmarks and strong scaling on others (particularly coding and mathematics).

The absence of formal ablations is a significant limitation of the paper for readers seeking to understand which specific improvements drove the performance gains. For example, Qwen2-72B's 17-point improvement on MATH over Qwen1.5-72B (51.1 vs. 34.1, Table 2) could be due to the increased math pre-training data, the improved general data quality, the larger total token count (7T vs. 3T), the long-context training phase, or interactions between these factors. The paper provides no decomposition.

Critical Assessment

Does the paper demonstrate that Qwen2 surpasses prior open-weight models? Yes, with strong evidence from Tables 2–9. The comparisons against Llama-3-70B, Mixtral-8x22B, Qwen1.5-72B/110B, and other baselines are consistent and comprehensive across knowledge, reasoning, coding, mathematics, and multilingual benchmarks. The margins are substantial on many benchmarks (e.g., +16.4 over Llama-3-70B on HumanEval, +17.0 over Qwen1.5-72B on MATH). However, the claim requires qualification on two dimensions:

First, Qwen2-72B does not universally outperform all baselines. On HellaSwag, Qwen2-72B scores 87.6 versus Mixtral-8x22B's 88.7 (−1.1). On ARC-C, Qwen2-72B scores 68.9 versus Mixtral-8x22B's 70.7 (−1.8). These are small deficits but indicate that the performance profile is not strictly dominant. For instruction-tuned models, Qwen2-72B-Instruct trails Llama-3-70B-Instruct on MBPP (80.2 vs. 82.3, −2.1) and Qwen2-7B-Instruct dramatically trails Llama-3-8B-Instruct on IFEval (54.7 vs. 72.1, −17.4), a concerning gap in instruction-following capability.

Second, the paper compares against open-weight models available at the time of writing, but the landscape moves quickly. Models released after the paper's submission are not included, and the claim of "surpassing most prior open-weight models" is time-bounded in ways the paper does not explicitly address.

Does the paper demonstrate competitive performance against proprietary models? Partially. The evidence is strongest for base model capabilities, where Qwen2-72B's 84.2 on MMLU approaches GPT-4's reported ~86.4 (though GPT-4's score is from an earlier evaluation protocol and may not be directly comparable). The multilingual human evaluation (Table 13) provides the most direct proprietary comparison, showing Qwen2-72B-Instruct (3.93) competitive with GPT-4-Turbo (3.98) but trailing GPT-4o (4.09) and Claude-3-Opus (4.15). The in-house Chinese evaluation (Table 10) shows Qwen2-72B-Instruct (69.58) slightly ahead of GPT-4o (68.87). However, the in-house English evaluation (Table 11) shows Qwen2-72B-Instruct (72.94) clearly behind GPT-4o (76.11) and Llama-3-70B-Instruct (74.06).

The "competitive with proprietary models" claim is thus conditionally supported: it holds for Chinese-language tasks, mathematics, and coding (where Qwen2-72B-Instruct leads GPT-4o on HumanEval 86.0 vs. a typical GPT-4 score in the high 80s, though the paper doesn't provide the exact GPT-4 comparison), but does not hold for English comprehension and general knowledge where a gap of 2–4 points persists. The claim would be more precise if scoped to specific capability domains rather than stated as a general property.

Does the data scaling hypothesis hold across all model scales? The paper's central narrative is that improved data quality and quantity drive the observed gains. The evidence for this is primarily the before/after comparison with Qwen1.5, which was trained on 3T tokens with lower-quality data. The consistent improvements across all model sizes (Tables 2–5 and 6–9) are consistent with the data-scaling hypothesis, but they do not isolate data from other changes (architecture, training duration, post-training methodology). The strong performance of Qwen2-0.5B — trained on 12T tokens — relative to models with 4–8× more parameters (Phi-2 at 2.5B, Gemma-2B at 2.0B) provides the most suggestive evidence that data quantity matters disproportionately at small scales, but this is confounded by the quality difference between Qwen2's curated dataset and the training data of the baselines.

What the experiments do not show:

No causal attribution of gains to specific data components. We do not know whether the code data expansion, math data expansion, or multilingual data expansion contributed most to the benchmark improvements. The paper hypothesizes that "enrichment is hypothesized to improve reasoning abilities" but provides no ablation where, say, math data is held out to measure its impact on GSM8K and MATH.
No comparison between data quality and data quantity. The 7T vs. 12T finding is reported but no benchmark results are shown for a model trained on 12T of the same quality as the 7T dataset. The claim that "increasing the volume of data does not necessarily benefit model pre-training" conflates quantity with quality — it may be that 12T tokens of equally high quality would improve performance, and the null result is specific to the lower-quality 12T dataset.
No ablation of post-training techniques. The relative contributions of SFT, offline DPO, online DPO, rejection sampling, execution feedback, data repurposing, constitutional feedback, and the Online Merging Optimizer are all unknown. The paper documents these techniques but makes no attempt to measure their individual or interactive effects.
No analysis of the 50% expert reinitialization fraction for MoE. The upcycling initialization method is described in detail but no experiments vary the reinitialization percentage to determine whether 50% is optimal or whether the method is robust to this choice.
No statistical significance testing. All benchmark results are reported as point estimates without confidence intervals, standard errors, or significance tests. For benchmarks with small test sets (e.g., HumanEval at 164 problems, GSM8K at 1,319 test examples), differences of 1–3 points may not be statistically significant, yet the paper treats all numerical differences as meaningful.
Single-annotator multilingual evaluation. The human evaluation in Table 13 uses one professional annotator per language. With a single annotator, there is no measure of inter-annotator agreement, and the results are vulnerable to individual annotator bias. A 0.22-point average difference between Qwen2-72B-Instruct (3.93) and Claude-3-Opus (4.15) may be within the range of annotator-specific rating tendencies.

Missing experiments that would strengthen the paper:

A data ablation study training models at a fixed scale (e.g., 7B) on subsets of the training data (no code expansion, no math expansion, no multilingual expansion, original Qwen1.5 data) to measure the marginal contribution of each data component to downstream benchmarks.
A post-training ablation comparing base model → SFT only → SFT + offline DPO → SFT + offline DPO + online DPO to measure the contribution of each alignment stage.
A controlled 12T vs. 7T comparison at a single model scale with quality held constant, to test whether the null result is due to quality dilution or a genuine saturation effect.
Multiple annotators per language for the multilingual human evaluation, with inter-annotator agreement statistics, to establish the reliability of the ratings.
Confidence intervals for all benchmark results, particularly for small test sets where sampling error could meaningfully affect rankings.
An evaluation of the MoE model's expert utilization (e.g., distribution of tokens across experts, evidence of specialization) to validate the claim that fine-grained experts "facilitate more diverse and dynamic expert utilization."

Conditional nature of the claims:

The claim that Qwen2 "surpasses most prior open-weight models" holds at release time for the specific baselines tested. It is a snapshot of a moving target.
The claim of "competitive performance relative to proprietary models" holds conditionally: strong on Chinese tasks, mathematics, and coding; weaker on English comprehension and knowledge; competitive on multilingual generation but not dominant. The paper's abstract wording "competitive" is appropriately hedged, but the detailed results show a more nuanced picture.
The claim that data scaling "remains an effective strategy for enhancing model performance, even in the domain of sub-billion parameter models" is supported by the Qwen2-0.5B and 1.5B results compared to Qwen1.5 counterparts, but the comparison with Phi-2 — which achieves higher HumanEval (47.6 vs. 31.1) and BBH (43.4 vs. 37.2) with textbook-quality data — suggests that data composition (textbook-style vs. web-scale) may matter as much as data quantity for small models, and the paper does not explore this tradeoff.
The contamination analysis claim that "data contamination does not significantly impact the model's performance" is supported for most benchmarks but warrants caution for MATH, where the 5.6-point improvement on the non-contaminated subset for Qwen2-72B-Instruct (69.0 → 74.6) suggests either that contaminated samples were genuinely harder (as the paper argues) or that the decontamination process removed training data that would have helped with the harder test problems — a distinction the paper does not investigate.

Overall assessment: The experimental evaluation is comprehensive in breadth (covering 30+ benchmarks across knowledge, reasoning, coding, mathematics, multilingual, long-context, safety, and alignment) and provides clear evidence that Qwen2 models represent a substantial improvement over their Qwen1.5 predecessors and are competitive with the strongest open-weight models available at release time. However, the evaluation is descriptive rather than explanatory — it documents performance levels without isolating the causal factors that produced them. The paper makes claims about the importance of data quality and composition, but the experimental design does not support causal attribution of specific performance gains to specific data improvements. This is characteristic of technical reports accompanying model releases and reflects the practical constraints of training large models (where comprehensive ablation studies would be prohibitively expensive), but it means the paper's central narrative — that data improvements drive performance — remains a plausible interpretation of the results rather than a demonstrated finding. The absence of statistical rigor (no confidence intervals, no significance tests, single-annotator human evaluation) is a weakness that limits the interpretability of small performance differences, though the large margins on key benchmarks (e.g., +17.0 on MATH for Qwen2-72B over Qwen1.5-72B) are unlikely to be statistical artifacts.

6. Limitations and Trade-offs

6.1 No Causal Attribution of Performance Gains to Specific Data or Training Improvements

The assumption or constraint. The paper's central narrative is that targeted improvements to pre-training data—more code, more mathematics, higher-quality filtering, and broader multilingual coverage across 7 trillion tokens—are the primary drivers of Qwen2's substantial performance gains over Qwen1.5. However, the experimental design does not isolate any of these factors. Every model in the Qwen2 series differs from its Qwen1.5 counterpart in multiple simultaneous ways: data quantity (3T vs. 7T tokens), data composition (explicitly more code and math), data quality (refined heuristic and model-based filtering), architecture (GQA replacing MHA, DCA added, RoPE base frequency changed), long-context training (32K native context vs. 4K), and post-training methodology (DPO with online merging vs. whatever Qwen1.5 used). The paper provides no ablation study that varies one factor while holding others constant—no model trained on Qwen1.5's 3T dataset with Qwen2's architecture, no model trained on Qwen2's data without the code expansion, no model with and without DCA at otherwise identical settings.

The authors acknowledge this implicitly by framing the work as a technical report rather than a research investigation:

"This enrichment is hypothesized to improve reasoning abilities of LLMs" (Section 1)

The word "hypothesized" is doing substantial work here. The paper presents a plausible mechanism—code and math data improve reasoning—but never tests it. Similarly, the finding that the 12T-token lower-quality dataset produced no significant improvement (Section 3.1) is reported without any benchmark numbers comparing a 7T and 12T model at the same scale, leaving the reader unable to assess the strength of this negative result.

The consequence. A practitioner reading this paper cannot determine which of Qwen2's many changes to replicate in their own model development. If an organization has a fixed budget for data collection, should they invest in higher-quality filtering, more code data, more math data, or broader multilingual coverage? The paper provides no guidance on the marginal return of each component. If the dominant factor was actually the architectural switch to GQA (which reduces memory pressure and may allow more effective training dynamics), or the long-context training phase (which exposes the model to different attention patterns), then a team that invests heavily in replicating the data pipeline while using a different architecture might see minimal gains.

More fundamentally, without causal attribution, the paper cannot predict whether Qwen2's approach will generalize to future scale increases. If the gains came primarily from data quality improvements that are now saturated (the 12T finding hints at this), then further data scaling will yield diminishing returns. If the gains came from architectural optimizations, then data scaling alone won't help and novel architectures will be needed. The paper provides no decomposition that could guide these predictions.

What evidence exists in the paper. The only implicit ablation is the comparison between Qwen2-0.5B (trained on 12T tokens) and Qwen2-1.5B (trained on 7T tokens) in Table 5. Both models share the same architecture family and presumably similar data quality, differing primarily in parameter count and training tokens. Qwen2-1.5B substantially outperforms Qwen2-0.5B on most benchmarks (MMLU: 56.5 vs. 45.4; GSM8K: 58.5 vs. 36.5), but this confounds parameter count with training tokens—the 1.5B has 4× the non-embedding parameters (1.2B vs. 0.3B, Table 5) while the 0.5B had 12T/7T ≈ 1.7× more training tokens. No comparison isolates the data effect.

The paper also provides before/after comparisons with Qwen1.5 across Tables 2–9, but as noted, Qwen1.5 differs on multiple axes simultaneously. The large jumps on specific benchmarks (MATH: 34.1→51.1 for 72B, +17.0; GSM8K: 79.5→89.5, +10.0; HumanEval: 46.3→64.6, +18.3) are consistent with the math/code data enrichment hypothesis but equally consistent with the architecture or training duration changes.

Mitigation status. The paper does not attempt to address this limitation. It presents the results as a package deal—"Qwen2 outperforms Qwen1.5"—without decomposing the sources of improvement. The authors suggest future work in Section 3.1 ("leaving further exploration for future model iterations") regarding the 12T vs. 7T finding, but this refers to data quantity scaling specifically, not to the broader attribution problem. Given the cost of training models at this scale, comprehensive ablation studies are likely infeasible for the authors, but this makes the causal claims in the paper's narrative unsupported.

6.2 Difficulty Estimation Is Computationally Prohibitive for Real-World Deployment

The assumption or constraint. The pre-training data pipeline's quality filtering relies on "the use of the Qwen models to filter out low-quality data" and employing Qwen models "to synthesize high-quality pre-training data" (Section 3.1). The data mixture optimization uses "experiments on scaled-down models to optimize the mixing of data from various sources and domains" (Section 3.1). The post-training data construction pipeline uses LLMs for ontology extraction (InsTag), instruction evolution (self-evolution strategy), rejection sampling (generating multiple reasoning paths and verifying correctness), execution feedback (generating solutions and test cases, compiling and executing), data repurposing (generating instructions from existing high-quality text), and constitutional feedback (generating aligned and deviating responses).

At no point does the paper estimate the compute cost of this data preparation relative to the actual pre-training and post-training model runs. Generating multiple candidate solutions per instruction for rejection sampling, compiling and executing code for execution feedback, running Qwen models as quality filters over web-scale corpora, and performing scaled-down proxy experiments to optimize data mixtures all consume substantial compute that is not accounted for in the reported "7 trillion token" pre-training budget.

The consequence. A practitioner attempting to replicate Qwen2's data pipeline faces unknown and potentially prohibitive upfront compute costs before training even begins. For rejection sampling on math data: if generating 16 reasoning paths per instruction and keeping only correct ones, the data generation cost alone could be 10–20× the token count of the final training data. For execution feedback on code: generating test cases, compiling, and executing adds CPU-bound computation that doesn't scale with GPU availability. For model-based quality filtering: running inference over a web-scale corpus to filter it could cost as much as a full training run if done naively.

This matters because the paper frames Qwen2 as "open-weight" and "accessible" (Section 1, abstract), but the process to create Qwen2 is not accessible to any organization without massive compute resources—not just for training, but for data preparation. The headline "7 trillion tokens" understates the total computational investment by omitting the data synthesis and filtering costs entirely.

Furthermore, the difficulty of estimating data quality and optimal mixture ratios creates a bootstrapping problem: to train a good quality-filtering model, you need a reasonably good model already. The paper uses Qwen models for filtering and synthesis, which assumes prior access to capable models. An organization starting from scratch cannot replicate this pipeline without first training capable models by some other means.

What evidence exists in the paper. The paper provides no estimate of data preparation compute costs anywhere. Section 4.1.2 describes the automated synthesis methods qualitatively but never quantifies how many GPU-hours were spent generating, filtering, and verifying the post-training data relative to the SFT and DPO training runs. The scaled-down data mixture experiments (Section 3.1) are mentioned but no details are given about the scale of the proxy models (how many parameters? how many tokens? how many mixture configurations tested?). The quality filtering via Qwen models (Section 3.1) is described in a single sentence with no information about model size, inference cost, or filtering ratio (what fraction of web data was discarded).

Mitigation status. The paper does not acknowledge this limitation. It is framed as an engineering implementation detail rather than a fundamental cost that affects the reproducibility and accessibility claims. No future work is suggested on reducing data preparation costs or on making the pipeline feasible for organizations with more limited compute.

6.3 Instruction-Following Degrades Sharply at Smaller Scales Despite Overall Gains

The assumption or constraint. The paper positions Qwen2 as a comprehensive suite where all model sizes benefit from the improved data and training methodology. The abstract states the release encompasses "a parameter range from 0.5 to 72 billion" and the smaller models are "designed for easy deployment on portable devices such as smartphones, earphones, and smart glasses" (Section 1). This implies that the small models are not merely research artifacts but are intended for practical deployment.

However, the instruction-following evaluation (IFEval, strict-prompt subset) reveals a catastrophic degradation at smaller scales that is not proportional to the general capability degradation. While Qwen2-7B-Instruct scores 54.7 on IFEval versus Llama-3-8B-Instruct's 72.1 (a 17.4-point deficit, Table 8), both Qwen2-1.5B-Instruct (29.0) and Qwen2-0.5B-Instruct (20.0) score far below even this already-weak level (Table 9). The paper explicitly acknowledges this for the 7B model:

"in terms of instruction following, Qwen2-7B-Instruct greatly falls behind the competitor. To address this limitation, we plan to augment the 7B model's instruction-following ability by enhancing the quality of post-training data" (Section 5.2.1)

The consequence. The instruction-following weakness makes the smaller Qwen2 models unfit for deployment in applications requiring reliable constraint satisfaction—precisely the kind of structured interaction (following formatting requirements, respecting length limits, adhering to specific output schemas) that is essential for on-device assistants, smart glasses interfaces, or earphone-based agents. A model that scores 20.0 on IFEval strict-prompt will fail to follow explicit instructions in roughly 4 out of 5 queries, making it unsuitable for any application where the user specifies concrete requirements.

This limitation is not merely an "it could be better" issue—it is a qualitative capability gap that undermines the deployment scenario the paper explicitly targets. The small models show strong improvements in raw knowledge and reasoning (Qwen2-0.5B jumps from 11.3 to 40.1 on GSM8K compared to Qwen1.5-0.5B, Table 9), suggesting they have become smarter but not more reliable at following directions. For on-device deployment where users issue specific commands ("set a timer for 10 minutes," "text Alice that I'll be late"), reliability at instruction following matters more than mathematical reasoning ability.

What evidence exists in the paper. Table 9 shows IFEval scores of 20.0 for Qwen2-0.5B-Instruct and 29.0 for Qwen2-1.5B-Instruct. Table 8 shows 54.7 for Qwen2-7B-Instruct versus 72.1 for Llama-3-8B-Instruct. The paper does not report IFEval results for Qwen2-57B-A14B-Instruct or Qwen2-72B-Instruct in the main comparison tables, though Table 7 shows 59.9 for the MoE model and Table 6 shows 77.6 for the 72B model (competitive with Llama-3-70B-Instruct's 77.3). The degradation is thus most severe at 7B and below—exactly the scales intended for on-device deployment.

The paper also reports in-house evaluations (Tables 10–11) that do not include an instruction-following-specific metric, making it impossible to determine whether the IFEval weakness reflects a general instruction-following problem or is specific to the IFEval format.

Mitigation status. The paper partially acknowledges this for the 7B model (quoted above) and proposes to "augment the 7B model's instruction-following ability by enhancing the quality of post-training data." However, no similar acknowledgment is made for the 0.5B and 1.5B models, despite their even more severe deficits. The proposed solution—better post-training data—assumes the weakness is in the data rather than in a fundamental capacity limitation at small scales (i.e., instruction following may require a certain minimum parameter count to simultaneously track multiple constraints while generating fluent text). This assumption is untested. No timeline or concrete plan for the improvement is provided.

6.4 English Performance Gap with Proprietary and Llama-3 Models Persists Despite Multilingual Strengths

The assumption or constraint. The paper emphasizes Qwen2's multilingual capabilities throughout, with approximately 30 languages supported (Section 3.1), comprehensive multilingual benchmarks (Tables 2–5, Multilingual rows), and a dedicated multilingual human evaluation (Table 13). The implication is that Qwen2 is a genuinely multilingual model that performs well across many languages, not merely an English model with weak additional-language support.

However, the in-house English evaluation (Table 11) reveals that Qwen2-72B-Instruct lags behind both proprietary models and Llama-3-70B-Instruct on English-language tasks, with the gap concentrated in comprehension and coding. The paper attributes this candidly:

"We assume both the amount of English tokens for pre-training and the quantity and diversity of data for post-training lead to the performance gap in English" (Section 5.2.2)

The consequence. For English-language applications—which still dominate the global deployment landscape for LLMs—Qwen2-72B-Instruct is not the best available open-weight model. Llama-3-70B-Instruct outperforms it on English comprehension (76.31 vs. 73.58 in Table 11) and coding (57.18 vs. 53.03), and GPT-4o outperforms it by larger margins on both (76.30 vs. 73.58 comprehension, 55.87 vs. 53.03 coding). On Arena-Hard, Qwen2-72B-Instruct's 48.1 versus Llama-3-70B-Instruct's 41.1 (Table 6) suggests an advantage, but Arena-Hard is an overall alignment metric, not an English-specific measure.

This creates a split recommendation for practitioners: if your application is primarily Chinese-language, Qwen2-72B-Instruct is arguably the best available model (Table 10: 69.58 average, ahead of GPT-4o's 68.87). If your application is multilingual across many languages, Qwen2 is competitive (Table 13: 3.93 vs. GPT-4-Turbo's 3.98). But if your application is primarily English, Llama-3-70B-Instruct is likely the better choice on several important dimensions, and GPT-4o remains substantially ahead. The paper's framing of "competitive performance relative to proprietary models" (abstract) obscures this language-dependent variability.

The English gap also suggests a fundamental tradeoff in multilingual pre-training: tokens spent on non-English languages are tokens not spent on English, and given finite total compute, a multilingual model may necessarily underperform an English-focused model of comparable size on English tasks. The paper does not explicitly discuss this tradeoff or provide guidance on how to navigate it—for instance, by quantifying how much English performance is sacrificed per unit of multilingual capability gained.

What evidence exists in the paper. Table 11 provides the direct English comparison: Qwen2-72B-Instruct at 72.94 average vs. Llama-3-70B-Instruct at 74.06 and GPT-4o at 76.11. The comprehension gap (73.58 vs. 76.31 vs. 76.30) is consistent. Table 2 shows Qwen2-72B base model trailing Llama-3-70B on some English benchmarks (HellaSwag: 87.6 vs. 88.0; ARC-C: 68.9 vs. 68.8—essentially tied). The multilingual benchmarks in Table 2 show Qwen2-72B leading on Multilingual Exam (76.6 vs. 70.0) and Multilingual Mathematics (76.0 vs. 67.1), confirming that the multilingual investment paid off for non-English tasks.

Mitigation status. The paper acknowledges the gap (Section 5.2.2, quoted above) and attributes it to English token quantity and post-training data diversity. However, it provides no analysis of whether this is an inherent tradeoff or a fixable data imbalance. It does not report the language distribution of the 7T pre-training tokens, so practitioners cannot assess whether simply increasing English data within the same total budget would close the gap (at the cost of multilingual performance) or whether the architecture/training recipe has an English-specific weakness that more data wouldn't fix. No future work is proposed to address this.

6.5 Contamination Analysis Reveals Widespread Test-Set Overlap in Code and Math Benchmarks Without Quantifying Impact on Score Validity

The assumption or constraint. The paper implements a rigorous contamination analysis (Section 5.2.6) and concludes that "the potential issue of data contamination does not significantly impact the model's performance" based on the small deltas between original and non-contaminated test set scores (Table 15). The underlying assumption is that if removing potentially contaminated samples doesn't change scores much, contamination is not inflating the reported benchmarks.

However, the contamination rates themselves are extraordinarily high under the strict 13-gram criterion: 75.0% of HumanEval, 37.7% of MultiPL-E, 31.7% of MATH, 29.6% of MBPP, 11.6% of MMLU-Pro, and 11.2% of MMLU samples are flagged (Table 15). The paper dismisses these as "primarily false positives, stemming from the mathematics and coding datasets" where "certain code snippets and mathematical equations are so common that they do not provide any meaningful advantage" (Section 5.2.6).

The consequence. The contamination analysis, while transparent in reporting the high overlap rates, does not actually validate that the overlap is benign. A 75% contamination rate on HumanEval under a 13-gram criterion means that three-quarters of the coding test problems share a 13-token subsequence with the training data. The paper argues these are common code patterns, but common code patterns are precisely what help models solve coding problems—a model that has memorized common patterns from training data has an advantage on test problems that use those same patterns, even if it hasn't seen the exact test problem. The distinction between "contamination" (unfair advantage from seeing the exact test problem) and "legitimate learning" (having seen similar patterns) becomes blurry when 13-gram matching flags 75% of test cases.

The MATH result is particularly concerning: Qwen2-72B-Instruct scores 69.0 on the original test set but 74.6 on the non-contaminated subset (a 5.6-point improvement). The paper interprets this as evidence that contaminated MATH samples were harder than average, not easier. But this interpretation raises questions it doesn't answer: why would test samples that share 13-grams with training data be harder? One possibility is that the decontamination during training removed training examples that would have helped with these harder test problems, meaning the model was artificially weakened on these problems—in which case the 69.0 score understates the model's true capability, and the contamination analysis is measuring the harm of overly aggressive decontamination rather than the benefit of data leakage. The paper does not explore this alternative.

More broadly, the contamination analysis measures only verbatim overlap (13-gram matching) and structural overlap (longest common subsequence). It does not measure semantic overlap—test problems that are paraphrases of training problems, or that test the same concepts using different notation. For coding and mathematics benchmarks, semantic overlap is likely far more prevalent than verbatim overlap, and it may provide a genuine training advantage that 13-gram matching cannot detect. A model trained on thousands of "sort a list" examples may solve a novel "sort a list" test problem not because it generalizes but because it has memorized the sorting pattern—and this is both legitimate learning (sorting is a general skill) and a form of contamination (the test problem is not truly novel). The paper's contamination framework cannot distinguish these cases.

What evidence exists in the paper. Table 15 provides the contamination percentages and deltas. The deltas are small for most benchmarks (within ±2 points), which is reassuring. But the paper provides no analysis of which specific test problems are flagged, what kinds of 13-gram overlaps they contain, or whether the overlaps represent problem-specific content (genuine contamination) versus generic patterns (false positives). The claim that they are "primarily false positives" is asserted based on qualitative inspection of "mathematics and coding datasets" but no quantitative breakdown (e.g., "30% of flagged HumanEval samples contain the exact problem description, 70% contain only generic code patterns like def function_name") is provided.

Mitigation status. The paper acknowledges the false positive issue ("we noticed that most of the identified contaminated samples are false positives") but does not provide a methodology for distinguishing false positives from genuine contamination, nor does it report benchmark scores adjusted for only genuine contamination. The two-stage decontamination (using LCS during data construction, n-gram only for post-hoc analysis) is methodologically sound, but the post-hoc analysis is presented as a sensitivity check rather than a rigorous investigation of whether reported scores are inflated. The paper does not propose future work on better contamination detection methods or on semantic overlap measurement.

6.6 Single-Annotator Multilingual Human Evaluation Undermines Reliability of Cross-Model Comparisons

The assumption or constraint. The multilingual human evaluation (Table 13) uses exactly one professional annotator per language:

"For the annotators, we invite one professional annotator for each language who majors in the language for the evaluation. For each test case, the annotator grades the response from model with a score from 1 to 5." (Section 5.2.4)

The underlying assumption is that a single professional annotator provides a reliable and unbiased estimate of model quality in each language, and that differences between models (e.g., Qwen2-72B-Instruct's 3.93 average vs. GPT-4o's 4.09) reflect genuine quality differences rather than annotator-specific rating tendencies.

The consequence. With a single annotator per language, there is zero measure of inter-annotator reliability. We cannot distinguish between:

A genuine 0.16-point quality difference between Qwen2-72B-Instruct (3.93) and GPT-4o (4.09)
The GPT-4o annotator being slightly more generous with high scores than the Qwen2 annotator (who is, presumably, the same person since they evaluate all models in their language—but we don't know this)
Random variation in a small sample of test cases

The 1–5 scale with 0.01 precision reporting (scores like 3.86, 4.15) implies a level of measurement precision that is completely unsupported by the annotation methodology. A difference of 0.22 between Qwen2-72B-Instruct (3.93) and Claude-3-Opus (4.15)—which the paper presents as meaningful—could easily be explained by a single annotator giving Claude-3-Opus one extra "4" instead of "3" on a handful of test cases.

This is not a minor methodological quibble; it is a fundamental validity problem for one of the paper's most distinctive claims—that Qwen2 is genuinely multilingual and competitive with proprietary models across many languages. The multilingual evaluation is more novel and potentially more impactful than standard English benchmarks (where many models compete), so the weakness of its methodology proportionally undermines one of the paper's strongest differentiating contributions.

The problem is compounded by the complete absence of information about the evaluation design: How many test cases per language? How were they constructed? What capabilities do they test? Are the test cases comparable in difficulty across languages? Without this information, even replicated multi-annotator scores would be difficult to interpret.

What evidence exists in the paper. Table 13 reports scores to two decimal places for five models across 10 languages, with averages computed across languages. The paper states only that "we design diverse test cases assessing different capabilities of large language models" with no further detail on test case count, design methodology, or capability coverage. The word "professional annotator" is not defined (professional translator? professional linguist? native speaker with domain expertise?). No inter-annotator agreement statistics are possible because there is only one annotator per language.

Mitigation status. The paper does not acknowledge this as a limitation. The evaluation is presented as a standard human evaluation without caveats about sample size, annotator reliability, or measurement precision. The reporting of scores to two decimal places implies a precision that the methodology cannot support. No future work is proposed to conduct multi-annotator evaluations or to establish the reliability of the multilingual assessment.

This limitation is particularly consequential because the multilingual evaluation results are prominently featured as evidence of Qwen2's competitiveness with proprietary models, and because the paper's abstract emphasizes "robust multilingual capabilities" as a distinguishing feature. A rigorous multilingual evaluation would require multiple annotators per language, inter-annotator agreement statistics, clearly defined evaluation criteria, and transparency about test case construction—none of which is provided.

7. Implications and Future Directions

How This Work Changes the Landscape

The Qwen2 technical report does not introduce a paradigm shift or a novel algorithm. It is, instead, an engineering demonstration of sufficiency: it shows that a combination of well-understood architectural choices, aggressive data curation, and transparent post-training methodology can produce a model family that competes with—and in specific domains exceeds—the strongest open-weight and proprietary systems available at release time. The conceptual shift is subtle but real: the paper reframes LLM development as primarily a data engineering discipline rather than an architectural innovation race.

This matters because the field has been in an ambiguous state regarding what drives progress. On one side, architectural innovations (mixture-of-experts, state-space models, novel attention mechanisms) have received disproportionate research attention and publication prestige. On the other, the empirical record increasingly suggests that data quality and composition explain a large fraction of the variance between models of similar scale. Qwen2 provides a large-scale, multi-model data point in favor of the data-centric view: the architectural changes from Qwen1.5 to Qwen2 are incremental and well-credited (GQA from Ainslie et al., 2023; DCA from An et al., 2024; YARN from Peng et al., 2023), yet the performance jumps are substantial—+17.0 points on MATH for the 72B model, +18.3 on HumanEval, +10.0 on GSM8K (Table 2). The paper's own explanation attributes these gains primarily to "enhancing the quantity and quality of code and mathematics content" in the 7-trillion-token pre-training dataset (Section 1).

The paper also resolves a latent tension between multilingual breadth and per-language depth. Prior open-weight models typically optimized for English and treated multilingual performance as an afterthought, leading to sharp cliffs on non-English benchmarks (Llama-3-70B scoring 65.2 on C-Eval versus Qwen2-72B's 91.0, Table 2). The paper demonstrates that a genuinely multilingual pre-training corpus—spanning approximately 30 languages with domain coverage in code and mathematics across those languages—produces a model that is simultaneously competitive on English benchmarks (72.94 in-house average, near Llama-3-70B-Instruct's 74.06, Table 11) and dominant on Chinese (69.58 in-house average, ahead of GPT-4o's 68.87, Table 10) while also performing strongly on multilingual exams, understanding, and mathematics (Table 2, Multilingual rows). This suggests that multilingual pre-training is not a zero-sum tradeoff where English performance must be sacrificed—with sufficient total data volume and careful composition, the model can be strong in many languages simultaneously. The English gap that does persist (Table 11) is modest and may close with further data scaling rather than requiring architectural changes.

The contamination analysis (Section 5.2.6, Table 15) sets a new standard for transparency in model releases that other teams will be expected to meet. The two-stage methodology (LCS+ngram for removal during construction; ngram-only for post-hoc assessment), the explicit reporting of contamination rates per benchmark, and the computation of performance deltas on non-contaminated subsets collectively raise the bar for what constitutes adequate evaluation hygiene. The counterintuitive finding that MATH scores improve by 5.6 points on the non-contaminated subset for Qwen2-72B-Instruct (69.0 → 74.6) is particularly valuable: it demonstrates that contamination analysis can reveal not just inflated scores (the expected worry) but also deflated scores caused by overly aggressive decontamination removing useful training data. This finding complicates the standard narrative that "data contamination = inflated benchmarks" and suggests that the relationship between training data overlap and test performance is more nuanced than the field has assumed.

The paper also shifts the perceived viability of Mixture-of-Experts for production deployment. Prior MoE releases, particularly Mixtral-8x7B (Jiang et al., 2024), demonstrated the efficiency potential of sparse architectures but showed benchmark gaps relative to similarly-sized dense models that limited enthusiasm. Qwen2-57B-A14B, by matching or exceeding ~30B dense model performance with only 14B activated parameters (Table 3: 76.5 MMLU vs. Yi-1.5-34B's 77.1; 80.7 GSM8K vs. Yi-1.5-34B's 82.7; 53.0 HumanEval vs. 46.3), demonstrates that the MoE efficiency tradeoff can be substantially reduced through careful design choices: fine-grained experts with shared routing, diversity-promoting upcycling initialization, and sufficient additional pre-training (4.5T tokens beyond the base 7T). The specific initialization method—shuffling intermediate dimensions and randomly reinitializing 50% of each expert's parameters—is a generalizable technique that other teams can adopt when converting dense checkpoints to MoE architectures.

Research directions that become more attractive after this work:

Data engineering at scale: automated quality filtering, synthetic data generation for under-represented domains, and optimal data mixture determination become first-class research problems rather than pre-processing details.
Rigorous contamination methodology: the paper's multi-stage analysis with explicit delta reporting provides a template that can be standardized across model releases, enabling meaningful cross-model comparisons of contamination impact.
Multilingual pre-training without English sacrifice: the paper provides existence proof that strong multilingual and English performance can coexist, opening questions about the optimal language mixture, the role of code data as a cross-lingual bridge, and whether certain language families benefit disproportionately from shared training.
Efficient MoE for deployment: the 2:1 activated parameter efficiency demonstrated by Qwen2-57B-A14B makes MoE architectures economically compelling for latency-tolerant serving scenarios, encouraging further work on expert load balancing, routing stability, and initialization methods.

Research directions that become less urgent as a result of this work:

Novel attention mechanisms for the sake of novelty: the paper's strong results with standard GQA and causal attention suggest that architectural innovation, while valuable, is not the bottleneck for current-generation models—data and scale are.
Pure scale without data curation: the paper's 12T-vs-7T finding (Section 3.1) suggests that simply adding more data without quality control may yield diminishing returns, redirecting effort from data acquisition to data refinement.

Follow-Up Research This Work Enables

Quantifying the marginal contribution of code and mathematics pre-training data to reasoning benchmarks. The paper hypothesizes that increased code and math data "improves reasoning abilities" (Section 1) and shows large gains on GSM8K (+10.0 for 72B, Table 2) and MATH (+17.0), but never isolates these data components. A strong follow-up would train a set of 7B models from scratch on the Qwen2 data pipeline with controlled variations: (a) full 7T dataset including code and math expansion, (b) the same dataset with code data held out during pre-training but included in post-training SFT, (c) the same with math data held out, (d) the original Qwen1.5 3T dataset with Qwen2 architecture. The key measurements would be GSM8K, MATH, HumanEval, and BBH scores for each variant. This would disentangle whether the gains come from code data, math data, general data quality, or post-training, and would inform data investment decisions for future models.

Understanding the failure mode of instruction following at sub-70B scales. The IFEval scores in Tables 8 and 9 reveal a sharp degradation: Qwen2-72B-Instruct scores 77.6 (competitive with Llama-3-70B-Instruct's 77.3), but Qwen2-7B-Instruct scores only 54.7 (versus Llama-3-8B-Instruct's 72.1), and Qwen2-1.5B-Instruct drops to 29.0. The paper attributes this to insufficient post-training data quality for the 7B model (Section 5.2.1) but does not investigate whether instruction following has a minimum capacity threshold below which no amount of data can compensate. A targeted study would fine-tune Qwen2 models of varying sizes (0.5B through 7B) on identical post-training datasets scaled in size, measuring IFEval and a custom fine-grained constraint-satisfaction benchmark (e.g., "write a 100-word paragraph, then summarize it in exactly 3 bullet points, each starting with a verb"). If even large post-training budgets fail to close the gap with larger models, this would establish a fundamental scaling law for instruction following distinct from general language understanding—a negative result with practical implications for on-device deployment claims like the paper's "smartphones, earphones, and smart glasses" target (Section 1).

Validating the MoE initialization method across base model scales and expert counts. The paper's upcycling procedure—replicating dense FFN weights, shuffling along the intermediate dimension, and randomly reinitializing 50% of each expert (Section 2.2.2)—is described without ablation on the key hyperparameter: the reinitialization fraction. A systematic study would start from a dense Qwen2 checkpoint at multiple scales (1.5B, 7B) and create MoE variants with reinitialization fractions of 0%, 25%, 50%, 75%, and 100%, each trained for a fixed additional budget (e.g., 500B tokens). The key metrics would be downstream benchmark performance and expert utilization entropy (measuring whether the router collapses to a few experts or spreads tokens evenly). This would establish whether 50% is near-optimal, whether the method transfers across scales, and whether the shuffling step alone (0% reinitialization) provides sufficient diversity. A negative result—showing that performance is insensitive to the reinitialization fraction—would simplify future MoE upcycling by eliminating the need to tune this parameter. A positive result—showing a sharp optimum at a specific fraction—would establish a scaling relationship between expert count, model size, and optimal randomization.

Stress-testing the contamination analysis methodology with known-contaminated and known-clean data. The paper's two-stage decontamination approach (LCS+ngram for removal, ngram-only for assessment) is presented as a sensitivity check, but the false positive rate and detection sensitivity are never calibrated. A methodological study would construct synthetic pre-training datasets with known contamination injection: insert exact copies of a subset of test problems (at varying rates: 0.1%, 1%, 10%) into the training data, then apply Qwen2's contamination detection pipeline to measure (a) what fraction of injected contamination is detected (recall), (b) what fraction of flagged samples are genuine contamination vs. false positives (precision), and (c) how benchmark scores inflate as a function of true contamination rate. This would provide the field with a calibrated understanding of what Qwen2's reported contamination rates (e.g., 75.0% of HumanEval flagged, Table 15) actually mean for score validity, and would establish whether the 13-gram criterion is appropriately stringent or needs adjustment based on benchmark domain (code vs. natural language vs. math).

Measuring the alignment tax of DPO with online merging across capability domains. The paper applies DPO with an Online Merging Optimizer "to mitigate the alignment tax" (Section 4.3) but provides no measurements of how much tax exists or how much the optimizer mitigates it. A controlled experiment would take the Qwen2-7B base model and compare four post-training variants: (a) SFT only, (b) SFT + offline DPO, (c) SFT + offline DPO + online DPO without merging, (d) SFT + offline DPO + online DPO with Online Merging Optimizer. For each variant, measure both alignment metrics (MT-Bench, IFEval, Arena-Hard) and capability retention metrics (MMLU, GSM8K, HumanEval, BBH). The difference in capability between (a) and (b) measures the offline DPO alignment tax. The difference between (b) and (c) measures the online DPO tax. The difference between (c) and (d) measures the Online Merging Optimizer's recovery. This would quantify the alignment-capability tradeoff for the specific DPO implementation used in Qwen2 and would inform practitioners about whether the Online Merging Optimizer is necessary or merely helpful for their deployment scenario.

Replication on a non-Asian-centric language family to test the multilingual pre-training hypothesis. The paper's multilingual strengths are most pronounced in Chinese and East Asian languages (Table 2: Qwen2-72B achieves 91.0 on C-Eval and 90.1 on CMMLU vs. Llama-3-70B's 65.2 and 67.2). The human evaluation (Table 13) covers a broader set but still shows variable performance: strong in Russian (4.15) and Korean (4.14), weaker in Japanese (3.63) and Thai (3.75). A critical follow-up would evaluate Qwen2-72B-Instruct on a controlled set of African and South Asian languages (Swahili, Amharic, Hindi, Bengali, Tamil) that were likely underrepresented in even the expanded multilingual pre-training data, comparing against models that explicitly target those languages. If Qwen2 performs competitively on these under-resourced languages, it would suggest that the multilingual pre-training transfers across language families—a strong result supporting "teach the model many languages and it learns to learn languages." If performance collapses, it would establish that Qwen2's multilingualism is broad but not deep, limited to the ~30 languages that received substantial token allocation in the 7T dataset, and would motivate targeted data collection for genuinely low-resource languages.

Practical Applications and Downstream Use Cases

On-device deployment for Chinese-language applications. Qwen2-1.5B-Instruct achieves 52.4 on MMLU, 61.6 on GSM8K, and 47.0 on HumanEval (Table 9) with only 1.2B non-embedding parameters—a scale that can run in 16-bit precision on smartphones with 4–6GB of available memory. The in-house Chinese evaluation (Table 10) shows Qwen2-1.5B-Instruct scoring 36.11 average across six capability categories, representing a 4.21-point improvement over Qwen1.5-1.8B-Chat despite having 33% fewer non-embedding parameters (1.2B vs. 1.8B, Table 5). For a Chinese-language smartphone assistant handling tasks like message summarization, scheduling, and basic math, this performance level is operationally useful. The practical deployment architecture would run the 1.5B model on-device for latency-sensitive queries and route complex reasoning tasks to a cloud-hosted Qwen2-72B-Instruct, with the on-device model handling perhaps 80% of queries at sub-100ms latency. The key metric to validate before deployment would be real-world instruction-following reliability: IFEval at 29.0 for the 1.5B model (Table 9) suggests that constraint-heavy queries (e.g., "summarize this email in exactly three sentences") will fail frequently, so the query router would need to classify whether a request requires precise constraint satisfaction and escalate accordingly.

Cost-efficient batch inference for multilingual document processing. Organizations processing large volumes of documents across English, Chinese, Arabic, Russian, and other languages currently face a choice between running multiple language-specific models (increasing operational complexity) or using a single proprietary multilingual API (increasing per-token cost and introducing data privacy concerns). Qwen2-72B-Instruct's multilingual performance—averaging 3.93/5.0 across 10 languages in human evaluation (Table 13) and scoring 76.6 on Multilingual Exam benchmarks (Table 2)—makes it a viable single-model solution for many multilingual processing pipelines. For a use case like "extract key entities and summarize 500,000 customer support tickets across 15 languages," running Qwen2-72B-Instruct on owned infrastructure eliminates per-token API charges and keeps data within organizational boundaries. The multilingual Mathematics score of 76.0 (Table 2) further suggests that numerical reasoning tasks embedded in multilingual documents (invoice parsing, financial report extraction) will not degrade significantly when switching languages. The practical caveat is that languages not among the approximately 30 explicitly supported may see quality degradation proportional to their representation in the 7T pre-training data, so a pilot evaluation on the organization's specific language distribution would be essential.

Base model for domain-specific fine-tuning with Chinese+English bilingual requirements. Many enterprise applications in Asia-Pacific markets require strong performance in both English and Chinese simultaneously: legal document review across jurisdictions, medical literature analysis, customer-facing chatbots for multinational brands, and technical documentation translation all demand bilingual capability that English-only models (Llama-3, Mistral) and Chinese-only models each fail to provide. Qwen2-7B (Table 4) achieves 83.2 on C-Eval and 83.9 on CMMLU for Chinese while maintaining 70.3 on MMLU and 79.9 on GSM8K for English/math reasoning—a bilingual profile that neither Llama-3-8B (49.5 C-Eval, 66.6 MMLU) nor Chinese-specific models can match. The practical workflow would start from Qwen2-7B and fine-tune on a domain-specific bilingual instruction dataset (e.g., 10,000 Chinese+English medical QA pairs), using LoRA for parameter efficiency. The expected outcome is a 7B-scale model deployable on a single A100-40GB that handles domain-specific queries in both languages without requiring separate English and Chinese fine-tuned variants, reducing model maintenance costs by roughly half compared to a two-model strategy. The key risk to validate before committing is whether the bilingual pre-training advantage survives heavy domain fine-tuning in a single language, which could catastrophically degrade the non-target language's performance—a phenomenon the paper provides no evidence about.

Safety-benchmarked model for applications requiring low harmful-output rates. Table 14 shows Qwen2-72B-Instruct achieving 0.00% harmful responses on Illegal prompts and 2.41% on Fraud, competitive with or better than GPT-4 (0.00% and 3.40% respectively). For applications where generating harmful content carries regulatory or reputational risk—educational tools for minors, public-facing government chatbots, healthcare information systems—these safety scores, combined with the open-weight availability (enabling additional safety fine-tuning and red-teaming on an organization's specific threat model), make Qwen2-72B-Instruct an attractive base. The practical approach would start from Qwen2-72B-Instruct and conduct domain-specific red-teaming (e.g., medical misinformation prompts for a healthcare application), collect the harmful outputs and their safe refusals, and apply additional DPO to further reduce the harmful-output rate in the target domain. The 22.91% harmful rate on Pornography (Table 14) indicates that this category requires the most attention during additional safety fine-tuning, consistent with the paper's note that this is "a conventionally difficult category to differentiate even for humans" (Section 5.2.5). The open-weight nature also permits deployment in air-gapped environments where proprietary API-based safety filters are unavailable.