Qwen3 Technical Report

ArXiv: 2505.09388

🎯 Pitch

One model, two modes—Qwen3 unifies rapid chat and deep reasoning in the same weights, switching on command or by budget, without needing separate models. Even the smallest 0.6B version, trained with 10x less compute via distillation, beats its much larger Qwen2.5 predecessors, showing that thinking depth can be decoupled from model size.

1. Executive Summary

Qwen3 introduces a family of dense and mixture-of-experts LLMs scaling from 0.6B to 235B parameters that unifies two distinct operating modes—thinking mode (for complex multi-step reasoning) and non-thinking mode (for rapid, context-driven responses)—into a single model, eliminating the need to switch between separate chat and reasoning models. The series also introduces a thinking budget mechanism, giving users fine-grained control over the number of reasoning tokens allocated per query (extending or truncating the chain-of-thought process at user-defined thresholds). The flagship Qwen3-235B-A22B achieves 85.7 on AIME'24 and 70.7 on LiveCodeBench v5 in thinking mode, while even the smallest dense models—trained via strong-to-weak distillation from larger teachers requiring only 1/10 the GPU hours of full-stack post-training—outperform prior Qwen2.5 models at larger parameter scales, establishing that test-time reasoning depth can be traded off against model size only when the underlying capability has been imparted through pretraining on 36 trillion tokens across 119 languages.

2. Context and Motivation

The Core Problem: Managing Dual Reasoning Modes in a Single Model

The central problem Qwen3 addresses is a fragmentation in how modern LLMs handle different types of queries. In the current ecosystem, users and developers face a forced choice: they must deploy separate models for different reasoning requirements. A non-thinking model (like GPT-4o, Claude 3.7 Sonnet, or Qwen2.5-72B-Instruct) generates rapid, context-driven responses suitable for straightforward questions, translation, summarization, and creative writing. A thinking model (like OpenAI-o1, DeepSeek-R1, or QwQ-32B) engages in extended chain-of-thought reasoning, producing step-by-step deliberation before answering—essential for competition-level mathematics, complex coding, and multi-step logical reasoning.

This dual-model paradigm creates several practical friction points. First, deployment complexity multiplies: organizations must maintain, serve, and route between two separate model instances. Second, cost and latency management becomes ad-hoc—there is no principled way to decide when to invoke expensive reasoning versus cheap direct answering on a per-query basis. Third, users must develop intuition about which model to use for which task, and getting it wrong means either wasted compute (using reasoning mode for simple queries) or degraded performance (using non-reasoning mode for complex ones).

Qwen3's core architectural contribution is mode unification: the same model weights serve both thinking and non-thinking behavior, switchable via simple flags (/think and /no think) embedded in the chat template. This is not merely a convenience feature. It implies that the model has learned to dynamically modulate its own reasoning depth based on the user's explicit signal, rather than requiring two independently trained parameter sets. The technical achievement is non-trivial because thinking and non-thinking capabilities are typically in tension—training for extended chain-of-thought can degrade the model's ability to give concise, direct answers, while optimizing for rapid responses can erode the disciplined step-by-step reasoning that mathematical and coding tasks demand.

The Thinking Budget: Bridging Static Modes

Beyond binary mode switching, Qwen3 introduces thinking budget control—the ability to specify (and interrupt) the number of tokens allocated to the reasoning process. This addresses a deeper problem: reasoning difficulty is continuous, not binary. Some math problems require only a few reasoning steps; others demand extended algebraic manipulation. Some coding tasks need brief planning; others require systematic algorithm design. A binary "think/don't think" toggle treats all reasoning tasks identically, allocating the same maximum-thinking-budget to a simple algebra problem as to an AIME-level competition question.

The thinking budget mechanism allows users to set a threshold on reasoning tokens. When the model's chain-of-thought reaches this threshold, the system manually inserts a stop-thinking instruction:

"Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n response.\n\n"

The model then generates its final answer based on the partial reasoning accumulated up to that point. Crucially, the paper reports that this ability emerges naturally from the thinking mode fusion training stage, without explicit training on interrupted reasoning trajectories. Figure 2 demonstrates that performance scales smoothly with increased thinking budget across mathematics, coding, and STEM benchmarks, with no saturation at the tested budget levels (the paper suggests further extension beyond 32K output tokens may yield additional gains).

This has direct practical implications. In production deployments where latency and cost vary by user or query priority, a thinking budget allows graduated resource allocation: allocate 512 tokens of reasoning for routine queries, 2048 for moderate ones, and 32K+ for the hardest problems. This is fundamentally different from the prior paradigm where reasoning models operated with a fixed, often maximal, thinking effort.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in the existing landscape:

1. Separate reasoning models fragment the deployment pipeline. Prior to Qwen3, state-of-the-art open-source reasoning was delivered through dedicated models trained via reinforcement learning on chain-of-thought trajectories: DeepSeek-R1 (Guo et al., 2025), QwQ-32B (Qwen Team, 2025), and the DeepSeek-R1-Distill series. These models excel at reasoning but are unidirectional—they always engage in extended thinking, even for queries where it is unnecessary or harmful (e.g., simple factual questions, translation, creative writing). Conversely, general-purpose chat models (Qwen2.5-72B-Instruct, Llama-4-Maverick, DeepSeek-V3) produce rapid responses but lack the disciplined multi-step reasoning needed for competition math and complex coding. Users deploying both types of tasks must maintain two separate model serving pipelines.

2. Chain-of-thought reasoning is computationally expensive and non-adaptive. Models like o1, DeepSeek-R1, and QwQ-32B apply a fixed reasoning strategy regardless of problem difficulty. For a query that could be answered correctly in 200 tokens of reasoning, they might still generate 20,000 tokens of deliberation—wasting compute and increasing latency. The thinking budget mechanism directly addresses this by making reasoning depth a user-controllable knob, but prior work offered no such mechanism in a unified model.

3. Strong-to-weak distillation for reasoning is underexplored. Prior work on reasoning distillation (e.g., the DeepSeek-R1-Distill series) typically used supervised fine-tuning on teacher-generated reasoning traces. The Qwen3 paper argues—and demonstrates empirically in Table 21—that on-policy distillation (where the student model generates its own trajectories and aligns them to teacher logits via KL divergence minimization) substantially outperforms both off-policy SFT distillation and direct reinforcement learning from the same starting checkpoint. On AIME'24, on-policy distillation achieves 74.4 versus 67.6 for RL and 55.0 for off-policy distillation, while requiring only 1,800 GPU hours versus 17,920 for RL. The performance gap on pass@64 metrics (93.3 vs. 90.0 on AIME'24) suggests that distillation from teacher logits expands the student's exploration space in ways RL does not. This finding challenges the assumption that RL is the most effective way to instill reasoning capabilities, at least at smaller model scales.

4. Multilingual reasoning benchmarks are sparse. Prior model evaluations heavily centered on English-language mathematics and coding (MATH, GSM8K, HumanEval, LiveCodeBench). Qwen3 introduces or leverages several multilingual benchmarks that specifically test reasoning across languages: MT-AIME2024 (AIME problems translated into 55 languages), PolyMath (mathematical reasoning in 18 languages), MLogiQA (logical reasoning in 10 languages), and Multi-IF (instruction following in 8 languages). This expanded evaluation suite reveals that cross-lingual reasoning transfer is non-trivial—models can show strong English reasoning (e.g., 80.8 on MT-AIME2024 for English) while struggling on lower-resource languages (e.g., 7.8 for the same model on certain languages in Table 19). The paper's extensive per-language breakdowns (Tables 24–35) provide a granular view of where reasoning capabilities do and do not transfer across the 119 supported languages.

5. Training efficiency at scale remains a bottleneck. The four-stage post-training pipeline (Long-CoT cold start → Reasoning RL → Thinking Mode Fusion → General RL) applied to the flagship 235B model is computationally intensive. For smaller models, replicating this pipeline would be prohibitively expensive. The paper directly addresses this with strong-to-weak distillation, demonstrating that lightweight models can inherit both thinking and non-thinking capabilities from larger teachers at roughly 1/10 the training cost. This efficiency finding is significant for the open-source community, where computational constraints often prevent comprehensive post-training of reasoning capabilities on smaller models.

How This Paper Positions Itself

Qwen3 positions itself at the intersection of several active research directions:

As a mode-unified model, it competes with reasoning-specialized models (o1, DeepSeek-R1, Grok-3-Beta Think) in thinking benchmarks while simultaneously competing with general-purpose chat models (GPT-4o, DeepSeek-V3, Llama-4-Maverick) in non-thinking benchmarks. The paper's results in Tables 11 and 12 claim state-of-the-art open-source performance in both modes, with the thinking mode matching or exceeding DeepSeek-R1 on 17 of 23 benchmarks and the non-thinking mode exceeding DeepSeek-V3 and GPT-4o-2024-11-20 on 18 of 23 benchmarks—from a single model with fewer total and activated parameters.

As a thinking budget pioneer, it establishes that reasoning depth can be continuously modulated without explicit training on interrupted trajectories. This is a novel capability not present in o1, DeepSeek-R1, or QwQ-32B, which always complete their thinking process once initiated. The emergence of this capability from mode fusion training (Section 4.3) is presented as an empirical finding rather than a designed feature, opening questions about what other capabilities might emerge from multi-mode training that have not been explicitly programmed.

As a distillation-first approach to reasoning, it challenges the dominant paradigm of applying RL directly to smaller models. The comparison in Table 21 is stark: RL on an 8B model achieves 67.6 on AIME'24 using 17,920 GPU hours, while on-policy distillation from a larger teacher achieves 74.4 using 1,800 GPU hours. This suggests that for models below a certain capability threshold, learning to mimic a stronger model's reasoning distribution is more effective than exploring the reasoning space through RL—at least within the training budget constraints tested.

As a multilingual scaling effort, it extends from Qwen2.5's 29 languages to 119 languages and dialects, tripling linguistic coverage. The pretraining data includes 36 trillion tokens, double Qwen2.5's corpus. The paper claims this expansion was achieved partly through automated data generation—using Qwen2.5-VL for PDF text extraction and Qwen2.5-Math/Coder for synthetic data generation—rather than purely through web-crawled data expansion, which represents a methodological contribution to scalable multilingual pretraining.

Relative to the Qwen lineage specifically, Qwen3 represents a leap in both capability and scope from Qwen2.5. Qwen3-32B-Base outperforms Qwen2.5-72B-Base on 10 of 15 benchmarks (Table 4) despite having less than half the parameters. Qwen3-30B-A3B, with only 3B activated parameters, achieves comparable performance to Qwen2.5-32B-Base with 32B activated parameters (Table 5)—roughly a 10× efficiency improvement in activated parameters on key benchmarks. The dense models show a consistent pattern: Qwen3-1.7B/4B/8B/14B/32B-Base achieve comparable performance to Qwen2.5-3B/7B/14B/32B/72B-Base respectively, representing an effective parameter efficiency gain of roughly 2× across the dense model range.

3. Technical Approach

3.1 Reader Orientation

Qwen3 is a family of transformer-based language models ranging from 0.6B to 235B parameters that integrates two operating modes—a deliberate, step-by-step thinking mode and a rapid, context-driven non-thinking mode—into a single unified model, with the additional ability to control the depth of reasoning through a user-specified thinking budget. The system solves the problem of deploying and managing separate reasoning and chat models by training a single set of weights that can dynamically switch between reasoning behaviors based on explicit user flags in the chat template, while also enabling graduated allocation of inference-time compute proportional to query difficulty. The solution is shaped as a multi-stage post-training pipeline applied to a 36-trillion-token, 119-language pretrained base model, followed by strong-to-weak distillation to efficiently transfer both modes to smaller models without replicating the full training cost.

3.2 Big-Picture Architecture (Diagram in Words)

The Qwen3 system comprises six major components arranged in a sequential pipeline:

1. Pre-trained Base Models — Dense or MoE transformer architectures (Tables 1–2) trained on 36T tokens across 119 languages through a three-stage pretraining process. These models have no thinking/non-thinking distinction yet; they are foundation models producing raw next-token completions.

2. Long-CoT Cold Start Dataset — A curated collection of verifiable reasoning problems (math, code, logic, STEM) paired with reference answers, filtered through a two-phase quality-control process. This dataset is used only to establish initial reasoning patterns without saturating the model's capacity.

3. Reasoning RL System — A GRPO-based reinforcement learning loop operating on 3,995 query-verifier pairs, using large batch sizes and high rollouts-per-query to train the cold-start model to generate extended chain-of-thought reasoning that leads to correct final answers.

4. Thinking Mode Fusion Training — Continual supervised fine-tuning on a combined dataset of thinking-mode responses (generated by the RL-trained model via rejection sampling) and non-thinking-mode responses (curated across diverse tasks). A custom chat template with /think and /no think flags teaches mode discrimination.

5. General RL System — A multi-task reinforcement learning stage with over 20 distinct reward functions (rule-based, model-based with reference, model-based without reference) covering instruction following, format adherence, preference alignment, agent tool use, and specialized scenarios like RAG.

6. Strong-to-Weak Distillation Pipeline — A two-phase process (off-policy then on-policy) that transfers both thinking and non-thinking capabilities from the fully-trained flagship models (235B or 32B) to smaller student models (30B-A3B, 14B, 8B, 4B, 1.7B, 0.6B) by minimizing KL divergence between student and teacher logits on student-generated trajectories.

Information flows as follows: the base model enters post-training → Stage 1 Long-CoT Cold Start instills initial reasoning patterns → Stage 2 Reasoning RL optimizes chain-of-thought quality on verified problems → Stage 3 Thinking Mode Fusion adds non-thinking capabilities and teaches mode switching via chat templates → Stage 4 General RL broadens capabilities across 20+ task types → finished flagship model. For smaller models: the flagship model serves as teacher → off-policy distillation on teacher outputs → on-policy distillation aligning student logits to teacher logits → finished lightweight model with both modes.

3.3 Roadmap for the Deep Dive

• First, the model architecture (both dense and MoE variants), including specific architectural choices like QK-Norm, removed QKV-bias, fine-grained expert segmentation, and global-batch load balancing—because all downstream capabilities rest on this foundation.
• Second, the three-stage pretraining process and data construction methodology—because the quality of the base model determines the ceiling for post-training improvements, and the paper's scaling claims (e.g., Qwen3-32B outperforming Qwen2.5-72B) depend critically on pretraining data and strategy.
• Third, the four-stage post-training pipeline in chronological order (Cold Start → Reasoning RL → Thinking Mode Fusion → General RL)—because this is where the thinking/non-thinking unification and thinking budget capability are actually built, and each stage depends on the outputs of the previous one.
• Fourth, the strong-to-weak distillation approach, including off-policy and on-policy distillation—because this is how capabilities transfer to smaller models, and the efficiency comparison with RL depends on understanding what each distillation phase does.
• Fifth, the thinking budget mechanism and its emergent properties—because this is the novel inference-time capability that distinguishes Qwen3 from prior reasoning models, and understanding it requires knowing what the mode fusion stage produces.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a system-building paper whose core contribution is the training methodology and architecture that produce a single model capable of both extended chain-of-thought reasoning and rapid direct response, switchable via chat template flags, with controllable reasoning depth.

Dense Model Architecture

The Qwen3 dense models (0.6B, 1.7B, 4B, 8B, 14B, 32B parameters) share a common architecture derived from Qwen2.5 but with specific modifications to improve training stability and downstream performance. The architecture is a standard decoder-only transformer with several well-motivated component choices.

Core components. Every dense model uses Grouped Query Attention (GQA, from Ainslie et al., 2023) as the attention mechanism. GQA reduces the number of key-value heads relative to query heads, which saves memory during inference by reducing the size of the KV-cache while preserving most of the model's attention quality. The specific head configurations are shown in Table 1: the 0.6B and 1.7B models use 16 query heads and 8 key-value heads; the 4B and 8B use 32 query heads and 8 KV heads; the 14B uses 40 query heads and 8 KV heads; the 32B uses 64 query heads and 8 KV heads. The ratio of query-to-KV heads ranges from 2:1 (smallest models) to 8:1 (largest models), reflecting a design choice where larger models benefit more from KV-cache compression.

The feed-forward layers use SwiGLU activation (Dauphin et al., 2017), which is a gated variant of the Swish activation combined with a GLU-style gating mechanism. Specifically, SwiGLU computes $\text{SwiGLU}(x) = \text{Swish}(xW_1) \odot (xW_2)$ where $\odot$ is element-wise multiplication, $\text{Swish}(x) = x \cdot \sigma(x)$, and $\sigma$ is the sigmoid function. The gating allows the network to learn which dimensions of the intermediate representation to pass through and which to suppress, providing the model with more expressive power than a simple ReLU or GELU activation at comparable computational cost.

Positional information is encoded using Rotary Positional Embeddings (RoPE, from Su et al., 2024). RoPE applies a rotation matrix to the query and key vectors before the attention dot product, where the rotation angle depends on the token's absolute position in the sequence. The key property is that the dot product between two rotated vectors depends only on their relative position, not their absolute positions—this gives the model translation invariance in its attention patterns while preserving the ability to attend based on distance. For the long-context stage of pretraining, the base RoPE frequency is increased from 10,000 to 1,000,000 using the ABF technique (Xiong et al., 2023) to support longer sequences without retraining from scratch.

Normalization follows the Pre-RMSNorm convention (Jiang et al., 2023), meaning the RMSNorm operation is applied before each sub-layer (attention and feed-forward) rather than after. Pre-normalization improves training stability by preventing the gradient from exploding through the residual connections. The norm operations use learned scale parameters (the "gain" in RMSNorm) but no learned bias, consistent with the general principle that biases in normalization layers are redundant when followed by affine transformations.

Changes from Qwen2/Qwen2.5. Two architectural modifications distinguish Qwen3 from its predecessors. First, QKV-bias—which was present in Qwen2 to provide an absolute position signal before RoPE rotation—has been removed entirely. The motivation is that RoPE already encodes positional information effectively, and the bias terms add parameters without clear benefit for models at this scale. The removal slightly reduces parameter count and simplifies the architecture.

Second, and more significantly, QK-Norm (from Dehghani et al., 2023) has been introduced to the attention mechanism. QK-Norm applies Layer Normalization to the query and key vectors after the linear projection but before the dot-product attention computation. This prevents the magnitude of the query-key dot products from growing with model depth or token position, which can cause attention weights to collapse to one-hot vectors (a failure mode where the model attends exclusively to a single token and ignores all context). The paper states this was introduced "to ensure stable training for Qwen3," which is particularly important given the 36-trillion-token training corpus and the large model sizes—without QK-Norm, attention logit variance tends to increase during long training runs, eventually destabilizing the softmax.

Context length and layer counts. Table 1 reports the layer counts and maximum context lengths for each dense model. The 0.6B model has 28 layers and 32K context; the 1.7B has 28 layers and 32K context; the 4B has 36 layers and 128K context; the 8B has 36 layers and 128K context; the 14B has 40 layers and 128K context; the 32B has 64 layers and 128K context. The jump from 32K to 128K context at the 4B threshold reflects a practical design decision: models below 4B are primarily intended for edge deployment where long-context processing is less critical, while models at 4B and above are expected to handle document-length inputs.

Tie embedding. The smallest models (0.6B, 1.7B, 4B) use tied embedding weights—the input token embedding matrix and the output projection (LM head) share the same parameters. This saves roughly $V \times d_{\text{model}}$ parameters, where $V = 151,669$ is the vocabulary size and $d_{\text{model}}$ is the hidden dimension. For the 0.6B model with likely $d_{\text{model}} \approx 1024$, this saves approximately 155M parameters—a significant fraction of the total. The 8B, 14B, and 32B models untie the embeddings, following the conventional wisdom that larger models benefit from separate input and output representations because the representations needed for encoding a token in context differ from those needed for predicting the next token.

Tokenizer. All models use Qwen's tokenizer (Bai et al., 2023), which implements byte-level byte-pair encoding (BBPE). BBPE operates on bytes rather than Unicode characters, meaning any text—regardless of script or encoding—can be tokenized without unknown tokens, because every character can be decomposed to bytes and every byte is in the vocabulary. The vocabulary size is 151,669 tokens, which is relatively large (GPT-4 uses roughly 100K, Llama-3 uses 128K) and reflects the need to represent 119 languages efficiently—a larger vocabulary means fewer tokens per sequence for multilingual text, reducing both training and inference costs. The BBPE algorithm (Sennrich et al., 2016; Brown et al., 2020; Wang et al., 2020) learns merge rules that combine frequently co-occurring byte sequences into subword tokens, with the vocabulary size controlling the granularity of these merges.

Mixture-of-Experts (MoE) Architecture

The Qwen3 MoE models (30B-A3B and 235B-A22B) share the same foundation as the dense models—GQA, SwiGLU, RoPE, RMSNorm, QK-Norm—but replace the feed-forward layers in each transformer block with an MoE layer. The key design decisions are the expert granularity, the routing mechanism, and the load-balancing strategy.

Expert configuration. Both MoE models use 128 total experts with 8 activated experts per token (Table 2). The 30B-A3B model has 48 layers and uses 32 query heads with 4 KV heads; the 235B-A22B has 94 layers and uses 64 query heads with 4 KV heads. The "A" in the model names denotes the activated parameter count: 30B-A3B has 30B total parameters but only 3B activated per token (the other 27B are in the inactive 120 experts on a given forward pass); 235B-A22B has 235B total but 22B activated per token.

This is a form of fine-grained expert segmentation (Dai et al., 2024). Rather than having a small number of large experts (e.g., 8 experts of size $d_{\text{ff}}$ each), the model has many small experts (128 experts, each a fraction of $d_{\text{ff}}$ in hidden dimension), and the router activates 8 of them per token. The result is that the total computational cost per token is 8/128 = 1/16 of the cost if all experts were active, but the model can still express 128 different feed-forward transformations. Fine-grained segmentation increases the combinatorial expressiveness: with 128 experts and 8 active per token, there are $\binom{128}{8}$ possible expert combinations (approximately $1.3 \times 10^{13}$), allowing the model to learn highly specialized processing pathways for different types of input.

Removed shared experts. Unlike Qwen2.5-MoE, which included both routed experts (activated by the router) and shared experts (always active regardless of routing), Qwen3-MoE excludes shared experts entirely. The motivation is not explicitly stated, but the design choice likely reflects an efficiency consideration: shared experts increase the fixed cost per token (since they are always active) and dilute the specialization benefit of the routed experts (since some computation is generic and not adapted to the input). By making all experts routed, every FLOP is allocated based on the input's specific characteristics, maximizing the return on the model's capacity.

Load balancing. The routing mechanism in MoE models faces a fundamental tension between specialization and utilization. A purely quality-maximizing router would send every token to the same small set of "best" experts, causing those experts to process most tokens while others remain idle—wasting capacity and creating compute bottlenecks (since expert computation is typically parallelized across devices). Load balancing losses encourage the router to distribute tokens more evenly across experts.

Qwen3 adopts global-batch load balancing loss (Qiu et al., 2025), which operates at the granularity of an entire training batch rather than per-sample. Standard load balancing losses compute the imbalance within each sample's token distribution and penalize deviations from uniformity; global-batch balancing instead computes the loss over all tokens in the batch. This matters because within a single short sequence, it may be natural for most tokens to route to the same expert (e.g., all code tokens going to a "code expert"), and penalizing this would force unnatural routing. By aggregating across the batch, the loss only penalizes biases that persist across many diverse samples—true load imbalance rather than natural per-sample specialization. The paper reports that this, combined with other training innovations, "yielded substantial improvements in model performance across downstream tasks."

Three-Stage Pretraining

The Qwen3 pretraining process is designed to build capabilities in a progressive sequence: first broad linguistic and world knowledge, then specialized reasoning, and finally long-context processing. This staged approach allows the learning rate schedule and data composition to be tailored to each training phase's objectives.

Stage 1: General Stage (S1). All Qwen3 models are initially trained on over 30 trillion tokens at a sequence length of 4,096 tokens. The objective is standard autoregressive next-token prediction: given a sequence of tokens $x_1, x_2, ..., x_{4096}$, the model predicts the probability distribution over the vocabulary for each position conditioned on all previous tokens, and the loss is the sum of cross-entropies across positions. The data at this stage covers all 119 languages and dialects and includes "high-quality content in various domains such as coding, STEM, reasoning tasks, books, multilingual texts, and synthetic data."

The 30-trillion-token budget is approximately double the Qwen2.5 pretraining corpus and represents a significant scaling effort. To put this in perspective: Llama-3 was trained on 15T tokens, DeepSeek-V3 on 14.8T tokens, and Gemma-3 on approximately the same scale as Llama-3. The Qwen3 corpus is roughly 2× larger than these comparable open-source models, which is one factor in the performance improvements over Qwen2.5 observed across scale.

A sequence length of 4,096 tokens is relatively short by modern standards but is standard for the initial training phase because (a) shorter sequences allow larger batch sizes (more sequences packed into the same memory), which improves throughput and gradient signal quality, and (b) most of the model's core language understanding can be learned from local context—the relationship between words within a few thousand tokens captures the vast majority of syntactic and semantic patterns.

Stage 2: Reasoning Stage (S2). Building on the general foundation, the models are further trained on about 5 trillion tokens—approx 5T—at the same 4,096-token sequence length, but with a data composition that increases the proportion of STEM, coding, reasoning, and synthetic data. The learning rate decay is accelerated during this stage, which means the model transitions from exploration (large step sizes that can move parameters significantly) to fine-tuning (small step sizes that refine existing knowledge) more quickly than in S1. This is appropriate because the model already has strong general capabilities from S1, and the goal is to adapt those capabilities to reasoning tasks without destabilizing the learned representations.

The reasoning stage is where the model acquires the knowledge and skills that will later be activated through chain-of-thought prompting during post-training. The increased STEM and coding data teaches the model mathematical notation, algorithmic patterns, logical deduction structures, and formal reasoning conventions—the "raw material" that the post-training RL phase will then shape into extended reasoning chains.

Stage 3: Long Context Stage. The final pretraining stage extends the maximum context length from 4,096 to 32,768 tokens. The model is trained on hundreds of billions of tokens at this extended sequence length, with the corpus composed of 75% text between 16,384 and 32,768 tokens in length and 25% text between 4,096 and 16,384 tokens. This deliberate distribution ensures that the model sees a mix of moderate and extreme-length contexts, learning to attend across the full range rather than only at the maximum length.

Two complementary techniques extend the context window beyond the trained length of 32K to 128K during inference. First, YARN (Yet Another RoPE extensioN, Peng et al., 2023) modifies the RoPE rotation frequencies for sequences longer than the training maximum by rescaling the frequency components differently for low and high frequencies. Low-frequency components (which encode long-range position differences) are interpolated to fit within the trained range, while high-frequency components (which encode short-range differences) are extrapolated. This respects the intuition that the model has seen the full range of short-distance relationships during training but only a compressed range of long-distance relationships—interpolating the latter prevents the model from encountering position encodings it has never seen. The paper uses a scaling factor of 4, converting the 32K training context to a 128K inference context.

Second, Dual Chunk Attention (DCA, An et al., 2024) decomposes attention into intra-chunk and inter-chunk components. The sequence is split into chunks of length $C$ (where $C$ is the original training context length, 32K). Within each chunk, standard full attention is applied. Between chunks, attention is computed using compressed representations or sliding windows. This allows the model to process sequences of 128K tokens without the $O(L^2)$ memory cost of full quadratic attention. DCA is a training-free technique—it is applied only at inference time, modifying how attention is computed without changing the model weights.

Pretraining Data Construction

The 36-trillion-token pretraining corpus is constructed through three primary data sources and a sophisticated annotation and filtering pipeline. The goal is to maximize both the scale and quality of training data, with particular attention to multilingual coverage and domain diversity.

PDF text extraction via vision model. A significant innovation in the data pipeline is the use of Qwen2.5-VL (Bai et al., 2025), a vision-language model, to perform text recognition on a large volume of PDF-like documents. PDFs pose a challenge for text extraction because they encode text as positioned glyphs rather than as structured text streams—simple text extraction yields garbled output because it ignores reading order, column layout, and table structure. By using a vision model that can "see" the document layout, the system can extract text in the correct reading order and with proper separation of columns, headers, and body text. The extracted text is then refined using the Qwen2.5 language model, which can correct OCR errors by leveraging linguistic knowledge (e.g., identifying that "thc" should likely be "the" based on context). The paper reports that this process yields "an additional set of high-quality text tokens, amounting to trillions in total."

Synthetic data generation. The paper employs domain-specific models to synthesize training data in specialized formats. Qwen2.5-Math (Yang et al., 2024c) generates mathematical content; Qwen2.5-Coder (Hui et al., 2024) generates code-related data; and Qwen2.5 itself generates general text. The synthetic data spans "dozens of domains" and includes formats such as textbooks, question-answering pairs, instructions, and code snippets. The total synthetic data volume is trillions of tokens. This approach addresses a fundamental bottleneck in language model pretraining: high-quality STEM and coding data is scarce on the open web relative to general web text, and synthetic generation allows controlled production of precisely the kind of data needed to improve reasoning capabilities.

The use of Qwen2.5-based models for generation creates a potential concern about inbreeding—models trained on their own predecessors' outputs can amplify errors and biases present in those outputs. The paper does not detail measures to prevent this, but the multi-stage process (base model generates, VL model extracts, LM refines) introduces multiple independent filtering steps that may reduce error propagation. Additionally, the 36T token corpus is large enough that synthetic data is mixed with substantial quantities of non-synthetic data (the 30T+ tokens from S1 come from diverse sources), which dilutes any potential synthetic-data artifacts.

Multilingual expansion. Qwen3 supports 119 languages and dialects, up from 29 in Qwen2.5. The paper states this was achieved by "incorporating additional multilingual data and introducing more languages" to the pretraining corpus. This represents roughly a 4× increase in the number of supported languages and likely required careful balancing to prevent the model from sacrificing performance on previously-supported languages when adding new ones. The multilingual data annotation system, described below, is crucial to maintaining this balance.

Data annotation and filtering system. The paper describes "a multilingual data annotation system" that annotates over 30 trillion tokens across multiple dimensions: educational value, fields, domains, and safety. These annotations are applied at a fine-grained level (presumably at the document or paragraph level, though the granularity is not specified). The purpose is to enable instance-level data mixture optimization, which the paper contrasts with prior work (Xie et al., 2023; Fan et al., 2023; Liu et al., 2024b) that optimized data mixtures at the source or domain level (e.g., "use 30% Wikipedia, 20% books, 50% web").

Instance-level optimization means that within a data source, individual documents or passages are weighted differently based on their annotated properties. For example, a Wikipedia article on linear algebra might be upweighted for its educational value in mathematics, while a Wikipedia article about a celebrity might be downweighted for contributing less to the model's targeted capabilities. The optimization is performed through "extensive ablation experiments on small proxy models"—small versions of the target architecture are trained on different data mixtures, and the mixture that yields the best proxy model performance is scaled up to the full training run. This is a standard technique in large-scale training (used by DeepSeek, Llama, and others) because running full-scale ablations would be prohibitively expensive.

Stage 1: Long-CoT Cold Start

The post-training pipeline begins with cold-start training on long chain-of-thought (long-CoT) reasoning data. The goal is to establish foundational reasoning patterns in the model before reinforcement learning, but without over-optimizing—the paper explicitly states that the cold-start phase should "minimize both the number of training samples and the training steps" to preserve the model's flexibility for the RL phase.

Dataset construction: query filtering. The dataset construction begins with a broad collection of problems spanning math, code, logical reasoning, and general STEM domains. Each problem must have a verifiable reference answer or code-based test case—this verifiability is essential for both the filtering process and the subsequent RL stage, which requires automated reward signals. The dataset undergoes two filtering phases.

In the query filtering phase, Qwen2.5-72B-Instruct is used as an automated filter to identify and remove queries that are "not easily verifiable." The paper gives specific criteria: queries containing multiple sub-questions (which complicate answer verification), queries asking for general text generation (where there is no single correct answer), and queries that Qwen2.5-72B-Instruct can answer correctly without using CoT reasoning. This last criterion is important: if a problem is so easy that the base model can solve it with direct answering, forcing the model to generate long reasoning chains for it teaches the wrong behavior—the model learns to produce unnecessary reasoning when a direct answer would suffice. By filtering these out, the cold-start data focuses exclusively on problems that genuinely benefit from multi-step reasoning.

Additionally, Qwen2.5-72B-Instruct annotates each query's domain to ensure balanced representation across mathematics, coding, logic, and STEM subfields. A reserved validation query set is held out from the cold-start data to be used in the RL stage, preventing the RL training from evaluating on problems seen during cold-start.

Dataset construction: response filtering. For each retained query, QwQ-32B (the Qwen team's prior reasoning model from March 2025) generates N candidate responses. QwQ-32B is chosen because it already produces long chain-of-thought reasoning—the cold-start data needs to demonstrate what good reasoning looks like so the base model can imitate the pattern. When QwQ-32B consistently fails to generate correct solutions for certain queries, human annotators manually assess the accuracy of the responses to determine whether the query should be retained or discarded.

For queries where at least some of the N responses are correct (positive Pass@N), further stringent filtering is applied to remove responses that exhibit specific failure modes. The paper lists six criteria: (1) incorrect final answers—the response arrives at a wrong conclusion despite potentially plausible reasoning; (2) substantial repetition—the model gets stuck in loops, repeating the same reasoning steps without advancing toward a solution; (3) guesswork without adequate reasoning—the model states an answer without showing the derivation, relying on pattern-matching rather than logical deduction; (4) inconsistencies between the thinking and summary contents—the model's chain-of-thought argues for one answer but the summary states a different one; (5) inappropriate language mixing or stylistic shifts—the reasoning switches between languages or registers in a way that would confuse the training signal; (6) suspicious similarity to potential validation set items—to prevent contamination, responses that appear to be memorized from the validation set are excluded.

After filtering, a carefully selected subset of the refined dataset is used for cold-start supervised fine-tuning. The paper does not disclose the exact size of this subset, the number of training steps, or the learning rate, stating only that the objective is to "instill foundational reasoning patterns" without "overly emphasizing immediate reasoning performance."

Why supervised fine-tuning before RL? The cold-start phase follows the strategy established by DeepSeek-R1 (Guo et al., 2025). Starting RL from a randomly-initialized or base-model policy on complex reasoning tasks is extremely inefficient because the initial policy has near-zero probability of generating a correct reasoning chain—the RL reward signal is sparse, and the model wanders aimlessly through the vast space of possible text sequences without guidance. The cold-start SFT provides an initial policy that already knows what reasoning looks like (the structural pattern of "thinking" followed by "response," the convention of showing step-by-step work, and basic problem-solving strategies). RL can then focus on improving the quality and correctness of the reasoning rather than discovering its basic form from scratch.

Stage 2: Reasoning RL

The reasoning RL stage takes the cold-start model and optimizes its chain-of-thought reasoning through reinforcement learning on verified problems. This is where the model transitions from imitating reasoning patterns to generating reasoning that actually leads to correct answers.

Query-verifier datasets. The RL training data consists of query-verifier pairs—each query comes with an automated verifier that can check whether a generated answer is correct. For math problems, the verifier is typically a ground-truth answer checker (compare the model's final answer string to the reference answer); for coding problems, the verifier is a set of unit tests that the generated code must pass. The paper specifies four criteria for selecting queries: (1) they were not used during the cold-start phase (to prevent the RL from simply memorizing cold-start solutions), (2) they are learnable for the cold-start model (the model has some non-negligible chance of solving them, so there is a learning signal), (3) they are as challenging as possible (to push the model's capabilities), and (4) they cover a broad range of sub-domains (to produce generalizable reasoning skills).

The final dataset consists of 3,995 query-verifier pairs. This is a small number relative to the scale of the model—3,995 problems is tiny compared to the trillions of pretraining tokens. The effectiveness of RL with so few problems depends on the quality and diversity of the reasoning trajectories, not just the number of distinct queries. Each query is used to generate many rollout trajectories (the paper mentions "a high number of rollouts per query"), and the model learns from both successful and unsuccessful attempts.

GRPO algorithm. The paper uses GRPO (Group Relative Policy Optimization, Shao et al., 2024) to update the model parameters. GRPO is a variant of policy gradient methods (like PPO, Proximal Policy Optimization) designed for language model training. The key difference from standard PPO is how advantage estimates are computed.

In standard PPO, for each query, the model generates one or more responses, and the advantage of each response is computed as the reward minus a learned value function baseline: $A(s, a) = R(s, a) - V(s)$. The value function $V(s)$ is estimated by a separate neural network trained concurrently, which adds computational overhead and introduces a source of variance in training.

GRPO eliminates the value function by computing advantages relative to the group mean reward. For a given query, the model generates a group of $G$ responses (rollouts). Each response receives a reward $R_i$ from the verifier. The advantage of response $i$ is computed as:

$A_i = \frac{R_i - \text{mean}(\{R_1, ..., R_G\})}{\text{std}(\{R_1, ..., R_G\})}$

where $R_i$ is the reward for the $i$-th response, $\text{mean}(\{R_1, ..., R_G\})$ is the average reward across the group, and $\text{std}(\{R_1, ..., R_G\})$ is the standard deviation of rewards across the group.

What this computes: For each query, GRPO generates $G$ complete responses, evaluates each with the verifier to get a scalar reward, standardizes the rewards within the group to have mean 0 and standard deviation 1, and uses these standardized rewards as advantages. Responses better than the group average get positive advantages (encouraging the policy to produce more like them); responses worse than average get negative advantages (discouraging the policy from producing them). The policy gradient then updates parameters to increase the log-probability of high-advantage tokens and decrease the log-probability of low-advantage tokens.

Why this form: Standardizing advantages within each query's group eliminates the need for a value function because the relative quality of responses is determined by comparison to other responses to the same query—there is no need to estimate an absolute "expected reward" for the query. This works because the verifier provides a deterministic, objective reward signal (unlike RLHF where rewards come from a noisy learned reward model), so the primary source of variance is across different responses to the same query, not across different queries. The standardization also normalizes the scale of advantages across queries of different difficulty (a math problem with rewards in {0, 1} and a coding problem with rewards in {0, 5} are brought to the same scale), preventing queries with larger reward magnitudes from dominating the gradient.

Training dynamics and hyperparameters. The paper reports several key observations about the RL training process. Using a "large batch size and a high number of rollouts per query" is beneficial—this is consistent with the GRPO algorithm's reliance on group statistics, where larger groups provide more reliable advantage estimates. "Off-policy training" is used to improve sample efficiency, meaning that trajectories generated in earlier training steps are reused in later steps rather than discarded—this is possible because the verifier always provides ground-truth rewards, so there is no distribution shift in the reward signal as the policy changes.

A crucial training challenge is balancing exploration and exploitation. If the model becomes too confident in its current reasoning strategy, it stops exploring new approaches and plateaus. If it explores too broadly, it wastes compute on low-quality reasoning that provides no learning signal. The paper reports addressing this by "controlling the model's entropy to increase steadily or remain stable"—higher entropy means the model's output distribution is more uniform (more exploration), while lower entropy means it is more peaked (more exploitation). The exact mechanism for entropy control is not specified, but common approaches include adjusting the sampling temperature during rollouts or adding an entropy bonus to the reward function.

The results show consistent and substantial improvements. For the Qwen3-235B-A22B model, the AIME'24 score increases from 70.1 to 85.1 over 170 RL training steps. This is a remarkable gain—over 15 percentage points on one of the most challenging math benchmarks—from only 170 parameter updates on 3,995 distinct problems. The efficiency stems from the quality of the cold-start initialization and the effectiveness of GRPO at rapidly refining reasoning patterns when the base policy is already competent.

Stage 3: Thinking Mode Fusion

The thinking mode fusion stage is the critical innovation that enables Qwen3 to operate as a single model with both thinking and non-thinking capabilities. The model emerging from Stage 2 is a pure reasoning model—it always generates long chain-of-thought before answering, similar to DeepSeek-R1 or QwQ-32B. The goal of Stage 3 is to add the ability to generate direct, non-reasoning responses without degrading the reasoning capabilities that Stage 2 developed.

Training approach: continual SFT. Stage 3 uses continual supervised fine-tuning on the Stage 2 model, meaning the model is further trained on a new dataset that combines both thinking-mode and non-thinking-mode examples. The key design challenge is constructing an SFT dataset where both behaviors can be learned simultaneously without interference—the thinking-mode data must reinforce the reasoning skills, while the non-thinking-mode data must teach direct-answer patterns, and the model must learn to discriminate which behavior is appropriate based on the chat template.

Construction of thinking-mode SFT data. For the thinking-mode portion of the dataset, the paper uses rejection sampling on the Stage 1 (cold-start) queries using the Stage 2 (RL-trained) model. Rejection sampling works as follows:

1. For each query from the cold-start dataset, the Stage 2 model generates multiple candidate responses with thinking.
2. Each response is evaluated against the verifier.
3. Correct responses are retained; incorrect responses are discarded.

The result is a dataset of queries paired with verified correct thinking-mode responses generated by the best available model. Because the Stage 2 model is much stronger than the cold-start model, its responses are higher quality, and using them for SFT reinforces the reasoning improvements achieved during RL. The selection of cold-start queries (not the RL queries) for this process prevents the SFT data from being on-policy with respect to the RL training, which could cause overfitting to the RL reward.

Construction of non-thinking-mode SFT data. The non-thinking data is "carefully curated to cover a diverse range of tasks, including coding, mathematics, instruction-following, multilingual tasks, creative writing, question answering, and role-playing." The paper implies this data is sourced from existing instruction-tuning datasets or generated by the model itself in non-thinking mode, though the exact provenance is not specified. The key quality control mechanism is "automatically generated checklists for assessing the response quality of non-thinking data"—presumably a rubric-based evaluation where the model's responses are scored against criteria like correctness, completeness, conciseness, and formatting.

The paper specifically notes that translation tasks are overrepresented in the non-thinking data to "enhance the performance on tasks with low-resource languages." This reflects a practical consideration: translation is a task where non-thinking mode is clearly appropriate (the model should translate directly without extended reasoning about the translation), and low-resource languages particularly benefit from the added training signal.

Chat template design. The mechanism for teaching the model when to think and when not to think is the chat template, shown in Table 9. The template encodes the mode as explicit flags in the user query:

• For thinking mode: the user's query ends with /think (e.g., "Solve the equation x^2 + 5x + 6 = 0 /think"). Alternatively, the /think flag can be omitted since thinking mode is the default—the model will engage in reasoning unless explicitly told not to.
• For non-thinking mode: the user's query ends with /no think (e.g., "Translate 'hello' to Spanish /no think").

The assistant's response format differs between modes. In thinking mode, the response contains:

<|im_start|>assistant
 thinking
{thinking content}
 response
{response}<|im_end|>

The {thinking content} is the chain-of-thought reasoning; the {response} is the final answer or output. In non-thinking mode, the response has the same structure but with an empty thinking block:

<|im_start|>assistant
 thinking
 response
{response}<|im_end|>

The empty thinking block is an intentional design choice. Rather than having completely different output formats for the two modes, the model always outputs the thinking-then-response structure but suppresses the thinking content when in non-thinking mode. This "ensures internal format consistency within the model and allows developers to prevent the model from engaging in thinking behavior by concatenating an empty think block in the chat template." In other words, at inference time, the system can force non-thinking behavior by pre-filling the assistant's response with the empty thinking block and letting the model continue generating from there—the model sees that thinking has already "concluded" and proceeds directly to the response.

For multi-turn dialogs, the paper introduces additional complexity: "we randomly insert multiple /think and /no think flags into users' queries, with the model response adhering to the last flag encountered." This teaches the model that the mode can change mid-conversation—a user might ask a simple follow-up question that doesn't require thinking after a complex reasoning question that did. The model learns to track the most recent flag and adjust its behavior accordingly.

By default, the model operates in thinking mode. The paper adds "some thinking mode training samples where the user queries do not include /think flags" to establish this default behavior, ensuring the model reasons by default and only suppresses reasoning when explicitly instructed.

The role of the /think flag in the tokenizer. The paper notes that this feature "has been implemented in the chat template supported by the Hugging Face's tokenizer, where the thinking mode can be disabled using an additional parameter enable_thinking=False." This means that the mode control is a feature of the inference code, not of the model weights themselves—the tokenizer constructs the appropriate chat template based on the parameter, and the model simply responds to what it sees in its context. The model has learned to associate the presence or absence of the flag with different response behaviors through the SFT training.

Stage 4: General RL

The General RL stage broadens the model's capabilities beyond the math and coding reasoning focus of Stages 1–2. By this point, the model already has strong thinking and non-thinking abilities, but its training has been heavily weighted toward verifiable reasoning tasks. Stage 4 applies reinforcement learning across a diverse set of tasks to improve the model's general helpfulness, instruction following, format adherence, and agent capabilities.

Reward system design. The paper establishes a "sophisticated reward system covering over 20 distinct tasks, each with customized scoring criteria." These tasks target five core capability areas:

1. Instruction Following: The model must accurately interpret and follow user instructions regarding content, format, length, and structured output. This is evaluated using rule-based rewards (Lambert et al., 2024)—for example, if the instruction says "respond in JSON," the reward checks whether the output is valid JSON; if the instruction specifies "no more than 100 words," the reward checks the word count.

2. Format Following: Beyond explicit instructions, the model must adhere to Qwen3-specific formatting conventions. This includes responding appropriately to the /think and /no think flags (switching modes correctly) and consistently using the designated tokens ( thinking and response) to separate the thinking and response parts.

3. Preference Alignment: For open-ended queries where there is no single correct answer (e.g., "write a poem about autumn"), the reward is based on human preference judgments—helpfulness, engagement, style, and overall quality. This uses a trained reward model.

4. Agent Ability: The model is trained to invoke tools via designated interfaces. Critically, during RL rollouts, "the model is allowed to perform complete multi-turn interaction cycles with real environment execution feedback." This means the model generates a tool call, the tool is actually executed (e.g., a search query is performed, a calculator is run), the result is fed back into the context, and the model continues. The reward is based on the final outcome of the multi-turn interaction, not just the correctness of individual tool calls. This trains the model to recover from tool errors and adapt its strategy based on intermediate results.

5. Specialized Scenarios: For Retrieval-Augmented Generation (RAG), the reward includes signals that guide the model to generate responses grounded in the retrieved documents rather than hallucinating. The paper does not detail the specific RAG reward design, but typical approaches include rewarding the model when its generated text has high overlap with the retrieved passages and penalizing when it makes claims not supported by the retrieved context.

Three types of rewards. The paper categorizes the rewards used in Stage 4 into three types:

Rule-based rewards are deterministic functions that check specific criteria. They are "useful for general tasks such as instruction following and format adherence" because the criteria are objective and unambiguous. The key advantage is that "well-designed rule-based rewards can assess the correctness of model outputs with high precision, preventing issues like reward hacking"—because the reward function is a fixed, transparent set of rules, the model cannot exploit ambiguity or inconsistency in the reward signal. The disadvantage is that rule-based rewards can produce false negatives when the model's output is correct but in an unexpected format (e.g., answering a math question in prose instead of the expected "Answer: 42" format).

Model-based rewards with reference answer provide a reference answer for each query and use Qwen2.5-72B-Instruct to score the model's response against this reference. This "allows for more flexible handling of diverse tasks without requiring strict formatting, avoiding false negatives that can occur with purely rule-based rewards." The judge model can assess whether the model's response is semantically equivalent to the reference even if phrased differently—for example, "the cat is on the mat" and "the mat has a cat sitting on it" would both be judged correct against a reference that says "a cat is on the mat." This flexibility comes at the cost of introducing the judge model's biases and occasional errors, but the reference answer provides a strong anchoring signal that reduces the risk of reward model drift.

Model-based rewards without reference answer use a reward model trained on human preference data to assign scalar scores to model responses. This "does not depend on a reference answer" and "can handle a broader range of queries while effectively enhancing the model's engagement and helpfulness." The reward model is trained separately (the paper does not detail its architecture or training) to predict human preference judgments—given two responses to the same query, it predicts which one a human would prefer. During RL, the model's response is scored by this reward model, and the score serves as the RL reward signal. The risk of this approach is reward hacking—the model may learn to produce responses that score highly with the reward model but are not actually better (e.g., adding flattery, excessive verbosity, or stylistic quirks that the reward model associates with high-quality responses). The combination with rule-based and reference-based rewards helps mitigate this by providing grounded signals that anchor the overall reward.

Training dynamics. Stage 4 uses the same GRPO algorithm as Stage 2, but applied across the diverse task set rather than only reasoning tasks. The paper does not provide specific hyperparameters for this stage, but the general approach is consistent: generate multiple responses per query, compute advantages relative to the group mean reward, and update the policy to favor high-reward responses.

Strong-to-Weak Distillation

The strong-to-weak distillation pipeline enables the smaller Qwen3 models (0.6B through 30B-A3B) to acquire both thinking and non-thinking capabilities without undergoing the full four-stage post-training process. The paper argues that this approach is not just more efficient but also produces better results than reinforcement learning applied directly to the smaller models.

The efficiency rationale. The four-stage post-training pipeline applied to the flagship 235B model involves substantial computational cost: the Reasoning RL stage alone requires 170 parameter updates with large batch sizes and many rollouts per query, each rollout being a potentially thousands-of-tokens-long chain of thought from a 235B-parameter model. Replicating this for every model size in the family would multiply the post-training cost by the number of models, making the overall project economically and environmentally infeasible.

The distillation approach replaces this with a two-phase process that the paper claims requires "only 1/10 of the GPU hours compared to the four-stage training method" for the 8B model (Table 21). The key insight is that a student model can learn from a teacher's output distribution (logits) more efficiently than it can learn from sparse RL reward signals, because the logit-level supervision provides rich, dense gradient information at every token position.

Phase 1: Off-policy distillation. In the first phase, the student model is trained on outputs generated by the teacher model. The teacher (either Qwen3-32B or Qwen3-235B-A22B) generates responses in both /think and /no think modes for a collection of prompts. The student model is then fine-tuned via supervised learning to predict these teacher-generated responses—standard next-token prediction, but the target tokens come from the teacher rather than from a ground-truth dataset.

This phase teaches the student the basic form of both thinking and non-thinking responses. By seeing teacher-generated reasoning chains, the student learns what good reasoning looks like structurally (the format, the step-by-step progression, the separation of thinking from response). By seeing teacher-generated direct responses, the student learns when and how to answer without extended reasoning. The mode-switching behavior is also established: the student sees that queries with /think flags elicit reasoning and queries with /no think flags elicit direct answers.

The term "off-policy" indicates that the training data is generated by a different policy (the teacher) than the one being trained (the student). This is the standard supervised fine-tuning paradigm and is efficient because the teacher outputs can be pre-generated once and reused across multiple student training epochs.

Phase 2: On-policy distillation. The second phase refines the student's capabilities by training on the student's own generated responses, but using the teacher's output distribution as the target rather than a hard-coded next-token prediction target. The process works as follows:

1. A prompt is sampled from the training distribution.
2. The student model generates a response in either /think or /no think mode. Crucially, the response is generated by the student, not the teacher—this is "on-policy" because the data comes from the current student policy.
3. For each token position in the generated response, the teacher model computes a probability distribution over the vocabulary given the same prefix. This produces a vector of teacher logits $z^T_t \in \mathbb{R}^{|V|}$ for each position $t$, where $|V| = 151,669$ is the vocabulary size.
4. The student model's logits at each position $z^S_t \in \mathbb{R}^{|V|}$ are compared to the teacher's logits, and the student is updated to minimize the Kullback-Leibler (KL) divergence between the student's output distribution and the teacher's output distribution.

The KL divergence from the teacher distribution $p^T$ to the student distribution $p^S$ at position $t$ is:

$D_{\text{KL}}(p^T_t \| p^S_t) = \sum_{v \in V} p^T_t(v) \log \frac{p^T_t(v)}{p^S_t(v)}$

where $p^T_t(v) = \text{softmax}(z^T_t)_v$ is the teacher's probability for token $v$ at position $t$, and $p^S_t(v) = \text{softmax}(z^S_t)_v$ is the student's probability for token $v$ at position $t$.

What this computes: For each token position in the student-generated sequence, the KL divergence measures how much information is lost when using the student's probability distribution to approximate the teacher's distribution. The student is penalized when it assigns low probability to tokens the teacher thinks are likely or when it assigns high probability to tokens the teacher thinks are unlikely. The total loss for a sequence is the sum (or average) of per-token KL divergences, and gradient descent updates the student parameters to reduce this loss.

Why this form: KL divergence has several desirable properties for distillation. First, it is asymmetric—the teacher's distribution is the "truth" and the student is measured against it, not vice versa. This makes sense because the teacher is the stronger model. Second, it provides a richer training signal than cross-entropy with a hard target. With a hard target (standard next-token prediction), the student is told "the correct next token is 'the'" and all other tokens are equally wrong. With KL divergence, the student is told "the teacher assigns 0.7 probability to 'the', 0.15 to 'a', 0.05 to 'this', and smaller probabilities to other tokens"—this conveys information about which alternatives are plausible, which helps the student learn nuanced language understanding rather than just memorizing token sequences. Third, KL divergence is differentiable and convex in the student's logits, making it well-behaved for gradient-based optimization.

The key difference from Phase 1 is that the student is trained on its own generated sequences, aligned to the teacher's distribution, rather than on teacher-generated sequences with hard targets. This addresses the distribution mismatch problem: during Phase 1, the student learns to predict tokens that appear in teacher-generated text, but when it generates text itself (at inference or during Phase 2), its own distribution may differ from the teacher's, leading to error accumulation. Phase 2 brings the student's generation distribution into alignment with the teacher's output distribution for the student's own typical outputs—a form of domain adaptation where the "domain" is the student's own generation distribution.

Empirical comparison with RL (Table 21). The paper provides a direct comparison of three approaches for training the Qwen3-8B model's reasoning capabilities, all starting from the same off-policy distilled checkpoint:

Numbers in parentheses are pass@64 scores—the fraction of problems where at least one of 64 generated responses is correct, measuring the model's exploration capability rather than its single-attempt accuracy.

The results show that on-policy distillation achieves substantially higher pass@1 scores (74.4 vs. 67.6 on AIME'24) and higher pass@64 scores (93.3 vs. 90.0 on AIME'24) compared to RL, while requiring only 1,800 GPU hours versus 17,920—approximately a 10× efficiency advantage. The pass@64 improvement is particularly significant: it indicates that distillation from the teacher's logits expands the student's "reasoning horizon"—the student learns not just which reasoning paths are correct, but which directions are promising to explore, enabling it to eventually find correct solutions that it cannot immediately identify.

The paper argues: "distillation from teacher logits enables the student model to expand its exploration space and enhance its reasoning potential, as evidenced by the improved pass@64 scores... after distillation, compared to the initial checkpoint. In contrast, reinforcement learning does not lead to any improvement in pass@64 scores." This suggests that RL, with its sparse reward signal (correct/incorrect), creates a narrow optimization pressure that improves the model's ability to produce its best reasoning more consistently but does not broaden the range of problems it can potentially solve. Distillation, with its dense token-level signal, transfers the teacher's broader reasoning distribution to the student, expanding the set of problems within the student's reach.

Thinking Budget Mechanism

The thinking budget mechanism allows users to control the depth of reasoning by specifying a maximum number of thinking tokens. When the model's chain-of-thought reaches this threshold during generation, the reasoning is interrupted and the model produces a final answer based on its partial reasoning.

Implementation (Section 4.3). The mechanism is surprisingly simple. During inference, the model generates tokens as usual. The generation process tracks the number of tokens produced within the thinking... block. When this count reaches the user-specified budget, the system manually inserts a stop-thinking instruction into the model's context:

"Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n response.\n\n"

After this instruction is inserted, the model proceeds to generate the response section and the final answer. The model is not retrained or fine-tuned on these interrupted trajectories—the paper explicitly states that "this ability is not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion" (Section 4.3).

Why this works. The emergence of budget control from the mode fusion training is an intriguing example of compositional generalization. During Thinking Mode Fusion (Stage 3), the model is trained on two extremes: full thinking-mode responses (with lengthy chain-of-thought) and full non-thinking-mode responses (with empty thinking blocks). The non-thinking mode training teaches the model that it is possible to produce valid responses with minimal reasoning—when the thinking block is empty or extremely short, the model still produces coherent answers.

The thinking budget introduces an interpolation between these two extremes. By providing partial reasoning (some thinking tokens, but fewer than the model would naturally produce) followed by the explicit instruction to conclude, the model leverages its non-thinking-mode training to generate a response from incomplete reasoning. This is possible because the non-thinking mode training taught the model that responses do not require complete reasoning chains—partial or implicit reasoning is sufficient. The stop-thinking instruction disambiguates the situation for the model, making explicit that this is a budget-constrained scenario where a complete reasoning chain is not expected.

Empirical scaling behavior (Figure 2). The paper presents thinking budget scaling curves for the Qwen3-235B-A22B model on four benchmarks spanning Mathematics, Coding, and STEM domains. The curves show "scalable and smooth performance improvements correlated to the allocated thinking budget"—as the budget increases, performance improves monotonically, with no observed saturation at the tested budgets. The paper notes that "if we further extend the output length beyond 32K, the model's performance is expected to improve further in the future."

The smooth scaling is significant because it demonstrates that the model's reasoning can be dynamically truncated without catastrophic failure. If the model required a fixed amount of reasoning to produce any correct answer, performance would show a step function—near zero below a threshold and near maximum above it. The smooth curve instead suggests that the model's reasoning is incrementally useful: partial reasoning provides partial benefit, and additional reasoning tokens provide diminishing but positive returns.

This has practical implications for deployment. In a production system, the thinking budget can be set based on query priority, latency requirements, or cost constraints without worrying about a "reasoning cliff" below which the model becomes useless. A budget of 2,048 tokens provides some reasoning benefit; a budget of 8,192 provides more; a budget of 32,768 provides the most. The system designer can make a continuous tradeoff between quality and cost rather than a binary one.

Relationship to test-time compute scaling. The thinking budget mechanism is conceptually related to the literature on test-time compute scaling, but it operates through a different mechanism. In test-time compute scaling as studied in, for example, the "compute-optimal test-time scaling" paper, additional compute is spent on search (beam search, best-of-N sampling) or revisions (iterative self-correction). In Qwen3's thinking budget, additional compute is spent on extended sequential reasoning—the model generates more intermediate tokens before committing to an answer, which is analogous to giving a human more time to "think through" a problem rather than running multiple solution attempts in parallel.

The paper does not compare the thinking budget mechanism to alternative test-time compute strategies (e.g., whether spending 8K tokens on extended reasoning is more effective than generating 4 independent 2K-token reasoning chains and selecting the best one), which leaves an open question about the optimal allocation of inference compute between reasoning depth and reasoning breadth.

Post-Training Ablation: Effects of Stages 3 and 4 (Table 22)

The paper provides an ablation study tracking the Qwen3-32B model's performance across the final three post-training stages: after Stage 2 (Reasoning RL), after Stage 3 (Thinking Mode Fusion), and after Stage 4 (General RL). This reveals both the benefits and costs of the mode unification and generalization training.

Gains from Thinking Mode Fusion (Stage 2 → Stage 3). The integration of non-thinking capabilities produces several notable improvements in thinking-mode performance:

• CounterFactQA (an in-house benchmark measuring the model's ability to identify counterfactual questions and avoid hallucinating): +10.9 points (50.4 → 61.3). This suggests that training on non-thinking data improves the model's factual grounding even when it is subsequently asked to reason—the model learns to check its reasoning against factual knowledge.
• LengthCtrl (creative writing with length requirements): +8.0 points (62.6 → 70.6). The non-thinking training likely improves the model's ability to follow explicit formatting constraints without overthinking them.
• ThinkFollow (multi-turn dialogs with randomly inserted /think and /no think flags): achieves 88.7. This is the first time the model can be evaluated on mode-switching, and the score indicates "an initial ability to switch between modes, though it still occasionally makes errors."

The new non-thinking mode itself shows strong initial performance: 57.1 on LiveBench, 88.5 on Arena-Hard, 78.4 on IFEval, and 86.7 on MMLU-Redux—competitive with dedicated non-reasoning models without any direct training on those benchmarks.

Losses from Thinking Mode Fusion. The mode fusion comes with performance costs on the model's core reasoning strengths:

• AIME'24: -1.9 points (83.8 → 81.9). The model's competition math performance degrades slightly as it learns non-thinking behaviors.
• LiveCodeBench v5: -1.2 points (68.4 → 67.2). Similar small degradation on coding.
• MMLU-Redux: -0.4 points (91.4 → 91.0). Knowledge tasks are minimally affected.

These small degradations are consistent with the interpretation that training on diverse non-thinking data slightly dilutes the model's specialized reasoning capabilities, but the effect is modest—roughly a 1–2 percentage point cost on the hardest benchmarks in exchange for a new operating mode.

Gains from General RL (Stage 3 → Stage 4). The final stage provides broad improvements across general capabilities:

• ThinkFollow: 88.7 → 98.9—the mode-switching capability is essentially perfected, "ensuring accurate mode switching."
• ToolUse (thought mode): 63.3 → 85.5 (+15.1), and ToolUse (non-thinking mode): 73.2 → 86.5 (+13.3). The RL training with real environment execution feedback dramatically improves the model's tool-calling stability and accuracy.
• Arena-Hard (thought mode): 89.4 → 93.8 (+4.4), and Arena-Hard (non-thinking mode): 88.5 → 92.8 (+4.3). General helpfulness improves in both modes.

Losses from General RL. The core reasoning benchmarks continue to show small degradations: AIME'24 drops from 81.9 to 81.4 (-0.5), LiveCodeBench v5 drops from 67.2 to 65.7 (-1.5), and MMLU-Redux drops from 91.0 to 90.9 (-0.1). The non-thinking mode shows small gains on these same benchmarks (AIME'24: 28.5 → 31.0; GPQA-Diamond: 50.4 → 54.6), suggesting that Stage 4's broad training partially recovers some of the cross-mode capability transfer.

The paper explicitly acknowledges this tradeoff: "For Knowledge, STEM, Math, and Coding tasks, Thinking Mode Fusion and General RL do not bring significant improvements. In contrast, for challenging tasks like AIME'24 and LiveCodeBench, the performance in thinking mode actually decreases after these two training stages. We conjecture this degradation is due to the model being trained on a broader range of general tasks, which may compromise its specialized capabilities in handling complex problems. During the development of Qwen3, we choose to accept this performance trade-off to enhance the model's overall versatility."

This is an honest acknowledgment of a fundamental tension in unified models: versatility and specialization are in conflict. A model that performs well across many tasks will not be optimal on any single task. The Qwen3 team's decision to accept a ~1.5 point degradation on AIME'24 in exchange for functional mode-switching and strong non-thinking performance reflects a deliberate product decision: the unified model's deployment simplicity and flexibility outweigh the small performance cost relative to maintaining separate specialized models.

4. Key Insights and Innovations

Innovation 1: Mode Unification as a Training Problem, Not an Architecture Problem

The dominant assumption in the field prior to Qwen3 was that thinking (extended chain-of-thought reasoning) and non-thinking (rapid direct response) require separate models. This was not an arbitrary convention—it reflected a genuine tension in training dynamics. Models trained for extended reasoning (DeepSeek-R1, QwQ-32B, o1) always engage in chain-of-thought, even for queries where it degrades user experience (simple factual questions, creative writing, translation). Models trained for general-purpose chat (GPT-4o, DeepSeek-V3, Qwen2.5-72B-Instruct) produce rapid responses but lack the disciplined multi-step reasoning needed for competition math and complex coding. The assumption was that these capabilities are fundamentally in conflict—that optimizing for one necessarily degrades the other, and the only solution is maintaining two separate model deployments.

Qwen3's fundamental insight is that this is a training data and template design problem, not an insurmountable capability conflict. The key move is the Thinking Mode Fusion stage (Section 4.3), which teaches a single model to discriminate between modes based on explicit flags in the chat template. The model learns to produce thinking... response sequences when it sees /think and to suppress the thinking block when it sees /no think. The empty thinking block in non-thinking mode (Table 9) is the critical structural innovation: rather than having two completely different output formats, the model always uses the same thinking-then-response structure but with suppressed thinking content. This "internal format consistency" (Section 4.3) means the model's output distribution is unified—non-thinking mode is not a separate behavior but a boundary condition of the same behavior with zero-length reasoning.

Why this is a framing shift, not just engineering. This reframes the thinking/non-thinking distinction from a model-level property (this model reasons, that model doesn't) to a query-level control (this query requires reasoning, that query doesn't). The model itself is agnostic—it will reason or not reason based on what its context tells it to do. This is analogous to the shift from task-specific fine-tuned models to instruction-tuned generalist models: rather than training separate models for summarization, translation, and QA, instruction tuning teaches a single model to read the instruction and produce the appropriate output format. Qwen3 extends this principle to reasoning depth, making "reasoning level" another dimension of instruction following.

The evidence that this is genuinely a training achievement rather than a trivial template trick comes from the ablation in Table 22. After Stage 2 (pure reasoning RL), the model has no non-thinking capability at all—it always reasons. After Stage 3, the ThinkFollow score (ability to switch modes based on flags) reaches 88.7, and non-thinking mode performance on LiveBench is 57.1. After Stage 4, ThinkFollow reaches 98.9 and non-thinking LiveBench reaches 59.8. This progression shows that mode-switching capability is acquired through training, not simply a feature of the chat template. The model had to learn to read the flag, suppress its reasoning impulse, and produce direct answers—behaviors it had never been trained on during Stages 1–2.

The tradeoff is real but modest: thinking-mode AIME'24 drops from 83.8 (Stage 2) to 81.9 (Stage 3) to 81.4 (Stage 4)—roughly a 2.4-point total degradation in exchange for an entirely new operating mode and strong non-thinking performance. The Qwen3 team consciously accepted this tradeoff (Section 4.7): "we choose to accept this performance trade-off to enhance the model's overall versatility." This is a product decision with intellectual implications: it demonstrates that near-Pareto-optimal mode unification is achievable, meaning the frontier between specialization and versatility is not as steep as the field assumed.

Innovation 2: Emergent Thinking Budget Control from Mode Interpolation

Prior reasoning models (o1, DeepSeek-R1, QwQ-32B) operate on a binary: either they think fully or not at all. There is no mechanism to say "think for 500 tokens and then give me your best answer." The thinking budget mechanism in Qwen3 is novel not just as a feature but as a discovery about what mode-unified training produces. The paper is explicit that "this ability is not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion" (Section 4.3).

What makes this intellectually significant is the implied generalization. During Thinking Mode Fusion, the model sees exactly two modes: full thinking (thousands of tokens of reasoning) and zero thinking (empty thinking block). It never sees examples of partial reasoning interrupted by a stop-thinking instruction. Yet when the stop-thinking instruction is injected into the model's context at an arbitrary point during reasoning, the model produces coherent final answers based on its incomplete chain of thought.

The conceptual move: modes as endpoints of a continuous space. The standard view treats thinking and non-thinking as categorical modes. The Qwen3 findings suggest they are actually endpoints of a continuous reasoning-depth spectrum, and the model's training on both endpoints allows it to interpolate. The mechanism is compositional: the non-thinking mode training taught the model that it is possible to generate valid responses with zero-length (empty) thinking blocks; the thinking-mode training taught it to generate valid responses with long thinking blocks; the interpolation between these extremes emerges because the model has learned a general "generate response conditional on thinking context" function where the thinking context can be any length from zero to its natural maximum.

Figure 2 provides the empirical support: performance on four benchmarks scales smoothly and monotonically with the thinking budget, with no observed saturation at the tested budgets. If the model required a fixed minimum reasoning depth to produce any correct answer, the curve would show a threshold—flat near zero until some critical budget, then steep improvement. The smooth scaling instead indicates that the model's reasoning is incrementally useful—each additional reasoning token provides marginal value, and partial reasoning provides partial benefit. This is a stronger claim about the nature of the model's reasoning than simply "thinking works"—it suggests the reasoning process is decomposable into individually useful steps rather than being an all-or-nothing trajectory.

Comparison to test-time compute scaling. Contemporary work on test-time compute scaling (beam search, best-of-N, iterative revision) treats additional compute as something applied externally—the model generates candidates, a verifier selects among them, or the model revises its own outputs. The thinking budget operates through a fundamentally different mechanism: additional compute is spent on extended sequential reasoning within a single forward pass, where the model itself determines how to use the extra tokens. This is closer to how humans allocate thinking time (spend more time on harder problems within a single attempt) than to ensemble or search-based approaches (try multiple times and pick the best). The paper doesn't compare these paradigms—a gap the authors acknowledge by leaving exploration of "extremely long contexts" to future work—but the thinking budget establishes that depth-first compute allocation is viable and scales smoothly, complementing the breadth-first approaches studied elsewhere.

Innovation 3: On-Policy Distillation as a Superior Alternative to RL for Reasoning in Small Models

The field's dominant approach to instilling reasoning in smaller models has been reinforcement learning—either directly (DeepSeek-R1 applied GRPO to all model sizes) or via distillation of RL-trained reasoning traces (DeepSeek-R1-Distill series used SFT on R1-generated outputs). The implicit assumption is that RL, with its ability to explore and discover new reasoning strategies, should be at least as effective as distillation, and likely more so for models above a certain capability threshold.

Qwen3 challenges this assumption directly with the comparison in Table 21. Starting from the same off-policy distilled 8B checkpoint, on-policy distillation achieves 74.4 on AIME'24 versus 67.6 for RL—a 6.8-point advantage—while requiring only 1,800 GPU hours versus 17,920. More revealing is the pass@64 comparison: on-policy distillation improves pass@64 from 90.0 to 93.3, while RL leaves it unchanged at 90.0. This difference captures something fundamental about the two optimization approaches.

The diagnostic insight: RL narrows, distillation broadens. RL with outcome-based rewards (correct/incorrect) provides a sparse signal: the model gets positive reinforcement for trajectories that happen to be correct and negative reinforcement for those that aren't. This improves the model's probability of generating its best reasoning—the trajectories it already sometimes produces correctly become more likely, hence the pass@1 improvement from 55.0 to 67.6. But RL doesn't teach the model about reasoning strategies that could lead to correct answers if explored further—the model only gets credit for complete correct trajectories, so it never receives signal about promising-but-incomplete reasoning directions. Hence pass@64 remains at 90.0: the model's exploration frontier—the set of problems it can solve in 64 attempts—doesn't expand.

Distillation from teacher logits provides a fundamentally different signal. At every token position, the student receives the teacher's full probability distribution—not just "the correct next token is X" but "the teacher assigns 0.7 to X, 0.15 to Y, 0.05 to Z, and small probabilities to many other tokens." This conveys rich information about which reasoning directions are plausible and how to navigate the space of possible continuations. Even when the student cannot immediately produce a correct trajectory, it learns to move in promising directions—the "exploration space" expands. The pass@64 improvement from 90.0 to 93.3 reflects this: the student now explores a wider set of reasoning paths, some of which lead to correct answers it couldn't previously find.

Why this matters beyond the Qwen3 family. This finding has implications for the economics of open-source reasoning model development. RL at scale is expensive—the 17,920 GPU hours for the 8B model is substantial—and the paper's results suggest that for models below a certain capability threshold, that expenditure is better directed toward distillation from a stronger teacher. The threshold is not identified in this paper (it may depend on model size, pretraining quality, and task domain), but the existence of a regime where distillation dominates RL is a significant empirical finding that should influence how smaller reasoning models are developed going forward.

The result also connects to broader questions about the nature of reasoning capabilities. If distillation from teacher logits expands a student's reasoning frontier more effectively than RL, it suggests that high-quality reasoning distributions are more transferable than high-quality reasoning policies. A policy is a specific mapping from states to actions—knowing what to do in a given situation; a distribution is knowledge about which actions are good in which situations. The teacher's logits encode the distribution, and transferring it gives the student richer information than simply showing it which complete trajectories succeeded.

Innovation 4: Multilingual Reasoning Capability as a Deliberate Scaling Target, Not an Emergent Side Effect

Most large language models treat multilingual capability as something that emerges naturally from diverse pretraining data—throw in enough non-English text, and the model will work across languages. Qwen3 takes a more deliberate approach: multilingual reasoning is actively measured, diagnosed, and optimized as a first-class capability, not assumed to follow from English-language reasoning training.

The paper introduces or leverages several benchmarks specifically designed to test cross-lingual reasoning transfer: MT-AIME2024 (AIME problems translated into 55 languages), PolyMath (mathematical reasoning in 18 languages), MLogiQA (logical reasoning in 10 languages), and Multi-IF (instruction following in 8 languages). The per-language breakdowns in Tables 24–35 provide a granular view of where reasoning does and does not transfer.

The diagnostic value of multilingual reasoning benchmarks. These benchmarks reveal patterns that would be invisible in English-only evaluation. For the Qwen3-235B-A22B in thinking mode, MT-AIME2024 performance ranges from 86.7 (Spanish, French) to 66.7 (Thai) to 36.7 (Bengali, from the Belebele results). This variation is not simply correlated with resource level—Korean (80.0) outperforms Italian (80.0, but with different language family characteristics), and Vietnamese (83.3) is competitive with European languages despite being from a different language family. The variation suggests that the model's reasoning capability has uneven cross-lingual transfer that depends on factors beyond just the quantity of training data per language.

The Belebele benchmark results (Table 37) provide systematic evidence by language family. Qwen3-32B (Thinking) achieves 90.7 on Indo-European languages, 89.7 on Sino-Tibetan, 84.8 on Afro-Asiatic, but only 83.5 on Tai-Kadai and 83.1 on "Other" languages. The variation across language families is modest—a range of roughly 7.6 percentage points—suggesting relatively even transfer, but the pattern exists and is measurable. In contrast, Qwen2.5-32B-Instruct ranges from 87.0 (Uralic) down to 67.8 (Dravidian)—a 19.2-point spread, indicating much less even cross-lingual capability. The narrowing of this gap in Qwen3 is evidence that the expanded multilingual pretraining (119 languages vs. 29) and the specific overrepresentation of translation tasks in the non-thinking SFT data (Section 4.3) have meaningfully improved cross-lingual transfer.

What this reframes about multilingual LLM development. The standard narrative is that scaling pretraining data across more languages automatically yields better multilingual performance. Qwen3's results suggest a more nuanced picture: pretraining scale provides the foundation, but post-training choices determine whether reasoning capabilities actually transfer. The cold-start and RL stages are conducted primarily on English-language math and coding problems, yet the resulting reasoning capabilities transfer to 119 languages to varying degrees. The paper's finding that translation tasks were deliberately overrepresented in the non-thinking SFT data "to enhance the performance on tasks with low-resource languages" (Section 4.3) indicates an active effort to improve cross-lingual transfer through training data composition, not just pretraining scale. This is a methodological contribution: multilingual reasoning is not purely emergent but can be shaped through post-training data mixture decisions.

Innovation 5: The Four-Stage Post-Training Pipeline as a Decomposed Capability Acquisition Sequence

The four-stage post-training pipeline (Cold Start → Reasoning RL → Thinking Mode Fusion → General RL) might appear at first glance to be an incremental combination of existing techniques: cold-start SFT is from DeepSeek-R1, GRPO is from DeepSeekMath, thinking mode fusion is Qwen3's novel contribution, and general RL is standard. But the staging and sequencing is itself a contribution—the paper demonstrates that the order in which capabilities are acquired matters, and that each stage builds on specific properties of the previous stage.

The intellectual structure of the pipeline. The pipeline is organized around a specific developmental logic: first establish the form of reasoning (Stage 1), then optimize its quality on verifiable tasks (Stage 2), then generalize to handle both reasoning and non-reasoning (Stage 3), then broaden to diverse capabilities (Stage 4). This is not an arbitrary sequence—it reflects hypotheses about what must come before what.

Stage 1 must come before Stage 2 because RL from a randomly-initialized policy on sparse-reward reasoning tasks is extremely inefficient—the model would spend most of its time generating nonsense with no learning signal. The cold-start SFT provides an initial distribution that already produces recognizable reasoning, allowing RL to focus on quality improvements rather than discovering the basic form of chain-of-thought from scratch.

Stage 2 must come before Stage 3 because the thinking-mode responses used to construct the Stage 3 SFT data are generated by rejection sampling from the Stage 2 model. If the Stage 2 model were weaker, the thinking-mode SFT data would be lower quality, and the resulting unified model's reasoning would be degraded. The timing creates a virtuous cycle: stronger Stage 2 reasoning → higher-quality thinking-mode SFT data → better-preserved reasoning capability after mode fusion.

Stage 3 must come before Stage 4 because Stage 4's general RL includes format-following rewards that depend on the model correctly responding to /think and /no think flags. If the model cannot yet reliably switch modes, the format-following reward signal would be noisy and potentially harmful—the model would be penalized for mode-switching failures it hasn't yet learned to avoid. By establishing mode-switching capability in Stage 3, Stage 4's rewards provide clean reinforcement of already-reliable behavior.

The tradeoff point as a finding, not a flaw. The performance trajectory in Table 22 reveals that Stage 3 and Stage 4 actually degrade core reasoning benchmarks: AIME'24 drops from 83.8 (Stage 2) to 81.4 (Stage 4), a total of -2.4 points. The Qwen3 team's decision to accept this degradation—"we choose to accept this performance trade-off to enhance the model's overall versatility"—is a specific point on a capability-versatility Pareto frontier. They judged that 1.5–2.4 points on AIME'24 is worth paying for a model that can switch modes with 98.9% accuracy, handles tool use at 85.5, and performs strongly on general benchmarks. This is not a universal optimum—a team optimizing purely for math competition performance would stop after Stage 2—but it's a deliberate choice that reveals what the team values in a production model.

The pipeline structure also reveals that capabilities are not monotonic with respect to training. More training (Stages 3–4) improves some capabilities (ThinkFollow, ToolUse, general helpfulness) while degrading others (specialized reasoning). This is a concrete instantiation of the "alignment tax" concept: making a model more generally useful and controllable incurs a cost on its specialized peak performance. Qwen3 quantifies this tax for a specific model family, providing evidence for a phenomenon that has been widely hypothesized but not systematically measured in open-source models.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses 15 benchmarks for base model evaluation and 23 benchmarks for post-trained model evaluation, spanning general knowledge, mathematics, STEM, coding, agent tasks, alignment, and multilingual capabilities. Benchmarks include MMLU, MMLU-Pro, MMLU-Redux, BBH, SuperGPQA, GPQA-Diamond, C-Eval, GSM8K, MATH, MATH-500, AIME'24, AIME'25, EvalPlus, MultiPL-E, MBPP, CRUX-O, MGSM, MMMLU, INCLUDE, IFEval, Arena-Hard, AlignBench v1.1, Creative Writing v3, WritingBench, ZebraLogic, AutoLogi, BFCL v3, LiveCodeBench v5, CodeForces, Multi-IF, MT-AIME2024, PolyMath, MLogiQA, Belebele, and RULER. For post-trained models, evaluation covers both thinking and non-thinking modes with separate configurations. The base model evaluation set draws primarily from established benchmarks used in prior Qwen releases and comparable open-source model reports.

• Base model(s). The Qwen3 series includes 6 dense models (0.6B, 1.7B, 4B, 8B, 14B, 32B) and 2 MoE models (30B-A3B, 235B-A22B), all pre-trained on 36 trillion tokens across 119 languages. The flagship Qwen3-235B-A22B has 235B total parameters with 22B activated per token. Post-trained variants of all sizes are evaluated. The models represent the Qwen family's latest iteration, building on the Qwen2.5 architecture with architectural modifications including QK-Norm, removed QKV-bias, fine-grained expert segmentation (128 experts, 8 active per token), and global-batch load balancing. The scale range—from 0.6B edge-deployable models to 235B datacenter-scale models—is chosen explicitly to "meet the needs of different downstream applications."

• Metrics. The primary metric is accuracy (percentage of questions answered correctly), computed differently per benchmark: for most knowledge and reasoning benchmarks, a single deterministic answer is compared against ground truth; for GPQA-Diamond, the paper samples 10 times per query and reports averaged accuracy; for AIME'24 and AIME'25, each year includes 30 questions total (Part I and Part II), and the paper samples 64 times per question, taking the average accuracy as the final score; for LiveCodeBench, the officially recommended prompt is used in non-thinking mode while a less restrictive prompt ("removing the restriction You will not return anything except for the program") is used in thinking mode to allow freer reasoning; for CodeForces, Elo ratings are computed with each problem solved by generating up to 8 independent reasoning attempts; for BFCL v3, models are evaluated using the FC format with a 64K context length (via yarn) for Multi-Turn evaluation, with some baselines derived from the leaderboard taking the higher scores between FC and Prompt formats; for long-context evaluation on RULER, the thinking budget is set to 8,192 tokens "to mitigate overly verbose reasoning on the extremely long inputs." For multilingual benchmarks (INCLUDE, MMMLU), only 10% of the original data is sampled "to improve evaluation efficiency."

• Baselines. For base model comparisons, the paper compares against Qwen2.5-0.5B/1.5B/3B/7B/14B/32B/72B-Base and Qwen2.5-Plus-Base (Yang et al., 2024b), DeepSeek-V3 Base (Liu et al., 2024a), Gemma-3-1B/4B/12B/27B (Team et al., 2025), Llama-3-8B (Dubey et al., 2024), Llama-4-Scout and Llama-4-Maverick (Meta-AI, 2025), and Qwen2.5-Turbo (Yang et al., 2024b). For thinking-mode (reasoning) comparisons, the baselines include OpenAI-o1 (OpenAI, 2024), DeepSeek-R1 (Guo et al., 2025), Grok-3-Beta Think (xAI, 2025), Gemini2.5-Pro (DeepMind, 2025), DeepSeek-R1-Distill-Llama-70B, QwQ-32B (Qwen Team, 2025), OpenAI-o3-mini (medium), DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-1.5B, and DeepSeek-R1-Distill-Llama-8B. For non-thinking-mode (general-purpose) comparisons, baselines include GPT-4o-2024-11-20 (OpenAI, 2024), DeepSeek-V3 (Liu et al., 2024a), Qwen2.5-72B-Instruct (Yang et al., 2024b), LLaMA-4-Maverick and LLaMA-4-Scout (Meta-AI, 2025), GPT-4o-mini-2024-07-18, Phi-4 (Abdin et al., 2024), Gemma-3-27B-IT/12B-IT/1B-IT (Team et al., 2025), Qwen2.5-32B/14B/7B/3B/1.5B-Instruct (Yang et al., 2024b), LLaMA-3.1-8B-Instruct (Dubey et al., 2024), and Phi-4-mini. All models are evaluated "using the same evaluation pipeline and the widely-used evaluation settings to ensure fair comparison."

• Generation budget / compute accounting. The paper does not standardize on a single generation budget metric across all evaluations. For base models, evaluation uses standard few-shot prompting (e.g., 5-shot for MMLU, 4-shot for GSM8K and MATH, 3-shot for BBH, 1-shot for CRUX-O) without multiple samplings per query (except as noted for specific benchmarks). For post-trained models, thinking-mode evaluation uses sampling temperature 0.6, top-p 0.95, and top-k 20; non-thinking mode uses temperature 0.7, top-p 0.8, top-k 20, and presence penalty 1.5. Maximum output length is set to 32,768 tokens except for AIME'24 and AIME'25 where it is extended to 38,912 tokens "to provide sufficient thinking space." The thinking budget experiments (Figure 2) explicitly vary the allowed thinking tokens. For the distillation efficiency comparison (Table 21), compute is measured in GPU hours, with on-policy distillation requiring 1,800 GPU hours versus 17,920 for reinforcement learning. For RULER long-context evaluation, thinking budget is capped at 8,192 tokens in thinking mode.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance testing for any benchmark comparisons. Base model and post-trained model evaluations are single-run measurements on standard benchmark test sets. For AIME and GPQA-Diamond, multiple samples per query (64 and 10 respectively) provide some variance reduction through averaging, but no error bars or standard deviations are reported. For the Belebele multilingual benchmark, 42 of 122 languages are excluded as "unoptimized," which may bias the reported multilingual performance upward by removing the hardest languages. The two-fold cross-validation protocol mentioned in Section 3.2 for the thinking budget experiments is not described in sufficient detail to assess its implementation. The experimental analysis section provides only point estimates throughout.

Main Quantitative Results

Base Model Performance: Pretrained Foundation Models

The base model evaluation establishes the quality of the pretrained foundation before any post-training. The headline result is that Qwen3 base models substantially outperform their Qwen2.5 counterparts at comparable parameter scales, with the flagship MoE model achieving state-of-the-art results among open-source base models.

Qwen3-235B-A22B-Base vs. comparable base models (Table 3). The flagship base model is compared against Qwen2.5-72B-Base (dense, 72B activated), Qwen2.5-Plus-Base (MoE, 271B total, 37B activated), Llama-4-Maverick-Base (MoE, 402B total, 17B activated), and DeepSeek-V3-Base (MoE, 671B total, 37B activated). The Qwen3-235B-A22B-Base, with 235B total and 22B activated parameters, achieves the highest score on 13 of 15 benchmarks. Specific numbers:

• MMLU: 87.81 vs. 87.19 (DeepSeek-V3-Base) vs. 86.06 (Qwen2.5-72B). A 0.62-point advantage over the nearest competitor.
• MMLU-Pro: 68.18 vs. 63.91 (Llama-4-Maverick) vs. 59.84 (DeepSeek-V3-Base). A 4.27-point lead—substantial on this harder benchmark.
• MATH: 71.84 vs. 63.32 (Llama-4-Maverick) vs. 62.78 (Qwen2.5-Plus). A roughly 8.5-point improvement over the next-best open-source base model.
• EvalPlus: 77.60 vs. 68.38 (Llama-4-Maverick) vs. 65.93 (Qwen2.5-72B). A 9.22-point advantage, the largest single-benchmark gap in the table.
• MultiPL-E: 65.94 vs. 62.26 (DeepSeek-V3) vs. 62.16 (Qwen2.5-Plus). A more modest but consistent lead on polyglot coding.
• INCLUDE: 73.46, which is second to DeepSeek-V3's 75.17 and Llama-4-Maverick's 73.47—one of two benchmarks where Qwen3-235B-A22B does not lead.

The comparison with DeepSeek-V3-Base is the most important, as DeepSeek-V3 is the previous state-of-the-art open-source base model. Qwen3-235B-A22B outperforms it on 14 of 15 benchmarks with approximately 1/3 the total parameters (235B vs. 671B) and 2/3 the activated parameters (22B vs. 37B). The one exception is INCLUDE, where DeepSeek-V3 leads by 1.71 points (75.17 vs. 73.46), suggesting that DeepSeek-V3's multilingual regional knowledge coverage slightly exceeds Qwen3's despite Qwen3's broader language support (119 vs. what DeepSeek-V3 supports—the paper does not specify DeepSeek-V3's language count for comparison). This is a plausible limitation given that expanding from 29 to 119 languages while maintaining performance on every language-specific benchmark is inherently challenging.

Qwen3-32B-Base vs. larger models (Table 4). The largest dense model in the Qwen3 series is compared against Qwen2.5-32B-Base, Qwen2.5-72B-Base (more than 2× the parameters), Gemma-3-27B-Base, and Llama-4-Scout-Base (MoE, 109B total, 17B activated). The most striking finding is that Qwen3-32B-Base outperforms Qwen2.5-72B-Base on 10 of 15 benchmarks despite having less than half the parameters (32B vs. 72B). Specific comparisons:

• MMLU-Pro: 65.54 (Qwen3-32B) vs. 58.07 (Qwen2.5-72B) — a 7.47-point advantage demonstrating dramatically improved reasoning on this harder benchmark.
• SuperGPQA: 39.78 vs. 36.20 — a 3.58-point lead on graduate-level QA.
• BBH: 87.38 vs. 86.30 — modest but consistent improvement.
• EvalPlus: 72.05 vs. 65.93 — a 6.12-point coding advantage.
• MultiPL-E: 67.06 vs. 58.70 — an 8.36-point advantage on polyglot coding.
• MATH: 61.62 vs. 62.12 — Qwen3-32B trails Qwen2.5-72B by 0.5 points on MATH, one of the benchmarks where the smaller model does not surpass the larger one, likely because MATH benefits directly from additional pretraining capacity.
• INCLUDE: 67.87 vs. 69.05 — trails by 1.18 points, consistent with the 235B model's relative weakness on this multilingual benchmark.

Against Llama-4-Scout-Base (109B total, 17B activated), Qwen3-32B-Base outperforms it on every single benchmark—15 of 15—with only 1/3 the total parameters (32B vs. 109B) but roughly 2× the activated parameters (32B vs. 17B). This is a clean sweep, indicating that for dense models of this scale, the Qwen3 architecture and training recipe produce systematically better results than Llama-4's MoE approach at comparable or smaller parameter counts.

Qwen3-14B-Base and Qwen3-30B-A3B-Base (Table 5). This comparison is particularly informative because it pits a dense model (14B) against an MoE model (30B-A3B with only 3B activated) and both against larger baselines. The results show:

• Qwen3-14B-Base outperforms Qwen2.5-14B-Base on all 15 benchmarks, with particularly large gains on MMLU-Pro (61.03 vs. 51.16, +9.87), EvalPlus (72.23 vs. 60.70, +11.53), and CRUX-O (68.60 vs. 61.10, +7.50).
• Qwen3-14B-Base achieves comparable or better performance than Qwen2.5-32B-Base (more than 2× parameters) on 8 of 15 benchmarks, including EvalPlus (72.23 vs. 66.25) and MATH (62.02 vs. 57.70).
• Qwen3-30B-A3B-Base, with only 3B activated parameters, achieves comparable performance to Qwen2.5-32B-Base (32B activated) across most benchmarks, and on some—like MultiPL-E (66.53 vs. 58.70) and MMLU-Pro (61.49 vs. 55.10)—it substantially exceeds it. This demonstrates roughly a 10× efficiency improvement in activated-parameter terms.
• Qwen3-30B-A3B-Base and Qwen3-14B-Base are closely matched overall, with the MoE model having a slight edge on general tasks (MMLU: 81.38 vs. 81.05) and multilingual tasks (INCLUDE: 67.00 vs. 64.55), while the dense model leads on some coding tasks (CRUX-O: 68.60 vs. 67.20).

Qwen3-8B/4B/1.7B/0.6B-Base vs. comparably-sized models (Tables 6–8). The edge-side dense models show a consistent pattern: Qwen3 models outperform their Qwen2.5 counterparts at the same or slightly larger parameter scales. For example:

• Qwen3-8B-Base (Table 6) outperforms Qwen2.5-7B-Base on 14 of 15 benchmarks (all except MMLU where it trails 76.89 vs. 74.16—the Qwen2.5-7B score seems anomalous relative to the Qwen3-8B result, but the paper reports both as-is). Notably, Qwen3-8B-Base even outperforms Qwen2.5-14B-Base on 7 of 15 benchmarks, including MATH (60.80 vs. 55.64) and GPQA (44.44 vs. 32.83)—a remarkable result where an 8B model surpasses a prior-generation 14B model on graduate-level science questions by 11.61 points.
• Qwen3-4B-Base (Table 7) outperforms Qwen2.5-3B-Base on all 15 benchmarks and achieves performance comparable to Qwen2.5-7B-Base on several benchmarks (EvalPlus: 63.53 vs. 62.18; MMLU-Redux: 72.79 vs. 71.06), with half the parameters.
• Qwen3-1.7B-Base (Table 8) outperforms Qwen2.5-1.5B-Base on all 15 benchmarks, with particularly large gains on BBH (54.47 vs. 45.10, +9.37) and MGSM (50.71 vs. 32.82, +17.89)—the latter reflecting the multilingual expansion.
• Qwen3-0.6B-Base (Table 8) represents the smallest model, achieving MMLU of 52.81 and GSM8K of 59.59—performance that would have been competitive with models 5–10× larger only a few years ago.

Scaling pattern across model sizes. A clear pattern emerges from Tables 3–8: Qwen3 dense models at size X achieve performance comparable to Qwen2.5 dense models at approximately 2× their parameter count. Specifically, Qwen3-0.6B ≈ Qwen2.5-0.5B (baseline, both very small), Qwen3-1.7B > Qwen2.5-3B on most benchmarks, Qwen3-4B ≈ Qwen2.5-7B, Qwen3-8B ≈ Qwen2.5-14B, Qwen3-14B ≈ Qwen2.5-32B, Qwen3-32B > Qwen2.5-72B on 10/15 benchmarks. This 2× parameter efficiency gain is attributed to the combination of expanded pretraining data (36T vs. Qwen2.5's ~18T tokens), improved data quality (PDF extraction, synthetic data), and the three-stage pretraining strategy. The gain is particularly pronounced on STEM, coding, and reasoning benchmarks—exactly the areas targeted by S2 (Reasoning Stage) of pretraining.

Post-Trained Model Performance: Thinking Mode (Reasoning)

Qwen3-235B-A22B Thinking vs. reasoning baselines (Table 11). The flagship model in thinking mode is compared against OpenAI-o1, DeepSeek-R1, Grok-3-Beta (Think), and Gemini2.5-Pro. The results show:

• Against DeepSeek-R1 (the leading open-source reasoning model): Qwen3-235B-A22B outperforms on 17 of 23 benchmarks with only ~60% of the activated parameters (22B vs. 37B) and ~35% of the total parameters (235B vs. 671B). Key advantages include AIME'24 (85.7 vs. 79.8, +5.9), AIME'25 (81.5 vs. 70.0, +11.5), BFCL v3 (70.8 vs. 56.9, +13.9), LiveCodeBench v5 (70.7 vs. 64.3, +6.4), and CodeForces rating (2056 vs. 2029). DeepSeek-R1 leads on MATH-500 (98.0 vs. 97.3—Qwen3's score is missing from the table, likely indicating they are tied or both near ceiling) and MMLU-Redux (92.9 vs. 92.7, margin of 0.2). The gap on IFEval (83.4 vs. 83.3, essentially tied) is negligible.
• Against OpenAI-o1: Qwen3-235B-A22B outperforms on most benchmarks including AIME'24 (85.7 vs. 74.3, +11.4) and LiveCodeBench v5 (70.7 vs. 63.9, +6.8), but trails on MMLU-Redux (92.7 vs. 92.8—essentially tied), GPQA-Diamond (71.1 vs. 78.0), and MATH-500 (98.0 for Qwen3 vs. 96.4 for o1—o1's score is lower, so Qwen3 leads here on the benchmark where DeepSeek-R1 leads Qwen3).
• Against Gemini2.5-Pro (the strongest closed-source model in the comparison): Qwen3-235B-A22B trails on MMLU-Redux (92.7 vs. 93.7), GPQA-Diamond (71.1 vs. 84.0, a 12.9-point gap), AIME'24 (85.7 vs. 92.0, -6.3), and MATH-500 (98.0 vs. 98.8), but leads on BFCL v3 (70.8 vs. 62.9) and PolyMath (54.7 vs. 52.2). The overall picture is that Qwen3-235B-A22B is competitive with but not exceeding the best closed-source models on reasoning tasks—the gap to Gemini2.5-Pro on GPQA-Diamond and AIME'24 is substantial.
• Multilingual reasoning: Qwen3-235B-A22B achieves 80.8 on MT-AIME2024, the highest among all compared models (Gemini2.5-Pro: 76.9, DeepSeek-R1: 73.5), and 54.7 on PolyMath (vs. 52.2 for Gemini2.5-Pro). The paper's emphasis on multilingual reasoning capability is borne out in these head-to-head comparisons.

Qwen3-32B Thinking vs. reasoning baselines (Table 13). The flagship dense model in thinking mode is compared against DeepSeek-R1-Distill-Llama-70B (a distilled 70B model), QwQ-32B (Qwen's previous reasoning model at the same size), and OpenAI-o3-mini (medium). The results:

• Against QwQ-32B: Qwen3-32B outperforms on 17 of 23 benchmarks, including AIME'24 (81.4 vs. 79.5, +1.9), BFCL v3 (70.3 vs. 66.4, +3.9), LiveCodeBench v5 (65.7 vs. 62.7, +3.0), and MT-AIME2024 (75.0 vs. 68.0, +7.0). The multilingual reasoning advantage is particularly striking. Qwen3-32B trails QwQ-32B on MATH-500 (97.2 vs. 98.0) and AutoLogi (87.3 vs. 88.1), but these are narrow gaps.
• Against DeepSeek-R1-Distill-Llama-70B (a 70B model): Qwen3-32B outperforms on 21 of 23 benchmarks with less than half the parameters. The only exceptions are MATH-500 (97.2 vs. 94.5, where Qwen3 leads) and—reading carefully—AIME'25 (72.9 vs. 56.3, Qwen3 leads significantly). The sweep is nearly total, with particularly large advantages on Arena-Hard (93.8 vs. 60.6), CodeForces (1977 vs. 1633), and Multi-IF (73.0 vs. 57.6).
• Against OpenAI-o3-mini (medium): Qwen3-32B is competitive, exceeding o3-mini on AIME'24 (81.4 vs. 79.6), BFCL v3 (70.3 vs. 64.6), MT-AIME2024 (75.0 vs. 73.9), and Multi-IF (73.0 vs. 48.4, a dramatic 24.6-point gap on multilingual instruction following). Qwen3-32B trails on GPQA-Diamond (68.4 vs. 76.8, -8.4) and MATH-500 (97.2 vs. 98.0). The overall pattern is that Qwen3-32B matches or exceeds o3-mini on reasoning benchmarks but lags on graduate-level science (GPQA-Diamond), likely reflecting o3-mini's stronger base scientific knowledge.

Qwen3-30B-A3B and Qwen3-14B Thinking vs. baselines (Table 15). These models are compared against DeepSeek-R1-Distill-Qwen-32B (32B) and QwQ-32B. The results demonstrate the effectiveness of strong-to-weak distillation:

• Qwen3-30B-A3B Thinking (3B activated parameters) achieves comparable performance to QwQ-32B (32B activated) on AIME'24 (80.4 vs. 79.5, +0.9), AIME'25 (70.9 vs. 69.5, +1.4), and LiveCodeBench v5 (62.6 vs. 62.7, essentially tied), with less than 1/10 the activated parameters. This is the paper's most dramatic parameter-efficiency demonstration for thinking mode.
• Qwen3-14B Thinking (14B parameters) also closely tracks QwQ-32B: AIME'24 79.3 vs. 79.5, AIME'25 70.4 vs. 69.5, LiveCodeBench v5 63.5 vs. 62.7. The 14B model outperforms DeepSeek-R1-Distill-Qwen-32B on 17 of 23 benchmarks despite having less than half the parameters.
• Both models show strong CodeForces ratings: Qwen3-30B-A3B at 1974 (97.7th percentile) and Qwen3-14B at 1766 (95.3rd percentile), compared to QwQ-32B at 1982 (97.7th percentile). The 30B-A3B model essentially matches QwQ-32B on competitive programming while using 1/10 the activated parameters.

Qwen3-8B, 4B, 1.7B, 0.6B Thinking vs. baselines (Tables 17, 19). The smallest models in thinking mode show impressive absolute performance:

• Qwen3-8B Thinking: AIME'24 of 76.0, AIME'25 of 67.3, MATH-500 of 97.4, outperforming DeepSeek-R1-Distill-Qwen-14B (69.7/44.5/93.9) on all three benchmarks despite having ~57% of the parameters. The CodeForces rating of 1785 (95.6th percentile) from an 8B model is remarkable.
• Qwen3-4B Thinking: AIME'24 of 73.8, AIME'25 of 65.6—numbers that would have been competitive with the best reasoning models from 2024, from a 4B model.
• Qwen3-1.7B Thinking: AIME'24 of 48.3, MATH-500 of 93.4. The AIME'24 score nearly doubles that of DeepSeek-R1-Distill-Qwen-1.5B (28.9) and exceeds DeepSeek-R1-Distill-Llama-8B (50.4) on MATH-500. The ZebraLogic score of 63.2 for the 1.7B model versus 37.1 for the 8B Llama distillate is an eye-catching instance where the smaller Qwen3 model dramatically outperforms a model nearly 5× its size.
• Qwen3-0.6B Thinking: MATH-500 of 77.6, AIME'24 of 10.7. The AIME'24 score is low in absolute terms but remarkable for a 0.6B model—for context, many 7–8B models score in the single digits on AIME benchmarks.

Post-Trained Model Performance: Non-Thinking Mode (General Purpose)

Qwen3-235B-A22B Non-Thinking vs. non-reasoning baselines (Table 12). The flagship model in non-thinking mode is compared against GPT-4o-2024-11-20, DeepSeek-V3, Qwen2.5-72B-Instruct, and LLaMA-4-Maverick. The results:

• Against GPT-4o-2024-11-20: Qwen3-235B-A22B outperforms on 18 of 23 benchmarks. Key advantages include MATH-500 (91.2 vs. 77.2, +14.0), AIME'24 (40.1 vs. 11.1, +29.0), AIME'25 (24.7 vs. 7.6, +17.1), LiveCodeBench v5 (35.3 vs. 32.7, +2.6), and CodeForces (1387 vs. 864). The math and coding advantages are dramatic, reflecting Qwen3's stronger base reasoning even in non-thinking mode. GPT-4o leads on IFEval (86.5 vs. 83.2), BFCL v3 (72.5 vs. 68.0), and Multi-IF (65.6 vs. 70.2—Qwen3 leads here, so GPT-4o does not lead on Multi-IF).
• Against DeepSeek-V3 (the leading open-source non-reasoning model): Qwen3-235B-A22B outperforms on 17 of 23 benchmarks. Notable advantages: AIME'24 (40.1 vs. 39.2, +0.9—essentially tied), Arena-Hard (96.1 vs. 85.5, +10.6), CodeForces (1387 vs. 1134), MT-AIME2024 (32.4 vs. 20.9), and MLogiQA (67.6 vs. 58.9). DeepSeek-V3 leads on GPQA-Diamond (59.1 vs. 62.9—Qwen3 leads here), IFEval (86.1 vs. 83.2), and LiveCodeBench v5 (33.1 vs. 35.3—Qwen3 leads). The actual count requires careful reading: Qwen3 leads DeepSeek-V3 on MMLU-Redux (89.2 vs. 89.1—essentially tied), GPQA-Diamond (62.9 vs. 59.1, Qwen3 leads), and most other benchmarks. The paper claims Qwen3 exceeds DeepSeek-V3 in 18/23 benchmarks.
• Against Qwen2.5-72B-Instruct: Qwen3-235B-A22B outperforms on 20 of 23 benchmarks. Advantages are particularly large on AIME'24 (40.1 vs. 18.9, +21.2), LiveCodeBench v5 (35.3 vs. 30.7), Arena-Hard (96.1 vs. 81.2), and Creative Writing v3 (80.4 vs. 61.8). The alignment and creative writing improvements suggest that Stage 4 General RL substantially improved these capabilities beyond what Qwen2.5 achieved.

Qwen3-32B Non-Thinking vs. baselines (Table 14). Compared against GPT-4o-mini-2024-07-18, LLaMA-4-Scout, and Qwen2.5-72B-Instruct:

• Against Qwen2.5-72B-Instruct (more than 2× parameters): Qwen3-32B outperforms on 15 of 23 benchmarks, including AIME'24 (31.0 vs. 18.9, +12.1), Arena-Hard (92.8 vs. 81.2, +11.6), CodeForces (1353 vs. 859), and AutoLogi (78.5 vs. 66.1). Qwen2.5-72B leads on MMLU-Redux (86.8 vs. 85.7) and BFCL v3 (63.4 vs. 63.0—essentially tied). The 32B model's ability to outperform a 72B model from the previous generation on a majority of benchmarks while matching it on others is a strong demonstration of the Qwen3 improvements.
• Against LLaMA-4-Scout (109B total, 17B activated): Qwen3-32B significantly outperforms on all benchmarks where the comparison is fair—LLaMA-4-Scout's lower CodeForces rating (981 vs. 1353), lower Arena-Hard (70.5 vs. 92.8), and lower AIME'24 (28.6 vs. 31.0) illustrate the gap.

Qwen3-30B-A3B and Qwen3-14B Non-Thinking vs. baselines (Table 16). The distilled models compete strongly with larger models:

• Qwen3-30B-A3B Non-Thinking (3B activated) and Qwen3-14B Non-Thinking (14B parameters) both outperform Qwen2.5-32B-Instruct (32B parameters) on most benchmarks. For example, Qwen3-14B achieves Arena-Hard 86.3 vs. 74.5 for Qwen2.5-32B-Instruct, and Qwen3-30B-A3B achieves AutoLogi 81.5 vs. 65.5—a 16.0-point improvement.
• Against Gemma-3-27B-IT (27B), Qwen3-14B leads on 14 of 23 benchmarks including MATH-500 (90.0 vs. 90.0—tied), LiveCodeBench v5 (29.0 vs. 26.9), and CodeForces (1200 vs. 1063). The models are broadly competitive, with Qwen3-14B having a slight edge on reasoning-heavy tasks.

Qwen3-8B, 4B, 1.7B, 0.6B Non-Thinking vs. baselines (Tables 18, 20). The smallest models show that distillation preserves strong general-purpose capabilities:

• Qwen3-8B Non-Thinking: AIME'24 of 29.1 vs. Qwen2.5-14B-Instruct's 15.2—nearly double the score at roughly half the parameters. MATH-500 of 87.4 vs. Qwen2.5-7B-Instruct's 77.6. CodeForces 1110 vs. Qwen2.5-14B-Instruct's 904.
• Qwen3-4B Non-Thinking: ZebraLogic 35.2, which is the highest among all compared models in Table 18—higher than LLaMA-3.1-8B-Instruct (12.8), Qwen2.5-14B-Instruct (19.7), and Gemma-3-12B-IT (17.8). This is a notable outlier where the 4B model excels at a specific reasoning benchmark in non-thinking mode.
• Qwen3-1.7B Non-Thinking: MATH-500 of 73.0, AIME'24 of 13.4. For a 1.7B model to achieve 13.4 on AIME in non-thinking mode is remarkable—consider that GPT-4o-mini-2024-07-18 achieves 8.1 on the same benchmark (Table 14).
• Qwen3-0.6B Non-Thinking: MATH-500 of 55.2, MMLU-Redux of 44.6. These numbers, while modest, represent functional capability at an extremely small scale—the 0.6B model outperforms Gemma-3-1B-IT on most benchmarks (MMLU-Redux 44.6 vs. 33.3, MATH-500 55.2 vs. 46.4, AutoLogi 37.4 vs. 16.4).

Long-Context Evaluation (Table 23)

The RULER benchmark evaluates long-context processing at context lengths from 4K to 128K tokens. The results in Table 23 show:

• Non-thinking mode: Qwen3 models generally outperform Qwen2.5 counterparts at similar sizes. For example, Qwen3-32B achieves an average RULER score of 93.7 vs. 92.9 for Qwen2.5-32B-Instruct, with particular advantages at 128K (85.6 vs. 82.0). Qwen3-14B achieves 94.6 average vs. 91.4 for Qwen2.5-14B-Instruct. At 128K, Qwen3-235B-A22B achieves 90.6, the highest among all non-thinking models, comparable to Qwen2.5-72B-Instruct's 128K score of 88.4.
• Thinking mode: Performance degrades compared to non-thinking mode at all context lengths for all models. Qwen3-235B-A22B achieves an average of 92.2 in thinking mode vs. 95.0 in non-thinking mode, with the gap widening at longer contexts (128K: 86.0 vs. 90.6). The paper hypothesizes that "the thinking content does not provide significant benefits for these retrieval tasks, which do not rely on reasoning and may instead interfere with the retrieval process." At very long contexts (128K), the non-thinking advantage is consistent across all model sizes: Qwen3-32B (85.6 non-thinking vs. 83.5 thinking), Qwen3-14B (85.1 vs. 79.0).
• Scaling with context length: All models show monotonic degradation as context length increases, but the degradation is relatively gentle. Qwen3-235B-A22B in non-thinking mode drops from 97.7 at 4K to 90.6 at 128K—only a 7.1-point decline across a 32× increase in context length.

Multilingual Evaluation: Per-Language Breakdowns (Tables 24–35, 37)

The paper provides granular per-language results for 12 languages across multiple benchmarks. These tables reveal substantial variation in cross-lingual performance:

• Spanish (Table 24): Qwen3-235B-A22B Thinking achieves PolyMath 57.3, MT-AIME2024 86.7. The non-thinking mode achieves MT-AIME2024 33.3—the thinking mode provides a 53.4-point boost on mathematical reasoning in Spanish, demonstrating the value of explicit chain-of-thought for cross-lingual transfer.
• Arabic (Table 28): Qwen3-235B-A22B Thinking achieves MT-AIME2024 76.7, PolyMath 53.7. The non-thinking mode achieves MT-AIME2024 40.0—a 36.7-point thinking boost, proportionally larger than for Spanish, suggesting that lower-resource languages benefit more from explicit reasoning.
• Japanese (Table 29): Qwen3-235B-A22B Thinking: MT-AIME2024 73.3, non-thinking: 26.7—a 46.6-point gap, the largest thinking boost among the languages in these tables. This may reflect Japanese's linguistic distance from English making implicit reasoning in Japanese harder, so the explicit chain-of-thought provides more value.
• Belebele by language family (Table 37): Qwen3-32B Thinking achieves strong performance across families: Indo-European 90.7, Sino-Tibetan 89.7, Afro-Asiatic 84.8, Austronesian 86.7, Dravidian 84.5, Turkic 89.3, Tai-Kadai 83.5, Uralic 91.3, Austroasiatic 88.0, Other 83.1. The range across families is 8.2 points (91.3 to 83.1), indicating relatively even cross-lingual transfer. In contrast, Qwen2.5-32B-Instruct ranges from 87.0 (Uralic) to 67.8 (Dravidian)—a 19.2-point spread, illustrating the improvement in cross-lingual consistency from Qwen2.5 to Qwen3.

Thinking Budget Scaling (Figure 2)

The thinking budget experiment on Qwen3-235B-A22B demonstrates that performance scales smoothly with allocated reasoning tokens across four benchmarks spanning Mathematics, Coding, and STEM. The paper states the curves show "scalable and smooth performance improvements correlated to the allocated thinking budget." The specific numeric values are not provided in the text—the results are presented only as line plots in Figure 2—but the paper notes that "if we further extend the output length beyond 32K, the model's performance is expected to improve further in the future." This suggests that at the tested budgets (up to ~32K tokens), performance is not saturating, and additional thinking tokens would continue to provide gains. The smooth scaling implies that the model's reasoning is incrementally useful—partial reasoning provides partial benefit—rather than requiring a critical minimum thinking depth to produce any correct answers.

Distillation vs. Reinforcement Learning (Table 21)

The comparison between reinforcement learning and on-policy distillation on Qwen3-8B provides a clear efficiency result:

• Off-policy distillation (starting point): AIME'24 55.0 (pass@64: 90.0), AIME'25 42.8 (83.3), MATH500 92.4, LiveCodeBench v5 42.0.
• + Reinforcement Learning: AIME'24 67.6 (90.0), AIME'25 55.5 (83.3), MATH500 94.8, LiveCodeBench v5 52.9, using 17,920 GPU hours.
• + On-policy Distillation: AIME'24 74.4 (93.3), AIME'25 65.5 (86.7), MATH500 97.0, LiveCodeBench v5 60.3, using 1,800 GPU hours.

On-policy distillation achieves higher performance on every benchmark while requiring approximately 1/10 the GPU hours. The pass@64 improvements are particularly informative: on-policy distillation improves pass@64 on AIME'24 from 90.0 to 93.3 and on AIME'25 from 83.3 to 86.7, while RL leaves both unchanged. This indicates that distillation expands the model's exploration frontier—the set of problems it can solve given multiple attempts—while RL only improves its best-guess accuracy within its existing frontier. The paper interprets this as evidence that "distillation from teacher logits enables the student model to expand its exploration space and enhance its reasoning potential."

Ablation Studies and Robustness Checks

Post-training stage ablation on Qwen3-32B (Table 22): The paper tracks performance across Stages 2, 3, and 4 of post-training. Key findings: Stage 3 (Thinking Mode Fusion) establishes mode-switching capability (ThinkFollow: not applicable in Stage 2 → 88.7 in Stage 3) and adds non-thinking mode functionality while causing small degradations in thinking-mode reasoning (AIME'24: 83.8 → 81.9, -1.9). Stage 4 (General RL) further improves mode-switching (ThinkFollow: 88.7 → 98.9) and substantially boosts tool-use capabilities (ToolUse thinking: 63.3 → 70.4 → 85.5 across stages) while causing additional small reasoning losses (AIME'24: 81.9 → 81.4, -0.5). The cumulative degradation from Stage 2 to Stage 4 is -2.4 points on AIME'24, traded for mode-switching at 98.9% accuracy and broad general capability improvements.

Long-context mode comparison (Table 23): All models are evaluated in both thinking and non-thinking modes on RULER. Thinking mode consistently underperforms non-thinking mode at all context lengths, with the gap widening at longer contexts. The Qwen3-235B-A22B achieves RULER average of 95.0 non-thinking vs. 92.2 thinking, with the 128K scores being 90.6 vs. 86.0. This confirms that chain-of-thought reasoning interferes with retrieval-based long-context tasks.

Multilingual Belebele by language family (Table 37): Qwen3 models show substantially improved cross-lingual consistency compared to Qwen2.5 models. The range across language families narrows from roughly 19 points (Qwen2.5-32B-Instruct) to roughly 8 points (Qwen3-32B), indicating that expanded multilingual pretraining (119 languages) combined with targeted post-training (translation task oversampling) produces more even cross-lingual transfer.

Strong-to-weak distillation at multiple scales (Tables 15–20): The distillation approach is validated across six student model sizes (30B-A3B, 14B, 8B, 4B, 1.7B, 0.6B). At every scale, the distilled Qwen3 models outperform comparably-sized models trained with RL (DeepSeek-R1-Distill series) and prior Qwen2.5 instruct models, often by large margins. The consistency across scales—from 0.6B to 30B-A3B—suggests that on-policy distillation is robust to model size and does not require per-size tuning of the distillation process.

Mode-switching accuracy (ThinkFollow in Table 22): The ThinkFollow benchmark specifically tests whether the model correctly switches between thinking and non-thinking modes based on /think and /no think flags in multi-turn dialogs. The score of 98.9 after Stage 4 indicates near-perfect mode discrimination, validating the chat template design and the Thinking Mode Fusion training approach.

Non-thinking data quality control: The paper mentions "automatically generated checklists for assessing the response quality of non-thinking data" (Section 4.3) and "we particularly increase the proportion of translation tasks" to enhance low-resource language performance. These are design choices rather than ablations, but the strong multilingual results (Tables 24–35, 37) provide indirect validation.

Negative result: ReST^EM for revision model (informal): While not in the post-training section, the paper's acknowledgment of tradeoffs in Section 4.7—specifically that General RL causes "performance in thinking mode actually decreases" on challenging math and coding tasks—is an honest negative result that quantifies the alignment tax for this model family.

Critical Assessment

Do the experiments support the central claims? The paper's primary claims are that Qwen3 achieves state-of-the-art performance among open-source models in both thinking and non-thinking modes, that the thinking budget mechanism enables smooth performance scaling with reasoning depth, and that strong-to-weak distillation efficiently transfers both modes to smaller models. The evaluation results in Tables 3–20 provide substantial evidence for the performance claims: Qwen3-235B-A22B outperforms DeepSeek-R1 on 17/23 benchmarks and DeepSeek-V3 on ~18/23 benchmarks, with particularly large gaps on mathematical reasoning and coding tasks. The distillation efficiency claim is well-supported by Table 21, which shows on-policy distillation achieving better results than RL at 1/10 the GPU hours. The thinking budget claim is supported by Figure 2, though the lack of exact numeric values in the text makes the magnitude of the effect difficult to assess from the paper alone.

Concerns about the evaluation methodology:

1. No confidence intervals or significance testing. Every result in Tables 3–22 is reported as a point estimate. For benchmarks like AIME'24 with 30 questions, a difference of 1–2 percentage points corresponds to less than one question—differences that could easily arise from sampling noise rather than genuine capability differences. The paper's practice of comparing models to one decimal place on benchmarks with small test sets creates a misleading impression of precision. For the 500-question MATH benchmark, a 1-point difference represents 5 questions—potentially meaningful but still within the range of sampling variability. The paper would be strengthened by reporting confidence intervals or at minimum the raw number of correct answers rather than just percentages.

2. No statistical protocol for model comparison. The paper declares winners and second-places (bold and underlined) based on raw scores without any test of whether differences are statistically significant. Given that many comparisons involve gaps of 1–3 points on benchmarks with 30–500 questions, the ranking may be partially determined by noise. The consistent patterns across benchmarks provide some reassurance, but the paper does not provide the tools to distinguish signal from noise.

3. Small test sets for key benchmarks. AIME'24 and AIME'25 each have only 30 questions. The 64-sample averaging increases measurement precision for a given model but does not change the fact that model comparisons are based on performance differences across 30 fixed questions—a small sample that may not generalize to the full distribution of competition math problems. LiveCodeBench v5 covers a specific time window (2024.10–2025.02); the questions from this period may not be representative of coding problems more broadly.

4. 10% data sampling for multilingual benchmarks. INCLUDE and MMMLU evaluations use only 10% of the original data "to improve evaluation efficiency." For MMMLU with 14 languages, this means the effective evaluation set per language may be very small, increasing noise in the per-language breakdowns. The paper does not report the actual number of questions evaluated per language, making it impossible to assess the reliability of the fine-grained multilingual comparisons in Tables 24–35.

5. Exclusion of "unoptimized" languages. The Belebele evaluation excludes 42 of 122 languages as "unoptimized." This is a significant caveat: the reported Belebele scores reflect performance on the 80 languages where Qwen3 performs best, creating an upward bias in the multilingual evaluation. The exclusion criteria are not defined, and the paper does not report performance on the excluded languages, so the reader cannot assess how much of the claimed 119-language coverage is functional versus optimized.

6. Model-based evaluation with reference answers. The General RL stage (Stage 4) uses Qwen2.5-72B-Instruct as a judge model for some rewards. If this judge model shares biases or blind spots with the Qwen3 model family (which is built on Qwen2.5 architecture and tokenizer), the evaluation may be systematically lenient—the model learns to produce responses that the judge likes, but the judge's preferences may not align with human preferences or task accuracy.

Missing baselines and comparisons:

1. No comparison of thinking budget vs. alternative test-time compute strategies. The paper demonstrates that performance improves with more thinking tokens (Figure 2), but does not compare this to spending the same inference compute budget on alternative strategies: generating multiple shorter reasoning chains and selecting the best (best-of-N reasoning), using the longer thinking budget for self-verification and revision, or combining thinking budget with parallel sampling. Without these comparisons, it is unclear whether extended sequential reasoning is the optimal way to spend inference compute, or whether the model could achieve better results by allocating the same number of total tokens differently.

2. No ablation of cold-start data sources. The cold-start dataset construction uses QwQ-32B to generate candidate responses and Qwen2.5-72B-Instruct for query filtering. The paper does not explore whether using a different teacher model or filtering model would change the cold-start quality, or whether the specific choice of QwQ-32B is essential to the pipeline's success.

3. No comparison of GRPO to alternative RL algorithms. The paper uses GRPO throughout but does not compare it to PPO, REINFORCE, or other policy gradient methods. Given that GRPO's key innovation is eliminating the value function, an ablation showing that this actually matters for training dynamics (rather than just computational efficiency) would strengthen the methodological contribution.

4. No distillation scaling law. The paper shows that distillation works for models from 0.6B to 30B-A3B, but does not explore how distillation quality scales with: the size gap between teacher and student, the amount of distillation data, or the number of distillation steps. These scaling relationships would be valuable for practitioners deciding whether to distill or train from scratch.

5. No comparison of Qwen3's thinking mode to Qwen3's non-thinking mode with best-of-N. The thinking mode evaluation uses 64 samples for AIME and takes the average. The non-thinking mode evaluation does not appear to use multiple samples (except for GPQA-Diamond with 10 samples). A comparison of thinking mode (single extended reasoning chain) against non-thinking mode with majority voting over 64 samples would reveal whether extended sequential reasoning per attempt is more effective than multiple rapid attempts with voting.

Where the claims hold conditionally:

• "State-of-the-art among open-source models": Holds for the specific benchmarks tested and the specific models compared. However, the paper does not compare against all recent open-source models (e.g., certain Mistral, Yi, or DBRX variants are omitted), and the rapidly evolving open-source landscape means "state-of-the-art" is a moving target. The claim is best understood as "competitive with or exceeding the strongest open-source models available at the time of writing on the specific benchmarks evaluated."

• "Outperforms DeepSeek-R1 on 17/23 benchmarks": Holds for the Qwen3-235B-A22B model in thinking mode on the exact benchmarks listed. However, DeepSeek-R1's performance numbers are taken from the published DeepSeek-R1 paper rather than re-evaluated under identical conditions. If there are differences in evaluation protocol (temperature, sampling, prompt format), the comparison may not be perfectly fair. The paper claims all models are evaluated "using the same evaluation pipeline," but for models from other organizations where the pipeline cannot be controlled, the comparison relies on published numbers that may reflect different evaluation conditions.

• "4× parameter efficiency over Qwen2.5": Holds for dense models on most benchmarks (Qwen3-32B ≈ Qwen2.5-72B on ~2/3 of benchmarks). However, the efficiency gain is not uniform—it is larger on STEM, coding, and reasoning tasks and smaller on general knowledge tasks. And the efficiency comparison is against Qwen2.5, not against all prior models; other model families (Gemma-3, Llama-4) are only compared at selected size points, not systematically across the full size range.

• "Thinking budget enables smooth performance scaling": The paper provides Figure 2 as evidence, but the specific shapes of the scaling curves, the rate of diminishing returns, and the variability across problem difficulty are not reported. Without knowing whether the improvement from 4K to 8K thinking tokens is +1 point or +10 points, it is difficult to assess the practical value of the thinking budget mechanism. The paper's statement that performance is "expected to improve further" beyond 32K is speculative without evidence.

What would strengthen the evaluation:

1. Reporting confidence intervals or standard errors for all benchmark comparisons, especially those on small test sets (AIME, LiveCodeBench, ZebraLogic).
2. Conducting and reporting the results of a thinking-budget vs. best-of-N sampling comparison at matched total token budgets.
3. Evaluating base models on additional benchmarks that test capabilities not explicitly targeted by the three-stage pretraining (e.g., factual recall, reading comprehension, commonsense reasoning) to verify that the STEM/coding emphasis of S2 does not regress general capabilities.
4. Evaluating post-trained models on out-of-distribution or adversarial benchmarks to test robustness of the mode-switching and thinking budget mechanisms.
5. Including the 42 "unoptimized" languages in the Belebele evaluation to provide an honest assessment of worst-case multilingual performance rather than only best-case.
6. Comparing the on-policy distillation approach to alternative distillation methods (e.g., distillation on teacher-generated trajectories only, distillation with auxiliary losses) to establish whether logit-level KL divergence is specifically important or whether any form of teacher supervision works.

6. Limitations and Trade-offs

Limitation 1: The Cost of Difficulty-Free Reasoning Is Not Isolated From the Unified Model's Architecture

The assumption or constraint. The paper presents Qwen3's thinking and non-thinking modes as capabilities of a single unified model, switchable via chat template flags. However, the model's default behavior—when no /think or /no think flag is present—is thinking mode (Section 4.3: "By default, the model operates in thinking mode"). This means that users who want non-thinking behavior must explicitly request it, and any deployment where the flag is omitted (due to user error, integration oversight, or legacy prompts) will silently invoke thinking mode, consuming up to 32,768 tokens of reasoning (the default max output length) before producing a response.

The consequence. For latency-sensitive or cost-sensitive deployments, this default creates a silent failure mode: a simple query like "What is the capital of France?" will trigger an extended chain-of-thought deliberation—potentially thousands of tokens of reasoning—before returning "Paris," wasting both compute and user time. The user may not realize the flag was omitted, and the excessive reasoning may appear as a bug rather than a configuration oversight. This is fundamentally different from dedicated reasoning models (which always reason and are deployed with that expectation) or dedicated chat models (which never reason). The unified model's default-to-thinking behavior introduces a new class of deployment error—mode misconfiguration—that does not exist in single-mode models.

The paper does not provide measurements of how much unnecessary reasoning is generated for simple factual queries when the /no think flag is omitted, nor does it quantify the latency or token-cost overhead of such misconfigurations. Without this data, practitioners cannot assess the risk of deploying the unified model versus maintaining separate thinking and non-thinking endpoints (where the mode is guaranteed by model selection, not flag adherence).

What evidence exists in the paper. Table 22 reports ThinkFollow scores (the model's ability to correctly switch modes based on flags) of 88.7 after Stage 3 and 98.9 after Stage 4, indicating that even after full post-training, ~1% of mode-switching events may fail. The paper does not report what happens when no flag is present—whether the model consistently enters thinking mode, whether it sometimes produces non-thinking responses anyway, or whether its behavior depends on query content. The default-to-thinking behavior is stated as a design choice ("we add some thinking mode training samples where the user queries do not include /think flags"), but its failure characteristics are not evaluated.

Mitigation status. The paper does not address this as a limitation. The default-to-thinking behavior is presented as a feature, not a bug. The empty thinking block in the chat template provides a technical mitigation (developers can force non-thinking mode by pre-filling the assistant response with an empty thinking block), but this requires deliberate action from the developer—it does not protect against accidental omission of the flag. The paper does not propose a query-content-based automatic mode detector that could route simple queries to non-thinking mode without requiring explicit user flags.

Limitation 2: The Thinking Budget Mechanism Has No Theoretic or Empirical Floor—Partial Reasoning Below Some Threshold May Be Harmful

The assumption or constraint. The thinking budget mechanism assumes that any amount of reasoning, no matter how truncated, produces a better answer than no reasoning at all. Figure 2 shows smooth performance improvements as the thinking budget increases, but the paper does not show what happens at very low budgets (e.g., 64, 128, or 256 tokens). The emergence of budget control from mode fusion training (Section 4.3: "this ability is not explicitly trained but emerges naturally") means the model has never been explicitly taught to produce useful answers from severely truncated reasoning—it learned to handle the extremes (zero tokens, full chain-of-thought) and the intermediate behavior is interpolated.

The consequence. There may exist a reasoning danger zone: thinking budgets that are too large for the model to treat as "no thinking" (producing its trained non-thinking behavior) but too small to complete any meaningful reasoning steps (producing garbled, incomplete, or misleading partial reasoning). Below some threshold budget, the model might generate reasoning that sets up a problem but fails to reach a conclusion, then produces a final answer that is actually worse than what it would have generated with zero thinking—because it is conditioning on an incomplete and potentially misleading partial chain of thought. If this danger zone exists, the smooth scaling curve in Figure 2 would actually be hockey-stick shaped (flat or negative at very low budgets, then rising), but the plot's x-axis resolution may be too coarse to reveal it.

This matters for practical deployment because users setting short thinking budgets (e.g., 512 tokens for latency-sensitive applications) might unknowingly be in the danger zone, getting worse results than if they had simply used non-thinking mode. The paper provides no guidance on minimum effective thinking budgets per task type.

What evidence exists in the paper. Figure 2 is the only evidence, and it is presented as line plots without numeric values. The x-axis range and resolution are not specified—the paper does not report whether budgets below some threshold were tested or whether the curves are monotonic over the full range of tested budgets. The paper notes that "if we further extend the output length beyond 32K, the model's performance is expected to improve further in the future," but says nothing about the lower bound. The non-thinking mode results (Tables 12, 14, 16, 18, 20) provide a baseline for zero-thinking performance, but the paper does not compare very-low-budget thinking against non-thinking mode on matched queries.

Mitigation status. Not addressed. The paper treats the thinking budget as a uniformly beneficial feature and does not discuss the possibility of a minimum effective budget. Future work on "adaptive difficulty estimation" that could route very easy queries to non-thinking mode (rather than low-budget thinking) would partially mitigate this, but the paper does not propose such a mechanism.

Limitation 3: Distillation Efficiency Claims Are Based on a Single Data Point (8B Model, Math/Code Tasks Only)

The assumption or constraint. The paper's headline finding on strong-to-weak distillation—that on-policy distillation achieves better performance than reinforcement learning at 1/10 the GPU hours—is based on a single experiment: Qwen3-8B on math and code-related queries only (Table 21). The paper states explicitly: "For simplicity, we focus solely on math and code-related queries in this comparison." This is a narrow slice of the model's capabilities. The distillation pipeline is applied to all six lightweight models (0.6B through 30B-A3B) across all task types, but the direct RL-vs-distillation comparison that establishes the 10× efficiency claim covers only one model size and two task domains.

The consequence. The efficiency advantage of distillation over RL may not generalize to other model sizes (particularly at the boundary where RL becomes more effective—e.g., larger student models with more capacity to explore independently) or to other task types (where the reward signal is less sparse, making RL more competitive). For general-domain tasks with dense, multi-dimensional reward signals (instruction following, creative writing, agent tool use), RL might be more efficient than distillation because the teacher's logit distribution provides less clear guidance for subjective quality (the teacher might assign high probability to multiple stylistically different but equally good responses, making the distillation target diffuse). The paper's strong claim that distillation is the preferred approach for all lightweight models rests on extrapolation from a single experiment.

Additionally, the 10× GPU-hour figure compares on-policy distillation against RL starting from the same off-policy distilled checkpoint. It does not compare against RL from scratch (no off-policy distillation) or against a hybrid approach (off-policy distillation + less RL). The comparison is between two specific points in a larger design space, and the optimal point may be different from either of them.

What evidence exists in the paper. Table 21 provides the only head-to-head comparison. The rest of the distillation evidence is indirect: Qwen3 distilled models outperform DeepSeek-R1-Distill models (which used SFT on teacher traces, not on-policy logit matching) on various benchmarks (Tables 15–20), but this compares different teacher models, different base models, and different distillation methods simultaneously—the cause of the performance difference cannot be isolated. The paper does not report whether on-policy distillation was attempted for the non-math/non-code tasks during the General RL stage, or whether those capabilities were transferred exclusively through off-policy SFT.

Mitigation status. The paper does not claim that the 10× efficiency ratio holds across all model sizes and tasks—the claim is specific to the 8B math/code experiment. However, the broader narrative of the paper (Section 4.5, Section 5 abstract) presents strong-to-weak distillation as a general solution for lightweight models. The limitation is partially mitigated by the strong overall performance of the distilled models across diverse benchmarks (Tables 15–20), which suggests the distilled models are genuinely capable, but this does not prove that distillation was more efficient than RL for those capabilities—only that distillation worked.

Limitation 4: No Systematic Comparison of the Thinking Budget Against Alternative Test-Time Compute Allocation Strategies

The assumption or constraint. The thinking budget mechanism allocates additional inference compute to extended sequential reasoning—the model generates more intermediate tokens in a single chain of thought before producing an answer. This is one point in a larger design space of test-time compute allocation strategies, which includes parallel sampling (generate N independent reasoning chains, select best via verifier or majority voting), iterative revision (generate, critique, revise), and search-based approaches (beam search over reasoning steps guided by a process reward model). The paper compares thinking mode against non-thinking mode, but never compares different ways of spending the same total token budget within thinking mode.

The consequence. Practitioners cannot determine whether the thinking budget's smooth scaling (Figure 2) represents an efficient use of compute. For example, if the model is allocated 8,192 tokens of total inference budget, it could: (a) generate one reasoning chain of 8,192 tokens, (b) generate 4 chains of 2,048 tokens each and select the best final answer, or (c) generate 2 chains of 4,096 tokens each and select the best. The paper provides no data on which allocation achieves higher accuracy at a given total budget. If parallel sampling with shorter chains outperforms a single long chain, the thinking budget mechanism would be suboptimal for accuracy—even though it provides smooth scaling, the absolute level of performance could be lower than alternative approaches.

This matters particularly for deployment scenarios where total token budget (and thus cost) is the binding constraint. The smooth scaling of accuracy with thinking budget is only valuable if single-chain sequential reasoning is the best way to convert tokens into correct answers. If parallelized shorter reasoning dominates, the thinking budget is a convenience feature (easy to implement, intuitive to control) but not a compute-optimal one.

What evidence exists in the paper. None. No experiment compares thinking budget allocation to alternative test-time compute strategies. The paper does not cite or engage with the literature on test-time compute scaling (e.g., best-of-N sampling, verifier-guided search, compute-optimal allocation), which makes this gap particularly notable—the relevant comparison is absent rather than inconclusive. The non-thinking mode evaluations use single responses (except the 64-sample averaging for AIME and 10-sample for GPQA-Diamond), but these are not positioned as test-time compute scaling comparisons.

Mitigation status. Not addressed. The paper treats the thinking budget as a self-evidently useful mechanism and does not discuss the relationship between sequential reasoning depth and parallel reasoning breadth. This is a significant omission given that contemporary work has made this a central question. The paper's suggestion that "if we further extend the output length beyond 32K, the model's performance is expected to improve further" implicitly endorses depth over breadth without evidence.

Limitation 5: No Measurement of Reasoning Interference on Tasks Where Deliberation Is Harmful

The assumption or constraint. The thinking mode fusion training (Stage 3) and general RL (Stage 4) teach the model to respond with extended chain-of-thought reasoning when the /think flag is present (or by default). The paper demonstrates that this reasoning improves performance on mathematics, coding, and STEM benchmarks. However, it does not systematically evaluate whether the reasoning process interferes with performance on tasks where deliberation is known to be unnecessary or counterproductive: simple factual recall, translation, sentiment analysis, text summarization, or following formatting instructions.

The consequence. The unified model may exhibit a reasoning tax on tasks that humans perform better without deliberation. For example, translation typically benefits from direct, fluent generation rather than explicit linguistic analysis—a model that reasons in English about how to translate a Spanish sentence may produce more literal, less idiomatic output. Creative writing may suffer from over-analysis—a model that plans the structure of a poem step by step may produce technically correct but lifeless verse. Factual recall tasks may trigger spurious reasoning that overrides correct memorized knowledge with incorrect logical deduction (a form of reasoning-induced hallucination).

The paper's Table 23 provides suggestive evidence: on the RULER long-context benchmark, thinking mode consistently underperforms non-thinking mode. The paper attributes this to reasoning content interfering with retrieval, but does not investigate whether similar interference affects other task categories. The ablation in Table 22 shows that CounterFactQA (identifying counterfactual questions to avoid hallucination) improves with mode fusion training—suggesting that some forms of reasoning actually help with factual grounding—but this is a single benchmark and may not generalize.

What evidence exists in the paper. Table 23 (RULER) shows thinking mode degradation relative to non-thinking mode at all context lengths for all model sizes. The paper hypothesizes that "the thinking content does not provide significant benefits for these retrieval tasks, which do not rely on reasoning and may instead interfere with the retrieval process," but this is a post-hoc interpretation, not a systematic investigation. No experiments directly measure whether thinking mode degrades translation quality, creative writing quality, or factual accuracy relative to non-thinking mode. The non-thinking and thinking mode results in Tables 11–20 are on different benchmarks or use different evaluation protocols (e.g., 64-sample averaging for AIME in thinking mode versus single-sample in non-thinking mode), making direct mode-to-mode comparisons on the same tasks largely impossible.

Mitigation status. Not addressed. The paper does not include a "no-thinking baseline" for reasoning-specific benchmarks (to measure how much the reasoning actually helps) or a "thinking overhead" measurement for non-reasoning benchmarks (to measure how much it hurts). The chat template flag system allows users to choose the appropriate mode, but the paper provides no guidance on which tasks benefit from reasoning, which are harmed, and which are mode-insensitive. Users must discover this through trial and error.

7. Implications and Future Directions

How This Work Changes the Landscape

Qwen3 introduces a capability-level reframing rather than a paradigm shift. The core insight—that extended chain-of-thought reasoning and rapid direct response can coexist in a single model, switchable via chat template flags—does not fundamentally alter our understanding of language model training or architecture. It does, however, change the default assumption about what a deployment-ready model should offer. Before Qwen3, the field operated on an implicit consensus: reasoning models and chat models are separate artifacts, optimized through different post-training pipelines, and served through different endpoints. Qwen3 demonstrates that this separation is a training design choice, not a fundamental constraint, and that a single model can serve both roles with near-Pareto-optimal tradeoffs (Section 4.7: the thinking-mode AIME'24 degradation from mode unification is ~2.4 points total, accepted in exchange for mode-switching capability and strong non-thinking performance).

This reframing has downstream consequences for how model families are developed. The four-stage post-training pipeline (Cold Start → Reasoning RL → Thinking Mode Fusion → General RL) establishes a decomposed capability acquisition sequence where each stage builds on specific properties of the previous one. The finding that certain capabilities are non-monotonic with respect to training—Stage 3 and Stage 4 improve mode-switching and general helpfulness while degrading specialized reasoning benchmarks (Table 22)—provides the first systematic quantification of the "alignment tax" in open-source reasoning models. Prior work (e.g., Ouyang et al., 2022; Bai et al., 2022) documented that RLHF degrades some capabilities, but Qwen3 provides granular, per-benchmark measurements across a complete post-training pipeline, showing exactly which capabilities are gained and lost at each stage. This transforms the alignment tax from a vague concern into a measurable engineering tradeoff that can inform decisions about when to stop post-training for a given deployment priority.

The strong-to-weak distillation results (Table 21) challenge a growing assumption in the field: that reinforcement learning is the most effective way to instill reasoning capabilities in smaller models. The DeepSeek-R1 project (Guo et al., 2025) applied GRPO directly to models across a range of sizes, and the R1-Distill series used SFT on R1-generated reasoning traces—both approaches implicitly treating RL or RL-trace distillation as the gold standard. Qwen3's finding that on-policy logit-level distillation from a strong teacher achieves 74.4 on AIME'24 versus 67.6 for RL (from the same starting checkpoint, at 1/10 the GPU hours) suggests that for models below a capability threshold (at least up to 8B parameters), exploration through RL is less efficient than imitation of a richer distribution. The pass@64 comparison is the diagnostic: RL improves pass@1 by narrowing the model's distribution toward its best-known reasoning patterns, but does not expand the set of problems it can potentially solve (pass@64 unchanged at 90.0). Distillation from teacher logits expands the exploration frontier (pass@64 improves from 90.0 to 93.3) because the teacher's full probability distribution conveys information about which reasoning directions are promising, not just which complete trajectories succeeded. This reframes distillation from a "cheaper but weaker" alternative to RL into a qualitatively different optimization that targets exploration breadth rather than exploitation depth.

The thinking budget mechanism introduces a new dimension to the test-time compute scaling conversation. Contemporary work on inference-time compute (e.g., Snell et al., 2024; Brown et al., 2024) focuses primarily on breadth-oriented strategies: parallel sampling, best-of-N, beam search, and iterative revision. Qwen3 demonstrates that depth-oriented compute allocation—extending the sequential reasoning chain within a single forward pass—also scales smoothly, with the added property that the scaling is user-controllable through a simple token budget rather than requiring algorithm-level changes. The paper does not compare depth-oriented against breadth-oriented strategies at matched total token budgets, leaving a central question unanswered, but by demonstrating the viability and smooth scalability of depth-oriented allocation, it establishes that this axis deserves systematic investigation alongside the better-studied breadth-oriented approaches. The emergent nature of budget control (Section 4.3: "this ability is not explicitly trained but emerges naturally") is particularly significant because it suggests that mode interpolation is a general property of models trained on boundary conditions (full thinking and zero thinking), not a capability that requires explicit per-budget training. This insight could generalize to other continuous control dimensions beyond reasoning depth.

The paper also reconciles several tensions in the prior literature:

• "Reasoning models must be separate from chat models" vs. "unified models are possible." Qwen3 resolves this in favor of unification, with the qualification that a measurable but modest performance tax (~2% on AIME'24) is paid for the integration. This suggests the prior separation was path-dependent (driven by how post-training pipelines were developed) rather than fundamental.
• "Distillation is a second-best alternative to RL for reasoning" vs. "distillation can outperform RL." Qwen3's evidence supports the latter for the 8B model on math/code tasks, with the caveat that the comparison is against one specific RL configuration (GRPO) from one specific starting point. The result is provocative but not definitive—different RL algorithms, reward designs, or model scales could change the ranking.
• "Multilingual reasoning transfer is largely emergent from pretraining scale" vs. "active post-training intervention matters." Qwen3 supports the latter: the deliberate oversampling of translation tasks in the non-thinking SFT data (Section 4.3) and the improved cross-lingual consistency on Belebele (Table 37, range narrowing from ~19 points for Qwen2.5 to ~8 points for Qwen3) suggest that post-training data composition can meaningfully shape the uniformity of cross-lingual transfer, beyond what pretraining scale alone achieves.

Follow-Up Research This Work Enables

Thinking budget vs. best-of-N reasoning at matched token budgets. The paper demonstrates that performance scales smoothly with thinking budget (Figure 2) but never compares this to spending the same total inference tokens on alternative allocation strategies. A strong follow-up would evaluate Qwen3-235B-A22B on AIME'24, MATH-500, and GPQA-Diamond under the following matched-budget conditions: (a) a single reasoning chain of length N tokens, (b) K chains of length N/K tokens each with majority voting, and (c) K chains of length N/K with best-of-N weighted selection using the model's own confidence or a separate verifier. The result would establish whether depth-oriented compute allocation (the thinking budget) is Pareto-optimal or whether breadth-oriented strategies dominate at some budget ranges. If breadth-oriented strategies prove superior, the thinking budget remains a convenience feature; if depth-oriented strategies dominate, it becomes a compute-optimal choice. The paper's existing infrastructure (the thinking budget intervention code, the benchmark evaluation pipelines) makes this experiment tractable without additional model training.

Minimum effective thinking budget and the "reasoning danger zone." The paper shows smooth performance scaling at tested budgets (Figure 2) but does not report very-low-budget behavior (e.g., 64, 128, 256, 512 tokens). A critical stress-test would evaluate Qwen3 models at these ultra-low budgets on tasks where non-thinking mode already achieves non-trivial performance (e.g., MATH-500, GSM8K, MMLU-Redux), and compare against non-thinking mode on the same queries. If low-budget thinking produces answers that are systematically worse than no thinking at all—because the model conditions on incomplete, misleading partial reasoning—this would establish a "reasoning danger zone" below which users should prefer non-thinking mode. The experiment would involve sweeping thinking budgets from 64 to 4096 tokens in doubling steps, evaluating on 500+ questions per budget, and reporting both accuracy and the fraction of cases where low-budget thinking changes a correct non-thinking answer to an incorrect one. The outcome would provide practical guidance for deployment (minimum recommended budgets per task type) and theoretical insight into whether chain-of-thought reasoning has a critical minimum length to be coherent.

On-policy distillation scaling laws across model sizes and task domains. The paper's distillation-vs-RL comparison (Table 21) is a single data point: Qwen3-8B on math and code only. A systematic follow-up would extend this comparison to the full model size range (0.6B through 30B-A3B) and to diverse task domains (general knowledge, creative writing, agent tool use, multilingual tasks) to establish whether distillation's advantage over RL is universal or size- and domain-dependent. The key hypothesis to test is that distillation dominates RL below some critical model size (where the model's independent exploration capability is weak) but RL catches up or surpasses distillation above that threshold. The experiment would train each model size under three conditions—off-policy distillation only, +RL, +on-policy distillation—evaluate on the full 23-benchmark suite, and report the crossover point (if any) where RL becomes competitive. The pass@64 metric should be reported alongside pass@1 at every size to test whether the "exploration broadening" effect of distillation persists across scales. This scaling relationship would directly inform the resource allocation decisions of teams building model families.

Domain-specific thinking tax: where does extended reasoning hurt? The paper documents that thinking mode underperforms non-thinking mode on the RULER long-context retrieval benchmark (Table 23), hypothesizing that reasoning "may instead interfere with the retrieval process." This phenomenon likely extends to other task categories. A systematic study would evaluate Qwen3-235B-A22B in both thinking and non-thinking modes on: translation (WMT test sets across multiple language pairs), factual recall (TriviaQA, Natural Questions with exact-match grading), creative writing (human evaluation of fluency and engagement), summarization (ROUGE and factuality metrics), instruction following (IFEval with format constraints), and multi-turn dialogue coherence. For each task, the experiment would measure whether thinking mode improves, degrades, or leaves unchanged the primary quality metric, and characterize the types of reasoning-induced errors (over-analysis, spurious corrections, tangential deliberation, confusion from partial reasoning). The result would produce a task × mode compatibility matrix that guides users on when to use which mode, replacing the current trial-and-error approach with evidence-based recommendations.

Mode-switching robustness under adversarial or ambiguous flags. The paper reports ThinkFollow scores of 98.9 after Stage 4 (Table 22), indicating near-perfect mode discrimination on the in-house benchmark. However, real-world deployment introduces ambiguity: users may use non-standard phrasings (e.g., "think about this" vs. the formal /think flag), the flag may be placed in confusing positions within long prompts, multiple flags may conflict, or the flag's intent may be ambiguous (e.g., "don't think too hard about this" as a colloquial request for non-thinking mode rather than a literal /no think command). A robustness study would construct an adversarial test suite including: flag variants (different phrasings, positions, languages), conflicting flags in multi-turn dialogues, flags embedded in code comments or quoted text, and user prompts that semantically request reasoning but omit the flag (e.g., "solve this step by step" without /think). The experiment would measure mode-compliance accuracy and characterize failure modes (does the model default to thinking, oscillate, or produce hybrid responses?). The results would inform safe deployment practices and potentially motivate training on a broader distribution of mode-specifying phrasings in future model versions.

Comparison of Qwen3's thinking budget to external verifier-guided search. The thinking budget mechanism operates entirely within the model's own generation—the model decides how to use the extra tokens without external feedback or selection. A complementary approach is to use an external verifier (process reward model or outcome reward model) to guide search over reasoning paths, as explored in work on test-time compute scaling. A strong follow-up would combine Qwen3's thinking mode with an external verifier (potentially a fine-tuned variant of Qwen3 itself or an existing PRM) to evaluate: (a) verifier-guided early stopping (terminate reasoning when the verifier's confidence exceeds a threshold, rather than at a fixed token budget), (b) verifier-guided best-of-N (generate multiple reasoning chains with moderate budgets and select the best via verifier score), and (c) verifier-guided beam search over reasoning steps. The comparison would use matched total FLOPs budgets across the thinking budget alone, the verifier-guided methods alone, and combinations thereof, establishing whether internal reasoning allocation and external verification are complementary or redundant. This experiment connects Qwen3's contribution to the broader test-time compute literature and tests the paper's implicit assumption that the model's internal allocation of reasoning tokens is efficient.

Practical Applications and Downstream Use Cases

Unified API serving with graduated cost tiers. Current LLM API deployments typically require separate endpoints for reasoning and non-reasoning models, each with its own pricing and latency characteristics. Qwen3 enables a single model deployment where API consumers control the cost-quality tradeoff per query via the /think flag and the thinking budget parameter. A service could offer tiered pricing: Tier 1 (non-thinking mode) for simple queries at low cost and latency (~$N$ tokens of output), Tier 2 (bounded thinking, e.g., 4,096 tokens) for moderate reasoning tasks at medium cost, Tier 3 (unbounded thinking) for hard problems at premium cost. The thinking budget scaling curves (Figure 2) provide the engineering basis for setting these tiers—the API provider can guarantee specific accuracy levels at specific budget levels for benchmarked task categories. This is substantially simpler than maintaining separate model instances and routing logic between them. The ThinkFollow score of 98.9 (Table 22) means mode misconfiguration from flag errors occurs in roughly 1% of cases, which is operationally manageable with client-side validation and retry logic.

On-device deployment with dynamic reasoning escalation. The Qwen3 edge-side models (0.6B through 4B) demonstrate functional reasoning capabilities: Qwen3-4B Thinking achieves AIME'24 of 73.8 and MATH-500 of 97.0 (Table 17), while Qwen3-1.7B Thinking achieves MATH-500 of 93.4 (Table 19). These numbers suggest that local, privacy-preserving reasoning is viable for educational and professional applications on consumer hardware (laptops, phones). A deployment architecture could run Qwen3-4B locally in non-thinking mode by default (for routine queries), switch to thinking mode with a modest budget (e.g., 2,048 tokens) when the user explicitly requests reasoning, and escalate to a cloud-hosted larger Qwen3 model (32B or 235B) only when the local model's thinking-budget performance proves insufficient. The difficulty estimation problem—deciding when to escalate—could be addressed by monitoring the local model's thinking-mode confidence (output token probabilities) or the structural properties of its reasoning chain (does it reach a definitive conclusion or trail off?). This architecture turns Qwen3 from a single-model deployment into a tiered intelligence system where compute scales with problem difficulty within and across model sizes.

Multilingual reasoning for global education and assessment. Qwen3's expanded language coverage (119 languages) combined with its strong multilingual reasoning performance makes it suitable for educational applications in low-resource language communities. The per-language results in Tables 24–35 show that Qwen3-235B-A22B Thinking achieves MT-AIME2024 scores above 73 in languages including Spanish (86.7), French (86.7), Arabic (76.7), Japanese (73.3), and Indonesian (80.0). For educational platforms serving students in these languages, Qwen3 can provide step-by-step mathematical reasoning in the student's native language without routing through English translation—preserving the pedagogical value of reasoning in the language of instruction. The thinking budget mechanism enables adaptive difficulty: easier practice problems receive shorter reasoning budgets (showing concise solution steps), while competition preparation problems receive extended budgets (showing detailed derivations and alternative approaches). The strong-to-weak distilled models (Qwen3-8B Thinking: MT-AIME2024 of 65.4, Table 17) make this feasible at lower infrastructure cost for organizations without datacenter-scale compute budgets.

Agent systems with mode-adaptive reasoning. The ToolUse benchmark results (Table 22: 85.5 in thinking mode, 86.5 in non-thinking mode after Stage 4) indicate that Qwen3 can effectively invoke tools in both modes. For complex agent applications—multi-step research tasks, code generation with execution feedback, interactive data analysis—different stages of the agent loop benefit from different reasoning depths. An agent architecture could use non-thinking mode for straightforward tool invocations (file operations, web searches, API calls where the instruction is unambiguous), thinking mode with moderate budget for planning (deciding which tools to invoke in which order), and thinking mode with extended budget for analyzing tool outputs and revising plans when execution results are unexpected. The mode-switching capability (ThinkFollow: 98.9%) allows the agent control loop to dynamically adjust reasoning depth based on the current subtask without maintaining separate models or complex routing logic. The agent's total inference cost is reduced because reasoning tokens are only spent where they provide value, rather than uniformly across all agent steps.