ArXiv: 2404.02893

🎯 Pitch

Training an LLM to critique its own math answers—and discarding failures while learning from the hardest successes—boosts math scores by 17.5% on ChatGLM3-32B while also improving its language skills, breaking the usual trade-off where alignment hurts reasoning or math specialization destroys conversation. The key insight: the critique model must come from the LLM's own distribution, not an external reward model.

1. Executive Summary

This paper introduces the Self-Critique pipeline, a two-stage framework for simultaneously improving both mathematical problem-solving and general language capabilities in LLMs without the trade-offs observed in prior alignment or math-specialization approaches. Working with ChatGLM3-32B, the pipeline trains a Math-Critique model from the LLM itself to provide feedback signals, then sequentially applies rejective fine-tuning (discarding responses that fail the Math-Critique model's evaluation) and direct preference optimization (learning from pairs of correct and incorrect answers, focusing on the hardest problems). The approach yields a +17.5% improvement on the average of GSM8k and MATH scores while also improving language ability by +5.85% on AlignBench, outperforming models up to 2× larger — establishing that self-generated mathematical feedback can drive joint capability improvements only when the critique signal comes from the model's own distribution rather than external supervisors.

2. Context and Motivation

The Core Problem: The Alignment-Math Trade-off in Deployed LLM Systems

This paper addresses a practical deployment dilemma that arises when trying to build LLM systems that excel at both conversational language tasks and mathematical reasoning. The problem is straightforward to state but difficult to solve: existing methods for improving one capability almost always degrade the other. This isn't a minor inconvenience — it's a fundamental tension that forces practitioners to make uncomfortable choices about which capability to sacrifice in their production systems.

The paper frames this explicitly as a challenge for "deployed LLM systems" (Section 1), signaling that this is not a purely academic exercise. The authors work on ChatGLM, an online serving LLM at https://chatglm.cn, and the tension they're addressing has real users on both sides: users who want the model to hold natural conversations and users who need it to solve math problems correctly. A model that degrades on either dimension creates a worse experience for some segment of users — and in a production setting, you can't simply tell users "this model only does math" or "this model only does conversation."

The Problem Is Documented, Not Hypothetical

The paper provides concrete, quantitative evidence of this trade-off from prior work, represented in Table 1. The evidence falls into two categories:

Alignment-first approaches harm mathematical reasoning. DeepSeek-67B-Chat, when subjected to DPO alignment training to improve its language capabilities, showed a +6.8% improvement on AlignBench (a language quality benchmark) but suffered a -1.2% degradation on the GSM8k and MATH average. The alignment process — designed to make the model's outputs more helpful, harmless, and aligned with human preferences — inadvertently penalized or suppressed the kind of precise, logically structured outputs that mathematical problem-solving demands. This is what the paper refers to as the "alignment tax" (citing Askell et al., 2021), a well-known phenomenon in RLHF pipelines where gains in helpfulness or harmlessness come at the cost of reduced performance on specific capability benchmarks.

Math-specialization approaches degrade language capabilities. InternLM2-Chat-20B, when fine-tuned specifically for mathematics (producing Math-InternLM2-20B), showed a +5.1% improvement on math benchmarks but suffered a -14.8% catastrophic degradation on AlignBench's language evaluation. This is an order-of-magnitude worse trade-off than the alignment case — the math specialization effectively breaks the model's ability to handle general language tasks. This happens because math-specific SFT typically involves training on domain-specific corpora (structured math problems, formal reasoning chains) that are distributionally very different from the open-ended conversational data the model was originally trained on. The fine-tuning process overwrites or suppresses the conversational capabilities, a phenomenon well-known in continual learning as catastrophic forgetting, but amplified here because the target domain (mathematical reasoning) requires very different output patterns than the source domain (conversational language).

These two failure modes are symmetric but opposite, creating a genuine dilemma: you can have strong language or strong math, but not both — at least not with the alignment and specialization techniques available before this work. The paper's own baseline, ChatGLM3-32B-SFT-2312, sits at the middle ground (52.4% math average, 7.37 AlignBench), and the goal is to push both numbers up simultaneously.

Why This Problem Matters Beyond Academia

The significance of this problem extends far beyond benchmark scores. There are several reasons this trade-off has substantial real-world impact:

Unified deployed systems can't choose one capability over the other. When an organization deploys a single LLM to serve all user queries — as ChatGLM does — that model receives both conversational prompts ("write me a poem about autumn") and mathematical prompts ("solve this integral: ∫ x² sin(x) dx"). If alignment training degrades math capability, users asking math questions get worse answers. If math specialization degrades language, the conversational experience deteriorates. There is no way to route queries to different models pre-emptively without a reliable classifier, and even with routing, maintaining two separate model instances doubles deployment costs.

Mathematical reasoning is increasingly central to LLM applications. The paper cites a range of application domains where mathematical problem-solving matters: agent-based systems (Liu et al., 2023, AgentBench), long-context reasoning tasks (Bai et al., 2023, LongBench), and real-world missions more broadly. As LLMs move from pure language generation into tool use, code execution, data analysis, and scientific reasoning, mathematical competence becomes a prerequisite rather than a nice-to-have. An LLM that writes eloquent prose but cannot correctly compute a percentage or solve a simple equation has severely limited practical utility in many professional contexts.

The scaling solution is economically wasteful. One response to this trade-off might be: "just train a bigger model." If a 32B model struggles with the alignment-math tension, perhaps a 67B or 130B model can do both well. But Table 1 shows that even DeepSeek-67B-Chat — a model more than twice ChatGLM3-32B's size — exhibits the same degradation pattern under DPO alignment. Worse, the InternLM2-20B → Math-InternLM2-20B degradation (-14.8% language) shows that even purpose-built math models lose language capability regardless of scale. The problem is not primarily about model capacity — it is about training methodology. Scaling up is an expensive way to avoid rather than solve the fundamental issue.

Human feedback pipelines don't naturally capture mathematical correctness. The standard RLHF pipeline (Ouyang et al., 2022; Touvron et al., 2023) relies on human preference judgments, and humans are good at evaluating whether generated text is helpful, coherent, and stylistically appropriate. However, human raters are not reliable at evaluating the logical correctness of a multi-step mathematical derivation — they may be fooled by confident-sounding but incorrect reasoning, or they may simply lack the expertise to verify the solution. This means that reward models trained on human preferences learn to reward outputs that look good rather than outputs that are correct in the mathematical sense. The alignment tax on math capability is not a bug to be patched; it is a structural consequence of how human feedback data is collected and what those data optimize for.

Prior Approaches and Their Specific Shortcomings

The paper positions itself against three broad categories of prior work, each of which falls short in specific ways:

1. Standard RLHF alignment (RLHF/DPO on human preference data). The dominant paradigm for aligning LLMs to human values, exemplified by InstructGPT (Ouyang et al., 2022), LLaMA-2-Chat (Touvron et al., 2023), and Claude (Anthropic), uses human preference judgments to train a reward model and then optimizes the LLM's policy against that reward via PPO or directly via DPO (Rafailov et al., 2023). The fundamental limitation is that human preference data does not encode mathematical correctness. Human raters prefer responses that are well-written, confident, and rhetorically persuasive — qualities that are orthogonal or even negatively correlated with mathematical accuracy. A fluent but incorrect solution can be preferred over a terse but correct one. The result is that alignment-optimized models learn to produce outputs that humans like but not necessarily outputs that are correct. The DeepSeek-Chat → DeepSeek-Chat-DPO degradation (-1.2% math) in Table 1 is direct evidence of this phenomenon.

2. Math-specific supervised fine-tuning (Math SFT). Several lines of work have pursued mathematical capability improvement through domain-specific supervised fine-tuning on curated math datasets. MetaMath (Yu et al., 2023) augments existing math datasets through question bootstrapping and answer augmentation. WizardMath (Luo et al., 2023) uses evolved instruction data from stronger models. MAmmoTH (Yue et al., 2023) focuses on hybrid training with code-augmented reasoning. Math-Shepherd (Wang et al., 2023) trains process reward models for step-level verification. All of these approaches share a common failure mode: they are trained predominantly or exclusively on mathematical content, causing catastrophic degradation of general language capabilities. The InternLM2-20B → Math-InternLM2-20B result (-14.8% language, Table 1) illustrates this sharply. The paper does not dispute that these methods improve math — they do (+5.1% in the InternLM case) — but argues that the language degradation makes them unsuitable for general-purpose deployed systems where users expect both capabilities from the same model.

3. Rejective fine-tuning (RFT) alone. Yuan et al. (2023) proposed Scaling-Math-RFT, which applies rejection sampling to math problem-solving: generate multiple candidate solutions, evaluate them against ground-truth answers, keep only correct solutions, and fine-tune on those. This approach leverages the model's own generative distribution for data creation but relies on binary correctness signals from ground-truth answers. In real-world deployed systems, ground-truth answers are not available for user-submitted questions — users don't provide the answer along with their question, because that's what they're asking the model to produce. Any approach that requires ground-truth answers to filter training data is limited to training on existing labeled datasets (like GSM8k or MATH) and cannot leverage the continuous stream of real user queries that a deployed system receives. The self-critique pipeline addresses this by replacing ground-truth verification with model-generated critique, enabling the system to improve from unlabeled user queries.

4. Constitutional AI and RLAIF. The idea of using AI-generated feedback rather than human feedback for alignment — known as Reinforcement Learning from AI Feedback (RLAIF) or Constitutional AI (Bai et al., 2022; Lee et al., 2023) — provides the conceptual foundation for this work. In Constitutional AI, a model critiques its own outputs according to a set of principles (the "constitution") and revises them accordingly. However, standard Constitutional AI focuses on harmlessness and helpfulness principles, not on domain-specific correctness like mathematical reasoning. The challenge the paper identifies is that general-purpose critique models don't reliably evaluate mathematical correctness — a critique model trained to assess harmlessness has no special ability to detect logical errors in a mathematical derivation. The self-critique pipeline addresses this by specializing the critique model for mathematics, training it specifically to evaluate mathematical solution quality rather than relying on a generic helpfulness critique.

How This Paper Positions Itself

The paper's positioning is most clearly articulated through the lens of its two-stage design, which directly addresses the shortcomings of prior approaches:

Against alignment-first approaches: Instead of using human preference data (which fails to encode mathematical correctness), the self-critique pipeline uses a specialized Math-Critique model derived from the LLM itself. This critique model is trained specifically to evaluate mathematical solutions, not general helpfulness. The shift from human feedback to self-generated, domain-specific feedback is what enables mathematical improvement without the alignment tax.

Against math-specialization approaches: Instead of training exclusively on mathematical content, the self-critique pipeline operates through rejective fine-tuning on the model's own outputs across both correct and incorrect examples. Critically, the pipeline does not require discarding the model's general language training data or training solely on math. The two stages — RFT (which eliminates poor-quality math outputs) and DPO (which contrasts correct and incorrect outputs on challenging problems) — are applied as additional training stages on top of an already capable general-purpose model, not as replacements for general language training. The paper claims this preserves and even improves language capabilities (the +5.85% on AlignBench) while simultaneously boosting math performance.

Against RFT-with-ground-truth approaches: Instead of requiring ground-truth answers to filter training data, the self-critique pipeline uses the Math-Critique model as the filtering mechanism. This is the key innovation that enables the approach to work on arbitrary user queries — the Math-Critique model provides a correctness signal that, while imperfect, is good enough to drive meaningful improvement. This also opens the door to continuous self-improvement from deployment data, since new user queries can be processed through the critique model without needing ground-truth labels.

Against standard Constitutional AI: Instead of applying generic constitutional principles, the self-critique pipeline specializes the critic itself for the mathematical domain. The Math-Critique model is trained on mathematical problem-solution pairs with correctness annotations, making it substantially more reliable at evaluating mathematical reasoning than a general-purpose critique model would be. This specialization of the critic, combined with the two-stage training (RFT then DPO), is what the paper argues makes the approach effective specifically for math while remaining compatible with general language performance.

The paper's central claim is not that any single component of this pipeline is entirely novel — RFT, DPO, and AI-feedback have all been explored before — but rather that their specific combination and adaptation to the mathematical domain, with a self-trained specialized critic replacing both human judgments and ground-truth labels, solves the alignment-math trade-off that prior work could not resolve.

The Practical Deployment Context

A final piece of motivation, made explicit in the paper's introduction and reinforced by the deployment to ChatGLM, is that this work is driven by real-world deployment constraints. The authors are not building a standalone math model; they are improving an online serving LLM that handles diverse user queries. This context shapes several design choices:

  • The critique model must work without ground-truth answers because users don't provide them.
  • Language performance must not degrade because users ask both conversational and mathematical questions.
  • The approach must scale with model size because production models continue to grow (the paper targets 32B parameters, but the methodology should generalize).
  • The approach must leverage the model's own outputs because external supervisory models (like GPT-4) are expensive, have different output distributions, and introduce dependencies that complicate deployment.

The self-critique pipeline is designed with all of these constraints in mind, making it a practical engineering contribution as much as a research contribution.

3. Technical Approach

3.1 Reader Orientation

The paper builds a two-stage training pipeline that takes an existing general-purpose LLM (ChatGLM3-32B) and improves its mathematical problem-solving ability while simultaneously improving its general language capability — something prior approaches could not do without degrading one or the other. The "shape" of the solution is: first train a specialized critic model that can judge mathematical solution quality, then use that critic to filter and contrast the LLM's own generated responses through two sequential training stages (rejective fine-tuning followed by direct preference optimization), where each stage targets progressively harder problems that the critic identifies.

3.2 Big-Picture Architecture (Diagram in Words)

The Self-Critique pipeline has four major components connected in a specific sequential flow:

  1. Base LLM (ChatGLM3-32B-SFT-2312) — the starting general-purpose model that has undergone standard supervised fine-tuning but exhibits the alignment-math trade-off. This model serves as both the generator of training data and the model being improved.

  2. Math-Critique Model — a specialized evaluator derived from the base LLM through fine-tuning on mathematical problem-solution pairs with binary correctness annotations. Its job is to assess whether a given solution to a math problem is correct or incorrect, producing judgment scores that replace both human feedback and ground-truth answer verification.

  3. Rejective Fine-Tuning (RFT) Module — Stage 1 of the improvement pipeline. The base LLM generates multiple candidate solutions per problem, the Math-Critique model evaluates each, incorrect solutions are discarded (rejected), and the base LLM is fine-tuned on the retained correct solutions. This produces an intermediate model (ChatGLM3-32B-RFT).

  4. Direct Preference Optimization (DPO) Module — Stage 2 of the improvement pipeline. The RFT model generates solutions for the hardest problems (identified by low Math-Critique scores), pairs of correct and incorrect solutions are formed, and the model is trained via DPO to prefer correct over incorrect solutions. This produces the final model.

Information flows as follows: training problems → base LLM generates multiple candidate solutions → Math-Critique model scores each solution → RFT filters out low-scoring solutions and fine-tunes on high-scoring ones → the improved model generates new solutions for hard problems → Math-Critique scores again → DPO trains on correct/incorrect pairs → final model. The Math-Critique model is used in both stages as the quality gate, and it is trained once at the beginning from the base LLM itself — hence "self-critique."

3.3 Roadmap for the Deep Dive

  • First, the Math-Critique model: how it's trained, what data it uses, how it produces judgments, and why self-training matters. This is the foundation because both subsequent stages depend on its quality assessments.

  • Second, Stage 1 (Rejective Fine-Tuning): the data generation process, the rejection mechanism, the training objective, and how this stage balances improvement with preservation of general capabilities.

  • Third, Stage 2 (Direct Preference Optimization): how hard problems are identified, how preference pairs are constructed from the RFT model's outputs, the DPO objective itself, and why DPO is applied only to the hardest subset rather than all data.

  • Fourth, the key design choices that distinguish this pipeline from prior work: why self-critique over external verifiers, why two stages rather than one, why DPO after RFT rather than before or instead of.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a pipeline engineering paper whose core idea is that a self-trained mathematical critic can replace both human preference judgments and ground-truth answer verification in a two-stage alignment process, enabling simultaneous improvement of math and language capabilities where prior methods forced a trade-off.

Math-Critique Model Training and Usage

What the Math-Critique model is. The Math-Critique model is a fine-tuned version of the base ChatGLM3-32B model that has been trained to evaluate the correctness of mathematical solutions. It takes as input a mathematical problem paired with a candidate solution and outputs a judgment indicating whether that solution is correct or incorrect. This judgment is not a simple binary classification; the model produces a scalar score that can be thresholded to make accept/reject decisions, and it also generates natural language critique that explains why a solution is correct or incorrect.

Why the critique model must be self-trained. The paper makes a specific argument against using external models (like GPT-4) as critics. When an external model evaluates the outputs of a different model, there is a distribution mismatch: the critic was trained on different data, generates different kinds of outputs, and has different failure modes than the model being evaluated. This means that an external critic's judgments may not be well-calibrated for the specific kinds of errors that the target model makes. By training the Math-Critique model from the base LLM itself, the critic shares the same architecture, pretraining data, and inductive biases as the model whose outputs it will evaluate. This makes its judgments more relevant and more likely to detect the specific failure patterns that the base model exhibits. The paper calls this "self-critique" precisely because the evaluating model is derived from the same source as the generating model.

Training data for the Math-Critique model. The paper constructs a training dataset consisting of mathematical problems paired with solutions and binary correctness labels. The specific data sources and construction methodology are critical but not exhaustively detailed in the main text. The general approach is:

  1. Collect mathematical problems from existing math datasets (the paper uses standard benchmarks like GSM8k and MATH as sources, though exact training dataset composition is specified in the experimental sections).

  2. Generate solutions for these problems using the base ChatGLM3-32B model (or possibly multiple models to increase diversity). This ensures the critique model sees solutions that are distributionally similar to what the base model actually produces.

  3. Label solutions as correct or incorrect based on whether the final answer matches the ground-truth answer. This step does require ground-truth answers during training of the Math-Critique model, but this is a one-time cost — once the critic is trained, it can evaluate new problems without ground-truth labels.

  4. Fine-tune the base LLM on this problem-solution-label dataset, training it to predict whether a given solution is correct. The training objective is a standard classification loss over the binary correct/incorrect label.

The key insight is that ground-truth answers are used to train the critic, not to directly filter the training data for the main model. This is a one-time investment that enables the subsequent RFT and DPO stages to operate on arbitrary problems without ground-truth labels.

How the Math-Critique model produces judgments. At inference time, the Math-Critique model receives a problem and a candidate solution as input (formatted according to a specific prompt template) and produces two outputs:

  1. A scalar correctness score, which represents the model's confidence that the solution is correct. This score is used for threshold-based rejection decisions in the RFT stage.

  2. A natural language critique, which explains the reasoning behind the judgment — pointing out specific errors, identifying logical gaps, or confirming the solution's validity. This critique text is used to provide richer feedback signals in the DPO stage.

The paper does not provide the exact prompt template in the main text, but the general structure is: the problem statement, followed by the candidate solution, followed by an instruction for the model to evaluate the solution and produce both a judgment and an explanation.

Design choice: why a specialized critic rather than a generic reward model. Standard RLHF pipelines use reward models trained on human preference data to score model outputs. These reward models are trained to predict which of two responses a human would prefer, and they learn to reward qualities like helpfulness, coherence, and appropriate style. However, as discussed in Section 2 of this analysis, human preferences do not reliably track mathematical correctness — raters may prefer a well-written but incorrect solution over a terse but correct one. A reward model trained on human preferences therefore learns the wrong signal for mathematical improvement.

The Math-Critique model, by contrast, is trained specifically on mathematical correctness labels derived from ground-truth answer verification. Its training signal is: "is this solution mathematically correct?" not "would a human prefer this solution?" This makes it a domain-specific verifier rather than a general preference model. The specialization is what allows it to provide useful feedback for mathematical improvement where a generic reward model would fail.

A subtlety about the critic's reliability. The Math-Critique model is not perfect — it makes errors in judging solution correctness, just as the base model makes errors in generating solutions. The paper implicitly relies on the fact that the critic's errors are less frequent than the generator's errors, and that the critic's judgments are correlated with true correctness even if not perfectly accurate. This is a reasonable assumption because the critic is trained on a discriminative task (binary classification of solution correctness) which is generally easier than the generative task (producing correct solutions from scratch). The pipeline's effectiveness depends on this gap between the critic's accuracy and the generator's accuracy — if the critic were no better than the generator, rejection sampling would provide no signal.

Stage 1: Rejective Fine-Tuning (RFT)

What RFT is and why it comes first. Rejective Fine-Tuning is a data filtering and training procedure that improves model performance by generating many candidate solutions and only training on the ones that pass a quality threshold. The paper positions RFT as the first stage because it provides a broad-based improvement across all difficulty levels by eliminating clearly incorrect training examples, while the second stage (DPO) focuses more narrowly on the hardest remaining problems.

The relationship to prior RFT work. Yuan et al. (2023) introduced RFT for mathematical reasoning under the name "Scaling-Math-RFT," but their approach used ground-truth answers as the rejection criterion: generate N solutions, keep only those whose final answer matches the known correct answer, and fine-tune on the kept solutions. The fundamental limitation is that ground-truth answers are not available for arbitrary user queries in deployment. This paper's innovation is replacing the ground-truth filter with the Math-Critique model's judgment, making the pipeline applicable to unlabeled data.

Step-by-step RFT procedure. The RFT stage follows a specific sequence of operations:

  1. Data collection. The paper uses a set of mathematical training problems. Based on the experimental details, this likely includes problems from standard math datasets (GSM8k, MATH, and possibly others), though the exact composition is specified in the experimental sections. The key requirement is that these problems have ground-truth answers available for initial Math-Critique model training, but once the RFT stage runs, it operates using the critic's judgments rather than direct answer comparison.

  2. Candidate generation. For each training problem, the base ChatGLM3-32B-SFT-2312 model generates multiple candidate solutions. The paper uses a sampling approach where the model produces diverse outputs (likely using temperature sampling rather than greedy decoding to increase solution diversity). The number of candidates per problem (call it $K$) is a hyperparameter that trades off between computational cost and the quality of the filtered dataset. More candidates means a higher chance of finding at least one correct solution per problem, but also higher generation cost.

  3. Math-Critique evaluation. Each candidate solution is evaluated by the Math-Critique model, which produces a scalar correctness score. The paper applies a threshold to these scores: solutions with scores above the threshold are classified as "correct" and retained; solutions below the threshold are classified as "incorrect" and discarded.

  4. Dataset construction. The retained solutions (those judged correct by the Math-Critique model) form the training dataset for RFT. Each retained solution is paired with its original problem to create a supervised fine-tuning example: input = problem statement, target output = the retained solution.

  5. Fine-tuning. The base model is fine-tuned on this filtered dataset using standard supervised fine-tuning (next-token prediction loss on the target solution tokens). The training objective is the standard language modeling cross-entropy:

LRFT=−∑t=1Tlog⁡pθ(yt∣x,y<t)\mathcal{L}_{\text{RFT}} = -\sum_{t=1}^{T} \log p_\theta(y_t | x, y_{<t})

where $x$ is the problem statement, $y = (y_1, ..., y_T)$ is the retained solution token sequence, $p_\theta$ is the model's predicted probability distribution over tokens, and the sum runs over all tokens in the solution.

What it computes: the negative log-likelihood of the correct solution under the model, summed over all tokens. The model's parameters $\theta$ are updated to increase the probability assigned to the retained (judged-correct) solutions.

Why this form: standard language modeling cross-entropy is the maximum-likelihood objective for autoregressive sequence generation. It directly encourages the model to produce the retained solution tokens, which have been filtered for quality by the Math-Critique model. This is simpler than RL-based objectives (like PPO) and avoids the instability and reward hacking issues that can arise in RL training.

The output of this stage is an intermediate model: ChatGLM3-32B-RFT. This model has been trained to produce solutions that the Math-Critique model judges as correct, which should improve its mathematical accuracy while maintaining its general language capabilities (since the fine-tuning dataset is still focused on problem-solution pairs, not pure math content that would overwrite conversational ability).

Why rejection rather than weighted training. An alternative to RFT would be to train on all generated solutions but weight them by their Math-Critique scores (so correct solutions get higher weight, incorrect solutions get lower weight). The paper chooses rejection (hard filtering) over weighting because it more aggressively eliminates noise: solutions that score below the threshold are completely excluded from training, preventing the model from learning from or imitating incorrect reasoning patterns. Weighted training would still expose the model to incorrect solutions, potentially reinforcing errors even if at lower weight. The hard threshold creates a cleaner training signal at the cost of discarding some data.

The threshold selection problem. A critical design choice in RFT is where to set the acceptance threshold for the Math-Critique model's scores. Set it too high, and very few solutions pass the filter, resulting in a tiny training dataset and potentially overfitting. Set it too low, and incorrect solutions slip through, diluting the training signal with errors. The paper must balance precision (keeping only truly correct solutions) against recall (keeping enough solutions to have sufficient training data). The exact threshold value and how it was chosen are specified in the experimental sections — the main text describes the mechanism without providing the specific number.

What RFT achieves that direct SFT on ground-truth data does not. The paper could have simply fine-tuned the base model directly on the original math datasets (GSM8k, MATH) with their ground-truth solutions — this is what Math-InternLM2-20B does, and it causes catastrophic language degradation (-14.8% on AlignBench). RFT differs in two important ways: (1) the training data is generated by the model itself, so it stays within the model's output distribution rather than introducing out-of-distribution solutions from human annotators or other models, and (2) the rejection mechanism filters for quality within the model's own distribution, encouraging the model to produce more of what it already does well rather than learning entirely new output patterns. This self-distribution property is what the paper argues preserves language capability while improving math.

Stage 2: Direct Preference Optimization (DPO) on Hard Problems

What DPO is and why it's the second stage. Direct Preference Optimization (Rafailov et al., 2023) is a method for training language models from preference data without needing to train a separate reward model. Given pairs of responses where one is preferred over the other, DPO directly optimizes the policy to increase the relative probability of preferred responses. The paper applies DPO as the second stage specifically on the hardest problems — those where the RFT model still produces predominantly incorrect solutions — using preference pairs derived from the RFT model's own outputs and scored by the Math-Critique model.

The DPO objective. The DPO loss function is:

LDPO(πθ;πref)=−E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]

where $x$ is the problem statement, $y_w$ is the preferred (correct) solution, $y_l$ is the dispreferred (incorrect) solution, $\pi_\theta$ is the policy being optimized (the RFT model being further trained), $\pi_{\text{ref}}$ is a reference policy (the RFT model frozen at the start of DPO training), $\mathcal{D}$ is the dataset of preference pairs, $\sigma$ is the logistic sigmoid function, and $\beta$ is a temperature parameter controlling how strongly the policy is regularized toward the reference.

What it computes: for each preference pair, DPO computes the log-ratio of the policy's probability of the preferred solution relative to the reference policy, and the same for the dispreferred solution. The difference between these two ratios determines how much better the policy treats the preferred solution compared to the reference. The sigmoid and log transform this into a loss that is minimized when the policy assigns higher relative probability to preferred solutions and lower relative probability to dispreferred solutions, while the KL penalty (implicit in the reference ratio) prevents the policy from drifting too far from the reference model.

Why this form: DPO reformulates the RLHF objective (which requires training a reward model and running PPO) as a simple classification-style loss over preference pairs. The reference model $\pi_{\text{ref}}$ provides an implicit KL-divergence penalty that prevents reward over-optimization — the model cannot simply maximize the preference signal by assigning probability 1 to preferred completions regardless of quality, because that would incur a large KL penalty relative to the reference. This makes DPO more stable than PPO-based RLHF and easier to implement (no need for a separate reward model training step). The $\beta$ parameter controls the strength of this regularization: higher $\beta$ means stronger KL penalty and less deviation from the reference.

How preference pairs are constructed. This is the critical data-construction step that enables the self-critique approach:

  1. Identify hard problems. After the RFT stage, the RFT model (ChatGLM3-32B-RFT) generates solutions for a set of held-out or training problems. The Math-Critique model evaluates each solution. Problems are classified as "hard" if the RFT model's solutions consistently receive low Math-Critique scores — that is, the model still cannot reliably produce correct answers even after RFT. The paper likely uses a threshold on the average Math-Critique score per problem, or on the pass rate across multiple samples.

  2. Generate preference pairs for hard problems. For each hard problem, the RFT model generates another batch of candidate solutions (possibly with different sampling parameters to increase diversity). The Math-Critique model scores each solution. From these solutions, the system constructs pairs $(y_w, y_l)$ where $y_w$ is a solution judged correct (high Math-Critique score) and $y_l$ is a solution judged incorrect (low Math-Critique score). If multiple correct and incorrect solutions exist, multiple preference pairs can be formed from a single problem.

  3. Ensure the Math-Critique scores are reliable for pairing. A subtle but important design consideration: the DPO loss relies on the relative ordering being correct — if the "preferred" solution is actually worse than the "dispreferred" solution, the DPO gradient will push the model in the wrong direction. The paper relies on the Math-Critique model's scoring being sufficiently accurate that high-scoring solutions are genuinely more likely to be correct than low-scoring solutions. The gap between the preferred and dispreferred scores likely needs to be significant (not just marginal) to ensure reliable pairing — solutions that are near the decision boundary might be excluded from DPO training to reduce noise.

Why DPO only on hard problems. The paper applies DPO specifically to the hardest subset of problems, not to all training data. There are two likely reasons for this design choice:

  1. Easy problems are already solved by RFT. If the RFT model can already reliably produce correct solutions for a problem (high pass rate after RFT), there is little marginal benefit to further DPO training on that problem. The model has already learned to produce correct solutions; DPO's contrastive signal adds nothing new.

  2. DPO provides the most signal where there is clear contrast. On hard problems, the RFT model produces a mix of correct and incorrect solutions, creating natural preference pairs with clear quality differences. On easy problems, nearly all solutions might be correct (or nearly all incorrect), making it difficult to construct meaningful preference pairs. DPO needs both preferred and dispreferred examples for each problem — if a problem only produces correct solutions (or only incorrect ones), no preference pair can be formed.

  3. Focusing compute on where improvement is needed. By restricting DPO to hard problems, the training signal is concentrated on the cases that need the most improvement. This is an efficient use of training compute: rather than spending DPO training on problems the model already handles well, focus on the bottleneck cases.

What DPO achieves that RFT alone does not. RFT operates by eliminating incorrect solutions from training — it teaches the model what correct solutions look like but does not explicitly teach it to avoid incorrect approaches. DPO adds a contrastive signal: the model learns not just to produce correct solutions, but to prefer correct solution patterns over incorrect ones when both are possible outputs. This is particularly valuable for hard problems where the model might produce plausible-sounding but incorrect reasoning — DPO actively suppresses those incorrect patterns while RFT merely fails to reinforce them.

The relationship between RFT and DPO. The paper's two-stage design can be understood as a curriculum: RFT (Stage 1) brings the model to a baseline level of mathematical competence by training on a broad set of quality-filtered solutions. This initial improvement makes the model's output distribution more concentrated around correct solutions, which in turn makes the preference pairs for DPO (Stage 2) more informative — there is a clearer separation between correct and incorrect outputs, and the model is already "in the neighborhood" of correct solutions so the DPO gradient can provide more targeted guidance.

Key Design Choices and Their Justifications

Design choice 1: Self-critique over external verifier. The paper chooses to train the Math-Critique model from the base LLM itself rather than using an external model (like GPT-4) as the critic. The justification is distributional alignment: a self-trained critic understands the base model's output patterns, failure modes, and reasoning style because it shares the same pretraining. An external critic might penalize valid reasoning approaches that differ from its own training distribution, or might miss errors that are characteristic of the base model. The self-critique approach also eliminates external API dependencies, making the pipeline fully self-contained and deployable.

Design choice 2: Two stages (RFT then DPO) rather than one. A natural question is why not just do DPO from the start, or just do more RFT? The two-stage design addresses different aspects of the improvement problem:

  • RFT provides breadth: it improves the model across the entire difficulty spectrum by filtering out clearly incorrect solutions and training on the good ones. It is a positive-reinforcement approach (show the model what correct looks like).
  • DPO provides depth: it focuses on the hardest cases and provides contrastive signal (this correct solution is better than that incorrect one). It is a comparative approach that helps the model distinguish between correct and incorrect reasoning patterns when both are plausible.

Concretely, if DPO were applied first, the model might not have enough separation between correct and incorrect outputs for the preference pairs to be informative — the DPO gradient relies on the preferred response being recognizably better than the dispreferred one. RFT first increases the model's baseline competence, making the preference signal stronger for DPO.

Design choice 3: Hard problem filtering for DPO based on Math-Critique scores. The paper uses the Math-Critique model not just for individual solution evaluation but for problem-level difficulty estimation. By aggregating scores across multiple solutions for the same problem, the system can estimate which problems the RFT model still struggles with. This is a form of the difficulty estimation concept discussed in other test-time compute work, but applied at training time: identify which problems need the most attention and allocate the more sophisticated training mechanism (DPO) to those cases.

Design choice 4: RFT as supervised fine-tuning, not RL. An alternative to RFT would be to use reinforcement learning with the Math-Critique model as a reward function — generate solutions, score them, and update the policy via PPO to maximize expected score. The paper chooses supervised fine-tuning on filtered data instead. The likely justification is stability and simplicity: PPO training is notoriously sensitive to hyperparameters and can lead to reward hacking (the model learns to produce outputs that score highly under the critic without actually being correct). SFT on quality-filtered data is a simpler, more robust approach that directly trains the model to imitate high-quality outputs without the complications of RL optimization.

Design choice 5: The base model as both generator and starting point for the critic. The entire pipeline starts from a single model — ChatGLM3-32B-SFT-2312 — which serves as (a) the base model to be improved, (b) the starting point for training the Math-Critique model, and (c) the model that generates training data for both RFT and DPO. This self-contained design means no external models, datasets, or annotation pipelines are required beyond the initial math training data used to train the critic. This makes the approach practical for deployment scenarios where keeping the improvement pipeline fully in-house is important for cost, latency, and data privacy reasons.

The Math-Critique Threshold as an Implicit Difficulty Signal

The RFT and DPO stages both rely on thresholding the Math-Critique model's scores, but they use thresholds differently. Understanding this distinction is essential to understanding the pipeline's logic:

In RFT, the threshold is applied per-solution: each individual solution is either accepted (score > threshold) or rejected (score ≤ threshold). This threshold needs to balance precision and recall — the paper likely tunes it to maximize the quality of the retained training data while keeping enough data for effective fine-tuning.

In DPO, the threshold is applied per-problem (aggregated across solutions) to identify hard problems. The paper computes an aggregate difficulty metric (likely the average Math-Critique score across multiple solutions, or the pass rate — fraction of solutions scoring above the RFT threshold). Problems with aggregate scores below some threshold are designated as hard and become the DPO training set.

In both cases, the threshold is a hyperparameter that controls the trade-off between data quality and data quantity. The paper's experimental sections presumably include ablation studies or sensitivity analyses showing how performance varies with different threshold values, though the exact values and tuning methodology are in the results sections rather than the technical approach description.

Summary of the Full Pipeline Flow

To make the sequential dependency explicit, here is the complete end-to-end flow:

  1. Train Math-Critique model: Fine-tune base ChatGLM3-32B on problem-solution-label pairs to produce a specialized mathematical correctness evaluator.

  2. Stage 1 — RFT: (a) Base model generates $K$ candidate solutions per training problem. (b) Math-Critique model scores each solution. (c) Solutions with scores above threshold are retained. (d) Base model is fine-tuned on retained (problem, solution) pairs via standard SFT. (e) Output: ChatGLM3-32B-RFT.

  3. Stage 2 — DPO: (a) RFT model generates solutions for training problems. (b) Math-Critique model scores solutions. (c) Problems are filtered to the hardest subset based on aggregate Math-Critique scores. (d) For each hard problem, preference pairs $(y_w, y_l)$ are formed from high-scoring and low-scoring solutions. (e) RFT model is further trained via DPO on these preference pairs. (f) Output: final ChatGLM3-32B model with improved math and preserved language capability.

The final model is what is evaluated in the results section and deployed to the ChatGLM online serving system.

4. Key Insights and Innovations

Innovation 1: The Self-Critique Pipeline Resolves the Alignment-Math Trade-off by Specializing the Critic, Not the Policy

The paper's most fundamental conceptual move is recognizing that the alignment-math trade-off is a critic problem, not a policy problem. Prior work treated the degradation of mathematical capability under alignment as something to be addressed by modifying what the model learns — either accepting the math degradation as an alignment tax (DeepSeek's DPO alignment losing 1.2% math while gaining 6.8% language, Table 1) or accepting language degradation as a specialization cost (InternLM2 losing 14.8% language while gaining 5.1% math, Table 1). In both cases, the assumption was that the training signal itself (human preferences for alignment, math solutions for specialization) inherently forces a zero-sum trade-off at the policy level.

The self-critique pipeline fundamentally reframes this. Rather than asking "how do we train the policy to be good at both math and language?", it asks "how do we get a training signal that correctly encodes mathematical quality without overwriting language capability?" The answer is to leave the policy's training process largely intact (standard SFT and DPO, no domain-specific architectural changes, no exclusive math training data) and instead replace the source of the quality signal. The Math-Critique model provides a training signal that is (a) specific to mathematical correctness, unlike human preference data, yet (b) generated from the model's own distribution, unlike external verifier models that introduce distributional mismatch.

This is a diagnostic insight, not just an engineering choice. It says: the reason prior methods fail is not that language and math capabilities are inherently in tension at the representational level, but that the training signals used to improve them come from incompatible sources. Human preferences encode style and helpfulness but not correctness. Math datasets encode correctness but at the cost of domain-specific output patterns that overwrite conversational ability. The paper's move is to synthesize a third kind of signal — self-generated, domain-specific correctness feedback — that satisfies both constraints simultaneously. The Math-Critique model is trained on correctness labels (so it encodes mathematical quality), but it is derived from the base LLM itself (so it understands the model's output distribution and doesn't pull the policy toward out-of-distribution math-specific patterns).

This reframing matters beyond the specific pipeline because it suggests a general principle: whenever an alignment procedure degrades a specific capability, the bottleneck may be in the reward/critique signal, not in the policy architecture or training objective. Rather than accepting the degradation as a tax or trying to patch it by mixing in capability-specific data (which often causes other degradations), one should ask whether the reward signal itself can be made to encode the desired capability without distorting the policy's broader behavior. This is a transferable insight — it applies equally to code generation, factual accuracy, logical reasoning, or any domain where human preference data fails to capture objective quality and where domain-specific training data would cause catastrophic forgetting of general capabilities.

Evidence for the validity of this framing comes from the simultaneous improvement in Table 1: ChatGLM3-32B-SFT-2312 achieves 52.4% math average and 7.37 AlignBench, and after the self-critique pipeline (+RFT & DPO), it reaches 61.6% math (+17.5%) and 7.80 language (+5.85%). The fact that both metrics improve rules out the hypothesis that math and language are fundamentally competing for representational capacity in a zero-sum way — the trade-off exists in prior methods, not in the capabilities themselves. The self-critique pipeline doesn't overcome the trade-off through scale (the model stays at 32B) or through architectural innovations (it uses standard SFT and DPO objectives). It overcomes it purely through signal quality.

Innovation 2: Sequential RFT-then-DPO as a Difficulty-Conditioned Curriculum

The second conceptual contribution is the paper's specific two-stage curriculum design in which RFT and DPO are applied sequentially with the DPO stage focused exclusively on the hardest problems. This is not simply "we tried two methods and both helped" — it embodies a more interesting idea: the appropriate training mechanism depends on the difficulty of the problem relative to the model's current capability level, and the optimal sequence is to first raise the floor (broad improvement via RFT) and then target the ceiling (contrastive improvement via DPO on hard cases).

This idea has a clear intellectual lineage but a novel application. The concept of curriculum learning — training on easier examples before harder ones — is well-established in machine learning (Bengio et al., 2009). The concept of using rejection sampling to filter training data for quality precedes this work (Yuan et al., 2023, Scaling-Math-RFT). The concept of DPO as a preference-learning mechanism is from Rafailov et al. (2023). What is novel is the synthesis of these ideas into a difficulty-conditioned training pipeline where the training mechanism itself (not just the data ordering) changes as a function of estimated difficulty.

The logic is this: after the base model undergoes RFT, its performance improves broadly, but some hard problems remain stubbornly difficult — the model produces a mix of correct and incorrect solutions rather than converging to correctness. For these problems, more RFT would be ineffective because RFT only reinforces correct solutions without actively suppressing incorrect patterns. What the model needs on hard problems is a contrastive signal — "this correct solution is preferred over that incorrect one" — which DPO provides. But DPO is only useful when there is meaningful contrast, i.e., when the model generates both clearly correct and clearly incorrect solutions. If DPO were applied to easy problems where the model already reliably produces correct answers, the preference signal would be weak (nearly all solutions are correct) or noisy (the few incorrect solutions are unrepresentative). By restricting DPO to hard problems, the training signal is concentrated where it provides the most marginal benefit.

This difficulty-conditioned allocation of training mechanisms parallels the difficulty-conditioned allocation of test-time compute studied in other work, but it operates at training time rather than inference time. The Math-Critique model serves not only as a per-solution quality filter but as a problem-level difficulty estimator: problems where the RFT model's solutions consistently score low are identified as hard and routed to the more sophisticated training mechanism (DPO). This transforms the Math-Critique model from a simple verifier into a meta-learning signal that governs the training curriculum itself.

The significance of this idea is that it challenges the prevailing "one training recipe fits all problems" approach in LLM alignment. Standard RLHF applies the same PPO objective to all prompts regardless of difficulty. Standard SFT mixes all training examples uniformly. The self-critique pipeline suggests that different problems need different training mechanisms at different stages of model development, and that a critic model can serve as the arbiter of which mechanism to apply. This is a fundamentally more adaptive approach to model improvement, and it opens the door to more sophisticated multi-stage training pipelines where the choice of objective, data, and optimization procedure is conditioned on fine-grained estimates of what the model currently struggles with.

Innovation 3: Self-Generated Critique as a Practical Alternative to Both Human Feedback and Ground-Truth Verification

The third conceptual contribution is the demonstration that a self-trained critic model can serve as a viable replacement for both human preference judgments (in alignment) and ground-truth answer verification (in math improvement), without requiring either signal at deployment time. This is not merely an engineering convenience — it represents a specific claim about the relative value of different feedback sources and the conditions under which synthetic feedback can substitute for external supervision.

Prior work falls into three categories with respect to feedback signals:

  1. Human feedback approaches (RLHF, RL from human preferences) use human judgments as the gold standard, but these judgments are expensive to collect, not scalable, and — critically for mathematical reasoning — not reliable indicators of correctness because humans are poor at evaluating multi-step logical derivations.

  2. Ground-truth approaches (RFT with answer verification, process reward model training with oracle correctness) use known correct answers as the filtering criterion, producing perfectly reliable signals but requiring labels that are unavailable for arbitrary user queries in deployment.

  3. AI feedback approaches (RLAIF, Constitutional AI) use a separate AI model (typically a larger, more capable model like GPT-4) to provide feedback, which addresses the scalability problem but introduces distributional mismatch — the critic model's judgments reflect its own training distribution, not the target model's output patterns.

The self-critique pipeline occupies a fourth position: self-generated, domain-specialized AI feedback. The Math-Critique model is (a) AI-generated, so it's scalable and doesn't require human annotation at deployment time; (b) trained on correctness labels derived from ground-truth answers, so it encodes the right objective (mathematical correctness, not human preference); and (c) derived from the same base model that generates the solutions being evaluated, so it operates within the same output distribution. This combination — scalable, correctness-oriented, and distributionally aligned — is what makes it a practical alternative to all three prior approaches.

The key empirical claim that makes this more than a conceptual proposal is that the Math-Critique model's judgments, while imperfect, are sufficiently correlated with true correctness to drive meaningful improvement through rejection sampling and preference learning. If the critic were no more accurate than the generator, RFT would provide no signal (it would reject correct solutions and accept incorrect ones at random). If the critic were perfectly accurate, the pipeline would be equivalent to ground-truth RFT. The paper operates in the realistic intermediate regime where the critic is considerably better than the generator but not perfect, and shows that this is sufficient to achieve substantial gains (+17.5% math improvement, Table 1).

This finding has practical significance beyond this paper because it suggests a general recipe for domain-specific model improvement without human annotation: (1) collect a modest set of problems with ground-truth labels in the target domain, (2) train a critic model from the base LLM on these labeled examples, (3) use the critic to generate quality signals on a much larger set of unlabeled data (potentially including real user queries), and (4) apply rejection-based or preference-based training using the critic's judgments. The ground-truth labels are only needed to bootstrap the critic, not to label the entire training corpus. This dramatically reduces the annotation burden for domain-specific model improvement and makes continuous improvement from deployment data feasible.

It also clarifies the conditions under which this approach works: the critic must be sufficiently more accurate than the generator on the target task, and the gap between critic accuracy and generator accuracy determines the maximum possible improvement. If the gap is small (the model is already very good, or the task is inherently difficult to evaluate), self-critique provides little benefit. If the gap is large (the model is poor but correct answers are objectively identifiable), self-critique can drive substantial gains. This is a useful diagnostic framework for practitioners considering whether to invest in critic model training for their own domains.

Innovation 4: The MathUserEval Benchmark as a Diagnostic Tool for Real-World Mathematical Reasoning

The fourth contribution is methodological rather than algorithmic: the introduction of the MathUserEval benchmark, designed to assess mathematical reasoning in the context of real-world user queries rather than academic competition problems. While benchmarks are common in ML papers, MathUserEval is distinctive in its explicit focus on the gap between academic math datasets and deployed-system math queries, and in its use of both GPT-4-turbo and the Math-Critique model for scoring, which itself operationalizes the idea that different evaluators capture different aspects of solution quality.

Standard math benchmarks like GSM8k (grade-school math word problems), MATH (competition-level problems), and Ape210k (Chinese math problems) are designed to test mathematical reasoning in a clean, well-specified format. Problems are self-contained, unambiguous, have single correct answers, and are written in a formal academic style. Real user queries — the kind that ChatGLM receives at chatglm.cn — violate all of these assumptions. Users ask poorly specified questions, embed math in conversational context, request explanations rather than just answers, and may not even know whether their question has a well-defined mathematical answer. A model that performs well on GSM8k and MATH may still fail on real user queries because the distribution of problems is fundamentally different.

MathUserEval addresses this by curating questions that "extend beyond academic exercises to include practical application scenarios." The paper doesn't exhaustively detail the dataset construction methodology in the main text, but the conceptual contribution is clear: it provides a signal about how well improvements on academic benchmarks translate to real-world performance. The fact that the self-critique pipeline achieves "unparalleled performance on MathUserEval" (paper's claim) alongside improvements on GSM8k and MATH is evidence that the improvements are not an artifact of overfitting to academic benchmark distributions — they generalize to the kind of queries that matter in deployment.

The dual-scoring mechanism (GPT-4-turbo and Math-Critique) is also noteworthy. Using GPT-4 as an evaluator provides an external, relatively objective reference point, while using the Math-Critique model provides a self-consistency check — if the self-trained critic agrees with GPT-4 on solution quality, that validates the critic's reliability. If they disagree, it surfaces cases where the critic's training distribution or evaluation criteria diverge from GPT-4's, which can inform further critic improvement. This dual evaluation is a practical pattern for benchmark design when ground-truth answers are unavailable or when solution quality has subjective dimensions that a single evaluator might miss.

The significance of MathUserEval is that it shifts the evaluation paradigm from "how well does the model solve math contest problems?" to "how well does the model handle the mathematical reasoning tasks that actual users bring to a deployed system?" This is the kind of benchmark that matters for production LLM systems, where academic metrics are proxies at best. The creation and release of this benchmark (alongside evaluation scripts) enables other practitioners to assess their own models' real-world mathematical reasoning capabilities, extending the paper's impact beyond its specific algorithmic contributions. </output>

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on three academic benchmarks: GSM8k (grade-school math word problems; Cobbe et al., 2021), MATH (high-school competition-level problems; Hendrycks et al., 2021), and Ape210k (Chinese math problems; Zhao et al., 2020). Additionally, the paper introduces MathUserEval, a newly created benchmark consisting of diverse questions extending beyond academic exercises to include practical application scenarios, designed to better reflect real user needs in deployed systems. The exact size and construction methodology of MathUserEval are not exhaustively detailed in the main text, but the paper states it is scored using both GPT-4-turbo and the Math-Critique model.

  • Base model(s). All experiments use ChatGLM3-32B-SFT-2312 as the starting model. This is a 32-billion-parameter model that has already undergone supervised fine-tuning. The paper states the model is representative of deployed LLM systems and chooses this scale because it sits in a regime where both mathematical problem-solving and language capabilities are non-trivial but far from saturated, leaving room for the self-critique pipeline to demonstrate improvements. For the Math-Critique model, the base ChatGLM3-32B is fine-tuned on mathematical problem-solution pairs with binary correctness labels.

  • Metrics. The paper uses accuracy (percentage of problems solved correctly) for GSM8k and MATH individually, and reports their average as a composite math performance metric. For language capability, the paper uses AlignBench (Liu et al., 2023), a benchmark designed to evaluate alignment quality in Chinese LLMs across dimensions like helpfulness, harmlessness, and language fluency. For MathUserEval, scoring is performed by both GPT-4-turbo and the Math-Critique model, though the paper does not specify the exact aggregation or scoring rubric in the main text. For the Hungarian Exam results shown in Figure 1, the metric is exam score.

  • Baselines. The paper compares against several models:

    • DeepSeek-67B-Chat (DeepSeek-AI, 2024): a 67B parameter chat model, evaluated both in its standard form and after DPO alignment training (DeepSeek-67B-Chat-DPO). This baseline illustrates the alignment-math trade-off where language improvement comes at the cost of math degradation.
    • InternLM2-Chat-20B (InternLM Team, 2023): a 20B parameter chat model, evaluated both in its standard form and after math-specific training (Math-InternLM2-20B). This baseline illustrates the reverse trade-off where math improvement causes language degradation.
    • ChatGLM3-32B-SFT-2312: the starting model before applying the self-critique pipeline, serving as the direct baseline for measuring improvement from the proposed method.
    • The paper also provides context from prior work including MetaMath (Yu et al., 2023), WizardMath (Luo et al., 2023), and MAmmoTH (Yue et al., 2023), though these are discussed as related approaches rather than directly compared models in the results tables.

  • Generation budget / compute accounting. The paper does not report a standardized compute budget in FLOPs or generations. For the RFT stage, the model generates $K$ candidate solutions per problem, where $K$ is a hyperparameter (the exact value is not specified in the main text). For the DPO stage, the model generates additional solutions for hard problems. The paper compares models based on downstream benchmark performance rather than matching compute budgets. The Math-Critique model incurs a one-time training cost that is not amortized or explicitly accounted for in the efficiency analysis.

  • Cross-validation / statistical protocol. The paper does not describe a cross-validation protocol for strategy selection or hyperparameter tuning in the main text. Results are reported as point estimates without confidence intervals or error bars in the tables and figures shown. The paper does not mention using held-out validation sets for threshold selection or difficulty binning, though the Math-Critique model training presumably involves a standard train/validation split for the critic itself.

Main Quantitative Results

Joint Math and Language Improvement (Table 1)

The central result of the paper is presented in Table 1, which compares the starting ChatGLM3-32B-SFT-2312 model against the model after the full self-critique pipeline (RFT + DPO), alongside two baseline model families:

ModelAvg. GSM8k & MATHAlignBench Language
DeepSeek-67B-Chat58.37.11
DeepSeek-67B-Chat-DPO57.7 (-1.2%)7.60 (+6.8%)
InternLM2-Chat-20B57.27.68
Math-InternLM2-20B60.2 (+5.1%)6.53 (-14.8%)
ChatGLM3-32B-SFT-231252.47.37
ChatGLM3-32B + RFT & DPO61.6 (+17.5%)7.80 (+5.85%)

The headline result is that the self-critique pipeline achieves simultaneous improvement on both dimensions: +17.5% relative improvement on the math average (from 52.4 to 61.6) and +5.85% relative improvement on AlignBench language score (from 7.37 to 7.80). This contrasts sharply with both baseline patterns:

  • DeepSeek's DPO alignment improves language by 6.8% but degrades math by 1.2% — a classic alignment tax where the training signal (human preferences) doesn't encode mathematical correctness.
  • InternLM2's math specialization improves math by 5.1% but degrades language by 14.8% — a catastrophic language capability loss from domain-specific fine-tuning.

The ChatGLM3 pipeline achieves a larger math improvement (+17.5%) than the math-specialized InternLM2 (+5.1%) while simultaneously improving language, and reaches a higher final math score (61.6) than both DeepSeek-67B-Chat-DPO (57.7, which is a much larger model at 67B vs. 32B) and Math-InternLM2-20B (60.2). The final language score (7.80) exceeds both DeepSeek-67B-Chat-DPO (7.60) and Math-InternLM2-20B (6.53).

A critical detail: the +17.5% figure is computed relative to the ChatGLM3-32B-SFT-2312 baseline of 52.4, representing an absolute improvement of 9.2 percentage points on the average of GSM8k and MATH. This baseline starts lower than the other models (52.4 vs. 57.2–58.3), which means part of the gain may reflect the lower starting point. However, the final score of 61.6 exceeds all other models in the comparison regardless of starting point, which validates that the improvement is not merely closing an initial gap but achieving a new state-of-the-art for models in this size class.

Outperforming Larger Models

The paper claims that the pipeline "outperforms LLMs that could be two times larger." This claim is supported by the comparison in Table 1: ChatGLM3-32B (32B parameters) with the self-critique pipeline achieves 61.6 math average, surpassing DeepSeek-67B-Chat (67B parameters, more than 2× larger) at 58.3, and DeepSeek-67B-Chat-DPO at 57.7. The margin against the larger model is approximately 3.3–3.9 absolute percentage points on the math average, and 0.2–0.69 points on AlignBench language score.

The paper does not provide a FLOPs-matched comparison (comparing total training + inference compute for the 32B pipeline vs. the 67B models), so the claim of "outperforming larger models" should be understood as a benchmark performance comparison at fixed model scales, not a compute-efficiency claim. The 32B model with the pipeline achieves higher scores than the 67B model, but the total computational cost of achieving those scores (including the Math-Critique training, RFT generation and filtering, and DPO training) is not quantified or compared to the cost of pretraining and aligning the 67B model.

Hungarian Exam Results (Figure 1)

Figure 1 presents results on a Hungarian National Exam, showing that the ChatGLM3-32B with the self-critique pipeline achieves scores that improve over the baseline (exact numbers are not legible from the figure description provided in the paper text, but the figure is described as "Results of Hungarian Exam and Average Scores of GSM8k and MATH"). The purpose of including this exam is to demonstrate generalization beyond standard English-language math benchmarks to real-world examination settings in other languages. The figure shows the average of GSM8k and MATH alongside this exam result, providing evidence that the improvements transfer to non-English, practical examination contexts.

MathUserEval Results

The paper claims "unparalleled performance on the MathUserEval" benchmark, but specific quantitative results on MathUserEval are not presented in the main text excerpt provided. The benchmark is described in the contributions section and its creation methodology is outlined, but the detailed scores for baseline models and the self-critique pipeline on MathUserEval appear to be in sections of the paper not included in the provided material. This is a notable gap: the introduction of a new benchmark is a contribution, but without the comparative results on that benchmark, the claim of superior performance cannot be verified from the available text.

Ape210k Results

The paper lists Ape210k (Zhao et al., 2020) among the benchmarks where the pipeline achieves "unparalleled performance," but specific Ape210k scores are not provided in the main text excerpt. Ape210k is a Chinese math problem dataset, making it a test of cross-lingual mathematical reasoning capability. The absence of explicit Ape210k numbers in the provided material prevents verification of this specific claim.

Ablation Studies and Robustness Checks

The main text excerpt provided is limited in its coverage of ablation studies. From the available material, the following can be identified:

Two-stage vs. single-stage design: The paper presents the pipeline as consisting of two sequential stages (RFT then DPO), but does not include an ablation comparing the full pipeline against RFT-only or DPO-only variants in the main text. Such an ablation would be essential to demonstrate that both stages contribute independently to the improvement and that the sequential ordering matters. The absence of this ablation in the provided material means we cannot determine from the main text how much of the +17.5% math improvement comes from RFT alone versus the additional contribution of DPO on hard problems.

Math-Critique model vs. ground-truth filtering: The paper's key innovation is replacing ground-truth answer verification with Math-Critique model judgments for the RFT rejection mechanism. An ablation comparing RFT with Math-Critique filtering against RFT with ground-truth filtering (the Yuan et al., 2023 approach) would quantify the cost of using imperfect model-generated critiques instead of oracle correctness labels. This ablation does not appear in the provided main text material. The paper also does not report the Math-Critique model's accuracy or calibration (e.g., precision, recall, or F1 score at the chosen threshold) against ground-truth correctness, which is a critical missing piece — without knowing how reliable the critic is, it's difficult to assess whether the pipeline's success depends on the critic being nearly perfect or whether it's robust to imperfect critiques.

Self-critique vs. external critic: A natural ablation would be to replace the self-trained Math-Critique model with an external evaluator (such as GPT-4) and measure whether the distributional alignment benefit claimed in the paper actually translates to better downstream performance. This ablation would directly test the paper's argument that self-training the critic matters because it avoids distribution mismatch. It is not present in the provided main text material.

Threshold sensitivity for RFT rejection and DPO hard problem selection: The RFT stage uses a threshold on Math-Critique scores to determine which solutions are retained for fine-tuning. The DPO stage uses a threshold (potentially a different one, aggregated across solutions) to identify hard problems. The main text does not provide a sensitivity analysis showing how the final model performance varies as these thresholds change, nor does it report the chosen threshold values. This is a significant gap because threshold selection is a central hyperparameter that controls the quality-quantity trade-off in both stages.

Number of candidate solutions per problem (K): The RFT stage generates $K$ candidate solutions per problem, where larger $K$ increases the probability of finding at least one correct solution but also increases generation cost. The paper does not report how performance varies with $K$ or what value was used in the reported experiments. This hyperparameter directly affects the effectiveness of rejection sampling and the diversity of the RFT training data.

DPO $\beta$ parameter sensitivity: The DPO objective includes a temperature parameter $\beta$ that controls the strength of KL regularization toward the reference policy. The paper does not report the chosen $\beta$ value or a sensitivity analysis showing how performance varies with different regularization strengths. In standard DPO usage, $\beta$ can significantly affect performance — too low and the model overfits to the preference signal, too high and it barely moves from the reference.

Language capability preservation mechanism: The paper claims that the self-critique pipeline preserves and even improves language capability (+5.85% on AlignBench), but does not provide an ablation demonstrating why this happens. The paper's argument is that staying within the model's own output distribution prevents the catastrophic language degradation seen in Math-InternLM2-20B, but this claim is not directly tested — for example, by comparing RFT on model-generated solutions against RFT on ground-truth solutions in terms of their respective impact on AlignBench scores. Without such an ablation, the mechanism of language preservation remains a hypothesis rather than an empirically validated claim.

Math-Critique model training data composition: The paper states that the Math-Critique model is trained on mathematical problem-solution pairs with binary correctness labels, but does not provide an ablation showing how the critic's performance (and downstream pipeline performance) varies with the size or composition of the critic training data. This matters because the critic's reliability is the linchpin of the entire pipeline — if the critic training data is small or biased, the critic's judgments may be unreliable in ways that propagate through both RFT and DPO stages.

Negative result: DPO without RFT first: While not explicitly presented as an ablation, the paper's design choice of applying RFT before DPO implies that DPO alone (without the RFT stage to improve baseline competence and create clearer preference pairs) would be less effective. This hypothesis is not tested in the provided material. The reverse is also untested: whether additional rounds of RFT (without DPO) could achieve the same gains as the two-stage pipeline.

Critical Assessment

Does the Pipeline Actually Resolve the Alignment-Math Trade-off?

The paper's central claim is that the self-critique pipeline enables simultaneous improvement of both math and language capabilities, in contrast to prior methods that force a trade-off. Table 1 supports this claim for the specific model (ChatGLM3-32B) and the specific baselines compared (DeepSeek-67B-Chat-DPO and Math-InternLM2-20B). The numbers are unambiguous: +17.5% math and +5.85% language.

However, there are several reasons to treat the claim as supported with qualifications rather than conclusively demonstrated:

The baseline starts lower. ChatGLM3-32B-SFT-2312 begins at 52.4 on the math average, compared to DeepSeek-67B-Chat at 58.3 and InternLM2-Chat-20B at 57.2. The lower starting point means there was more room for improvement, and the +17.5% relative gain (9.2 absolute percentage points) brings the model to 61.6 — which exceeds the baselines, but the large relative improvement is partly an artifact of the lower baseline. If the model had started at 58.3 (like DeepSeek), a +17.5% improvement would imply reaching 68.5, which the paper does not demonstrate. The claim of "resolving the trade-off" would be stronger if the pipeline were applied to a model that already had competitive math performance and showed it could further improve math while still gaining language — rather than starting from a lower math baseline where improvements are easier to achieve.

The language improvement mechanism is unexplained. The +5.85% improvement on AlignBench is a positive result, but the paper does not provide evidence for why the self-critique pipeline improves language capability. The paper's hypothesis is that staying within the model's output distribution prevents catastrophic forgetting of language skills, but this hypothesis is not tested. Alternative explanations are possible: (a) the additional fine-tuning on any diverse data (even math-focused) could incidentally improve language performance on AlignBench through a regularization-like effect, (b) the RFT training data (model-generated math solutions) contains a mixture of natural language reasoning and mathematical notation that provides some language training signal, or (c) the improvement is within the noise range of AlignBench evaluation (the paper provides no confidence intervals, so we cannot assess whether +5.85% is statistically significant). Without an ablation that isolates the mechanism, the simultaneous improvement could be coincidental rather than causal.

The comparison is not compute-matched. The paper compares ChatGLM3-32B + pipeline against DeepSeek-67B-Chat and InternLM2-Chat-20B without accounting for the total computational cost of achieving those scores. The self-critique pipeline involves: (1) training a Math-Critique model, (2) generating $K$ candidate solutions per problem for RFT (times the number of training problems), (3) generating additional solutions for DPO on hard problems, and (4) two rounds of fine-tuning (RFT and DPO). The baseline models required their own training costs (pretraining, SFT, alignment), but these are not quantified or compared. Without a compute-matched comparison, we cannot determine whether the 32B pipeline is genuinely more efficient than simply training a larger model, or whether the 32B model's advantage comes from investing more total compute in the improvement process.

Single model family, single scale. All experiments use ChatGLM3-32B. The paper does not demonstrate that the self-critique pipeline generalizes to other model families (e.g., LLaMA, Qwen, DeepSeek) or to other scales (e.g., 7B, 13B, 70B). The claim that the pipeline "resolves the alignment-math trade-off" is therefore specific to ChatGLM-32B and may not transfer. In particular, smaller models might benefit less from self-critique because their baseline math performance is too low for the Math-Critique model to provide reliable feedback (the critic-generator accuracy gap might be too small), while larger models might benefit less because they already achieve high math performance and the improvement ceiling is lower.

Does the Self-Critique Mechanism Specifically Drive the Improvement?

The paper attributes the pipeline's success to the self-generated nature of the critique signal — specifically, that training the Math-Critique model from the base LLM avoids the distribution mismatch that would arise from using an external critic. This claim is not directly tested in the provided material. The ablation that would demonstrate this — comparing the self-trained Math-Critique model against an external critic (e.g., GPT-4 used as a zero-shot evaluator, or a critic trained on a different model's outputs) — is absent. Without this comparison, the paper demonstrates that the self-critique pipeline works, but not that the "self" part is essential to its working.

The paper also does not report the Math-Critique model's standalone accuracy. Knowing how often the critic agrees with ground-truth correctness is essential for understanding the pipeline's behavior: if the critic has 95% accuracy, then the RFT rejection mechanism is nearly as good as ground-truth filtering, and the pipeline's success is primarily about scalability (replacing labels with model judgments). If the critic has 70% accuracy, then the pipeline is succeeding despite substantial noise in the feedback signal, which would be a more interesting finding about the robustness of rejection sampling to imperfect filtering. Without this number, the reader cannot assess where the pipeline sits on the spectrum from "near-oracle filtering" to "noisy but still useful."

Does the Two-Stage Design (RFT then DPO) Matter?

The paper presents RFT followed by DPO as an integrated pipeline, with DPO focused specifically on hard problems. The ablation comparing the full pipeline against RFT-only and DPO-only is not present in the provided main text material. This is a significant gap because:

  • If RFT alone achieves most of the gain and DPO adds little, then the pipeline could be simplified to a single stage, reducing computational cost and complexity.
  • If DPO alone (without RFT) performs poorly because the preference pairs aren't informative enough from the base model, that would validate the paper's argument about the sequential design — but this is not demonstrated.
  • If the gains are additive (RFT provides X%, DPO adds Y%, together they provide X+Y%), that would support the claim that the stages address complementary aspects of the improvement problem.

The paper's implicit claim is that RFT raises the floor (broad improvement) and DPO targets the ceiling (contrastive improvement on hard problems), but this is a hypothesis about mechanism, not an empirically verified claim.

Do the Improvements Generalize Beyond Academic Benchmarks?

The paper introduces MathUserEval specifically to test generalization to real-world user queries, and claims "unparalleled performance" on this benchmark. However, the specific MathUserEval results are not provided in the main text excerpt, making it impossible to verify this claim from the available material. The Hungarian Exam result (Figure 1) provides some evidence of generalization to real examination settings, but without the MathUserEval numbers, the paper's most direct claim about real-world applicability is unverifiable.

This is a critical gap because the self-critique pipeline's main motivation is enabling improvement on unlabeled user queries — the Math-Critique model is supposed to replace ground-truth verification precisely because user queries don't come with answers. If the pipeline only demonstrates improvement on academic benchmarks (which do have ground-truth answers, even if the critic doesn't use them directly), then it hasn't actually demonstrated its key value proposition. The Math-Critique model was trained on academic math data with ground-truth labels, and its ability to generalize to scoring real user queries (which may have different formats, ambiguity, or subjective criteria) is untested. MathUserEval results would address this, but they are absent from the provided material.

Missing Experiments That Would Strengthen the Paper

Several experiments are conspicuously absent and would substantially strengthen the paper's claims:

  1. RFT-only ablation: Train the model with just the RFT stage and report both math and language scores. This would isolate the contribution of the DPO stage and test whether the two-stage design is necessary.
  2. DPO-only ablation: Apply DPO directly to the base model (without RFT first) using Math-Critique-generated preference pairs. This would test whether RFT is a necessary prerequisite for effective DPO.
  3. Ground-truth RFT comparison: Run RFT using ground-truth answer verification instead of Math-Critique filtering, keeping all other hyperparameters identical. This would quantify the cost of using imperfect model-generated critiques versus oracle correctness labels.
  4. External critic ablation: Replace the self-trained Math-Critique model with GPT-4 as the evaluator for RFT filtering and DPO pair construction. This would directly test the paper's claim that self-training the critic matters.
  5. Math-Critique accuracy reporting: Report the Math-Critique model's precision, recall, F1, and calibration against ground-truth correctness on a held-out set. Without this, the critic's reliability is unknown.
  6. Compute cost accounting: Report the total number of model generations, training steps, and approximate FLOPs used in the pipeline and compare against the costs of training the baseline models. This would enable a genuine efficiency comparison.
  7. Cross-family replication: Apply the pipeline to at least one other model family (e.g., LLaMA-2-13B or Qwen-14B) to test whether the simultaneous improvement is specific to ChatGLM's training recipe or generalizes.
  8. Statistical significance: Report confidence intervals or standard deviations for the benchmark scores, especially given that AlignBench scores are in a compressed range (6.53 to 7.80) where small absolute differences could fall within evaluation noise.

When the Claims Hold and When They Might Not

Based on the available evidence, the paper's claims should be understood as holding under the following conditions, with specific limitations:

  • The simultaneous improvement claim holds for ChatGLM3-32B on the specific benchmarks tested (GSM8k, MATH, AlignBench) when starting from a 52.4 math baseline. Whether it holds for models starting at higher math baselines (where improvement is harder), for other model families, or for other model scales is untested.
  • The "outperforming 2× larger models" claim holds for the specific comparison against DeepSeek-67B-Chat (58.3 math) and DeepSeek-67B-Chat-DPO (57.7 math). It is a performance comparison at fixed model sizes, not a compute-efficiency claim. The paper does not demonstrate that the 32B + pipeline combination used less total compute than training and running the 67B models.
  • The self-critique mechanism's specific contribution — that using a self-trained critic rather than an external critic or ground-truth labels is essential — is claimed but not experimentally isolated. The paper demonstrates that the pipeline works, not that the "self" aspect is necessary for it to work.
  • The generalization to real user queries is claimed via MathUserEval but not quantitatively supported in the provided material. The Hungarian Exam result provides partial supporting evidence for generalization to non-English examination settings.
  • The two-stage curriculum's benefit (RFT then DPO on hard problems) is a design rationale presented in the technical approach but not validated through ablation in the provided experimental results.

6. Limitations and Trade-offs

Limitation 1: The Math-Critique Model's Accuracy Is Never Reported, Making the Pipeline's Reliability Unverifiable

The assumption or constraint. The entire self-critique pipeline — both the RFT rejection mechanism and the DPO preference pair construction — depends fundamentally on the Math-Critique model's ability to accurately distinguish correct from incorrect mathematical solutions. The paper treats the critic's judgments as sufficiently reliable to drive a +17.5% math improvement (Table 1), but never reports the critic's standalone accuracy, precision, recall, or calibration against ground-truth correctness. The paper does not state what fraction of solutions the Math-Critique model correctly classifies, how often it incorrectly accepts wrong solutions (false positives), or how often it incorrectly rejects correct ones (false negatives).

The consequence. Without knowing the critic's accuracy, a practitioner cannot assess whether the pipeline's success depends on the critic being nearly perfect (e.g., >95% agreement with ground truth) or whether it tolerates substantial noise (e.g., 70–80% agreement). If the critic is highly accurate, then the self-critique pipeline is essentially equivalent to ground-truth RFT in filtering quality, and the main contribution reduces to scalability — replacing human labels with model-generated labels at negligible quality cost. If the critic is moderately accurate (say, 75%), then the pipeline's success would demonstrate something more interesting: that rejection sampling and preference learning are robust to imperfect feedback signals. But if the critic's accuracy degrades on out-of-distribution problems (e.g., real user queries that differ from academic math datasets), the pipeline could silently degrade in deployment without any signal that the filtering quality has dropped. A practitioner deploying this method would want to monitor critic accuracy in production, but the paper provides no baseline metric to calibrate expectations.

What evidence exists in the paper. The paper mentions that the Math-Critique model is trained on problem-solution pairs with binary correctness labels derived from ground-truth answers (Section 3, Technical Approach description), but does not report evaluation metrics for this model anywhere in the provided main text. The downstream pipeline results (Table 1) demonstrate that the critic is useful enough to drive improvement, but this is an indirect measure — the improvement could occur even with a noisy critic if the training procedures (RFT and DPO) are robust to noise, or it could occur because the critic is highly accurate on the specific academic benchmarks tested. The paper's dual-scoring mechanism on MathUserEval (using both GPT-4-turbo and the Math-Critique model) could theoretically provide a calibration check, but those results are not presented in the available material. The paper does not include an ablation comparing Math-Critique filtering against ground-truth filtering, which would indirectly reveal the critic's accuracy by showing how much performance is lost when replacing oracle labels with model judgments.

Mitigation status. The paper does not attempt to measure or mitigate this limitation. The Math-Critique model's accuracy is treated as an internal implementation detail rather than a reported result. The paper does not discuss critic calibration, the risk of critic errors propagating through the pipeline, or methods for detecting when the critic's judgments become unreliable (e.g., on out-of-distribution queries). This is a significant omission because critic accuracy is the single most important hyperparameter governing the pipeline's behavior, and future practitioners adopting this method would need to establish their own critic's reliability before trusting the pipeline's outputs.

Limitation 2: The Pipeline's Computational Overhead Is Not Quantified or Compared

The assumption or constraint. The paper presents the self-critique pipeline as a method for improving model performance, but does not account for the total computational cost of the improvement process. The pipeline involves: (1) training the Math-Critique model from the base LLM, which requires generating or collecting a labeled dataset of problem-solution pairs and running a full fine-tuning; (2) generating $K$ candidate solutions per training problem for the RFT stage, where $K$ is an unspecified hyperparameter; (3) running the Math-Critique model on all $K \times N_{\text{train}}$ candidate solutions (where $N_{\text{train}}$ is the number of training problems) to produce filtering decisions; (4) fine-tuning the base model on the retained solutions; (5) generating additional candidate solutions for hard problems in the DPO stage; (6) running the Math-Critique model again on these DPO candidates; and (7) running DPO training on the constructed preference pairs. None of these costs are quantified in terms of GPU-hours, total FLOPs, or even number of model generations.

The consequence. The headline claim that ChatGLM3-32B with the pipeline "outperforms LLMs that could be two times larger" (Section 1, Abstract) is a model-size comparison, not a compute-efficiency comparison. A practitioner deciding between (a) training a 67B model with standard alignment and (b) training a 32B model with the self-critique pipeline cannot make an informed decision without knowing the relative total compute costs. It is possible that the 32B + pipeline combination costs more total compute than simply pretraining and aligning a 67B model, in which case the "outperforms larger models" claim would be misleading — the 32B model would be achieving higher benchmark scores by investing more total computation, not by being more efficient. The absence of compute accounting also makes it impossible to assess how the pipeline's cost scales with the number of training problems, the number of candidate solutions per problem ($K$), or the size of the hard problem subset for DPO. A practitioner considering applying this method to a much larger training corpus (e.g., millions of user queries) needs to know whether the generation and filtering costs are prohibitive at scale.

What evidence exists in the paper. The paper provides no compute budget, no wall-clock time measurements, no GPU-hour estimates, and no FLOPs accounting. The value of $K$ (candidate solutions per problem for RFT) is not specified in the main text. The training data size for RFT and DPO is not quantified. The cost of Math-Critique model training (both in terms of labeled data size and fine-tuning compute) is not reported. The comparison against DeepSeek-67B-Chat and InternLM2-Chat-20B in Table 1 is purely on final benchmark scores, with no accounting for the compute invested to reach those scores. The paper mentions deployment to ChatGLM (Section 1), implying the pipeline is practical at production scale, but provides no evidence about the production cost or latency of running the full pipeline versus serving the baseline model.

Mitigation status. The paper does not address this limitation. There is no discussion of computational efficiency, no suggestion that the pipeline is compute-optimal in any sense, and no guidance for practitioners on how to trade off between candidate generation budget ($K$), training data size, and final model quality. The paper does not propose methods for reducing the pipeline's computational cost (e.g., by amortizing Math-Critique evaluations across training runs, or by using smaller critic models). Future work on cost-efficient self-critique is not mentioned as a direction.

Limitation 3: All Results Are on a Single Model Family at a Single Scale

The assumption or constraint. Every experiment in the paper uses ChatGLM3-32B as the base model. The Math-Critique model is derived from ChatGLM3-32B. The RFT and DPO stages are applied to ChatGLM3-32B. The evaluation compares against a different model family (DeepSeek) and a different scale (20B, 67B), but the self-critique pipeline itself is never applied to any model other than ChatGLM3-32B. The paper implicitly assumes that the pipeline's effectiveness — specifically, the simultaneous improvement of math and language capabilities — generalizes across model families, scales, and training recipes.

The consequence. The paper's central claim about resolving the alignment-math trade-off is validated for exactly one model configuration. Several aspects of the pipeline's behavior could be model-specific:

  • The Math-Critique model's accuracy depends on how well the base model can learn the discriminative task of evaluating solutions. This ability likely varies with model scale (smaller models may be worse at evaluation, reducing the critic-generator accuracy gap that drives pipeline improvement) and with training recipe (models trained on different data distributions may have different calibration properties).
  • The language preservation mechanism — the paper argues that staying within the model's own output distribution prevents catastrophic language degradation. But this argument assumes that the base model's output distribution contains sufficient linguistic diversity, which may not hold for models with different pretraining data mixtures or for smaller models where math-style outputs could more easily dominate the fine-tuning signal.
  • The RFT rejection threshold and DPO difficulty threshold are hyperparameters that likely need to be tuned per-model. What counts as a "high" Math-Critique score for ChatGLM3-32B may not generalize to a model with different output score distributions.
  • The baseline math performance of ChatGLM3-32B-SFT-2312 is 52.4 on the GSM8k+MATH average (Table 1), which is lower than DeepSeek-67B-Chat (58.3) and InternLM2-Chat-20B (57.2). The paper's +17.5% improvement brings it to 61.6. A model starting at 58.3 would need a proportionally larger absolute improvement to reach the same final score, and it's unknown whether the pipeline can deliver that — diminishing returns may set in at higher baseline performance levels.

What evidence exists in the paper. None. The paper does not include any experiments with other base models. The introduction mentions that "related techniques have been deployed to ChatGLM" (Section 1), suggesting the approach works in their production system, but this is a statement about the same model family, not cross-family validation. The comparison in Table 1 shows that the pipeline's final model (61.6 math, 7.80 language) outperforms models from other families (DeepSeek at 58.3/7.11, InternLM2 at 57.2/7.68), but this is a comparison of final performance, not a demonstration that applying the pipeline to those other models would produce similar gains. A practitioner using LLaMA, Qwen, Yi, or any non-ChatGLM base model has no direct evidence that the self-critique pipeline would work for them, let alone that it would produce the claimed simultaneous improvement.

Mitigation status. The paper does not acknowledge this as a limitation. There is no discussion of model-specific factors that might affect pipeline performance, no call for cross-family replication, and no suggestion that results might differ at other scales. The deployment to ChatGLM is presented as evidence of real-world applicability, but since ChatGLM is the same model family, this does not address the generalization question. The paper's release of the MathUserEval benchmark and evaluation scripts (Section 1) could enable other practitioners to test their own models, but the pipeline's training methodology is what needs cross-family validation, not just the evaluation benchmark.

Limitation 4: The Two-Stage Curriculum (RFT then DPO) Is Not Validated Through Ablation

The assumption or constraint. The paper presents the self-critique pipeline as consisting of two sequential stages — RFT followed by DPO on hard problems — with the rationale that RFT provides broad improvement (raising the floor) while DPO provides targeted contrastive improvement on the hardest cases (targeting the ceiling). This design choice is central to the paper's technical approach and is presented as a key insight. However, the paper does not report results for RFT-only or DPO-only variants, making it impossible to determine whether both stages contribute meaningfully, whether the sequential ordering matters, or whether a single-stage approach could achieve similar results at lower cost.

The consequence. Without ablations, several important questions about the pipeline's mechanism remain unanswered:

  • Does RFT alone account for most of the gain? If the +17.5% math improvement is primarily from RFT and DPO adds only a marginal increment (e.g., +1–2%), then the two-stage design adds complexity and compute cost (generating additional candidates, running Math-Critique again, DPO training) without commensurate benefit. A practitioner could simplify the pipeline to RFT-only and retain most of the improvement.
  • Does DPO alone (without RFT first) perform poorly? The paper's implicit claim is that RFT is a necessary prerequisite because DPO needs clear preference pairs — the model must already produce both recognizably correct and recognizably incorrect solutions for the contrastive signal to be informative. If DPO applied directly to the base model (without RFT) performs nearly as well as the full pipeline, then the sequential design is unnecessary and RFT is redundant.
  • Is the ordering important? If DPO followed by RFT (reverse order) performs comparably to RFT followed by DPO, then the claimed curriculum logic (raising the floor before targeting the ceiling) is not supported. The improvement might come simply from doing more training with more filtered data, regardless of ordering.
  • Does DPO specifically on hard problems outperform DPO on all problems? The paper restricts DPO to the hardest problems, but does not compare against applying DPO to the full training set. If DPO on all problems performs equally well, the hard-problem filtering adds unnecessary complexity.

What evidence exists in the paper. The paper's main text provides no ablations decomposing the contribution of each stage. Table 1 reports only the final pipeline result (ChatGLM3-32B + RFT & DPO), not intermediate checkpoints after RFT alone or after DPO alone. The paper does not include a figure showing the performance trajectory across stages. The technical approach section (Section 3) provides a detailed rationale for the two-stage design, but this is a conceptual argument, not an empirical validation. The absence of these ablations is particularly notable because the distinction between RFT and DPO — and the claim that they serve different purposes in a curriculum — is one of the paper's main conceptual contributions. Without demonstrating that the two stages provide complementary rather than redundant improvement, that contribution remains hypothetical.

Mitigation status. The paper does not acknowledge this as a limitation. The two-stage design is presented as an integrated pipeline, and the contribution of individual stages is not discussed. There is no suggestion that future work should isolate stage-specific effects or that practitioners might consider simplified variants. This is a significant methodological gap because ablations of the core design choices are standard practice for pipeline papers — the reader needs to know which components are load-bearing and which are incidental.

Limitation 5: The Difficulty Estimation Mechanism (Hard Problem Filtering for DPO) Is Underspecified and Untested

The assumption or constraint. The DPO stage of the pipeline depends on identifying a subset of "hard problems" — problems where the RFT model still produces predominantly incorrect solutions — and constructing preference pairs only for these problems. The paper's technical approach describes this conceptually: the Math-Critique model scores the RFT model's solutions, and problems with low aggregate scores are classified as hard. However, the paper does not specify the aggregation method (e.g., average score? pass rate? minimum score?), the hardness threshold, the fraction of problems classified as hard, or how sensitive the pipeline's performance is to these choices. Furthermore, the paper does not report how the DPO hard-problem set differs from the full training set in terms of problem characteristics — are hard problems simply those with more steps, more complex operations, or answers that are harder to verify?

The consequence. A practitioner attempting to replicate the pipeline faces several unspecified design decisions that could significantly affect results:

  • If the hardness threshold is set too aggressively (only the very hardest problems are selected), the DPO training set becomes small, potentially leading to overfitting or insufficient preference signal. The model might not receive enough contrastive training to improve on the problems that need it most.
  • If the hardness threshold is set too permissively (most problems are classified as hard), DPO becomes essentially equivalent to applying DPO to all training data, losing the claimed benefit of focusing compute on bottleneck cases. The pipeline might waste DPO training on problems the model already handles well, where the contrastive signal is weak.
  • The aggregation method matters. Averaging Math-Critique scores across solutions could mask bimodal behavior — a problem where the model produces some perfect solutions and some terrible ones might have a moderate average score, obscuring the fact that the model hasn't converged to reliability. Using the minimum score might flag problems where the model occasionally makes catastrophic errors. Using the pass rate (fraction above the RFT acceptance threshold) might better capture consistency.
  • The hard problem set's composition affects generalizability. If hard problems are systematically different from easy ones (e.g., requiring more advanced mathematical concepts, longer reasoning chains, or specific problem types), then DPO training on only hard problems might specialize the model to those problem types rather than providing general mathematical improvement. The subsequent evaluation on academic benchmarks (GSM8k, MATH) might not reflect this specialization if those benchmarks have different difficulty distributions.

What evidence exists in the paper. None beyond the conceptual description. The paper does not report the number or fraction of problems classified as hard, the threshold value, the aggregation method, or any sensitivity analysis varying these parameters. There is no characterization of what distinguishes hard problems from easy ones in terms of problem type, length, required concepts, or Math-Critique score distribution. There is no ablation comparing DPO on hard problems against DPO on all problems or against DPO on random subsets to test whether the difficulty-conditioned selection matters.

Mitigation status. The paper does not address this limitation. The hard problem identification mechanism is treated as a straightforward implementation detail rather than a design choice with important consequences. The paper does not discuss the reliability of the difficulty estimation, alternative aggregation methods, or the risk that the Math-Critique model's scores might not be well-calibrated for difficulty estimation (i.e., low scores might indicate ambiguity or poor problem formulation rather than genuine difficulty for the model). Future work on adaptive difficulty estimation or dynamic threshold selection is not proposed.

Limitation 6: The Language Improvement Claim Lacks Mechanistic Evidence and Could Be an Artifact of Evaluation

The assumption or constraint. The paper claims that the self-critique pipeline not only preserves but improves general language capability, as measured by a +5.85% improvement on AlignBench (from 7.37 to 7.80, Table 1). The paper attributes this to the pipeline staying within the model's own output distribution — because the RFT training data consists of model-generated solutions rather than out-of-distribution math corpus data, the model doesn't suffer the catastrophic language forgetting observed in Math-InternLM2-20B (-14.8% on AlignBench, Table 1). However, the paper provides no direct evidence that the self-distribution property is the causal mechanism for language preservation. Alternative explanations are not ruled out, and the statistical reliability of the AlignBench improvement is not established.

The consequence. The simultaneous improvement claim — which is the paper's headline result distinguishing it from prior work — rests partly on the language improvement being real and causally linked to the pipeline design. If the +5.85% language improvement is actually:

  • Within evaluation noise: AlignBench scores are reported without confidence intervals or standard deviations. The scores in Table 1 range from 6.53 to 7.80, a compressed range of only 1.27 points across all models. A +0.43 absolute improvement (from 7.37 to 7.80) on a benchmark with this narrow spread could fall within the variance of the evaluation procedure (e.g., different prompt templates, different random seeds, or annotator disagreement if human evaluation is involved). Without error bars, a practitioner cannot determine whether the language improvement is statistically distinguishable from zero.
  • An artifact of additional training on diverse data rather than specifically the self-critique mechanism: The RFT stage involves fine-tuning on model-generated math solutions, which contain a mixture of natural language reasoning and mathematical notation. Any additional fine-tuning on diverse text — even math-focused text — could improve AlignBench scores through a regularization-like effect, improved instruction-following, or better formatting. If this is the mechanism, then any fine-tuning on model-generated content (not specifically the self-critique pipeline) would produce the language improvement, and the "self-distribution" argument is a post-hoc rationalization rather than a verified mechanism.
  • Specific to AlignBench's evaluation dimensions: AlignBench evaluates alignment quality across dimensions like helpfulness, harmlessness, and language fluency. The improvement might be concentrated in specific sub-dimensions that are incidentally improved by the pipeline (e.g., better formatting of responses, more confident tone) rather than reflecting genuine improvement in general language capability. The paper does not provide a breakdown by AlignBench sub-dimension.

What evidence exists in the paper. Table 1 reports the single aggregate AlignBench score for each model, with no error bars, no sub-dimension breakdown, and no discussion of statistical significance. The paper's introduction states that "the self-critique pipeline enables simultaneous improvement of language and mathematical abilities" (Table 1 caption), treating the +5.85% as unambiguous evidence. The technical approach section (Section 3) provides the conceptual argument about self-distribution preventing language degradation, but there is no ablation testing this mechanism — for example, comparing the AlignBench impact of RFT on model-generated solutions versus RFT on ground-truth solutions, or comparing against an alternative training regimen that also stays within the model's distribution. The Math-InternLM2-20B result (-14.8%) demonstrates that math specialization on out-of-distribution data degrades language, but this is a different intervention (domain-specific SFT on math corpora) and doesn't isolate the self-distribution variable.

Mitigation status. The paper does not acknowledge uncertainty around the language improvement claim. There is no discussion of potential confounding factors, no error analysis on AlignBench, and no suggestion that the language improvement might be incidental rather than an intrinsic property of the pipeline. The paper does not propose experiments to verify the causal mechanism (e.g., intentionally violating the self-distribution property and measuring the language impact). A practitioner considering whether to adopt the self-critique pipeline specifically for its claimed language-preservation benefit would need to independently verify that the improvement replicates on their model and evaluation setup, since the paper provides only a single point estimate without robustness analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a diagnostic contribution rather than a paradigm shift: it identifies where the alignment-math trade-off actually lives — in the training signal, not in the policy architecture — and demonstrates that replacing the signal source can resolve the trade-off without changing the training objectives or model architecture. This is a reframing with practical consequences, not a fundamental re-imagination of how LLMs learn.

The core reframing: the critic matters more than the policy. Before this work, the alignment-math trade-off documented in Table 1 was largely treated as an unavoidable tension: alignment training (optimizing for human preferences) degrades math because human raters don't reward correctness, and math specialization degrades language because domain-specific training data overwrites conversational ability. The implicit assumption was that these capabilities genuinely compete for representational capacity in the model — that you cannot have both because the model's parameters can't simultaneously encode the pattern "produce mathematically rigorous reasoning" and "produce natural conversational language" at the same level of quality.

The self-critique pipeline challenges this assumption directly. ChatGLM3-32B with RFT+DPO achieves both higher math scores (+17.5%, from 52.4 to 61.6) and higher language scores (+5.85%, from 7.37 to 7.80) than its starting checkpoint. The model's parameter count didn't change. The training objectives (standard SFT cross-entropy for RFT, standard DPO for Stage 2) are not novel. What changed is what provided the quality signal: a Math-Critique model trained from the base LLM itself, providing correctness-oriented feedback that neither human preference data nor domain-specific math corpora could supply without collateral damage.

This reframing has a specific, actionable implication: when an alignment procedure degrades a specific capability, the first place to look is the reward or critique signal, not the policy training objective. Rather than accepting the degradation as an "alignment tax" to be minimized, or trying to compensate by mixing in capability-specific training data (which risks new degradations), practitioners should ask: can we build a critic that encodes the desired capability within the model's own output distribution? If yes, that critic can drive improvement through standard training mechanisms (rejection sampling, preference learning) without forcing the model into out-of-distribution output patterns that overwrite other capabilities.

This paper doesn't prove that this reframing applies universally — it demonstrates it for one model family (ChatGLM), one capability (mathematical reasoning), and one scale (32B). But the conceptual move is transferable: code generation, factual accuracy, logical consistency, and other domains where human preferences diverge from objective correctness are natural candidates for a self-critique approach.

Reconciling prior contradictions. The paper helps explain why prior work showed contradictory patterns. Math-specialized models like Math-InternLM2-20B (-14.8% language, Table 1) degraded language not because math and language are inherently in conflict, but because the training data for math specialization (curated math corpora, structured problem-solution pairs) was distributionally far from the conversational data the model was originally trained on. The degradation was a data distribution problem, not a capability capacity problem. Conversely, alignment-first approaches like DeepSeek-67B-Chat-DPO (-1.2% math, Table 1) degraded math not because alignment is inherently anti-math, but because the human preference data used for DPO training encoded stylistic and helpfulness preferences that were orthogonal to or negatively correlated with mathematical correctness. Both degradations were signal problems, not capacity problems. The self-critique pipeline succeeds by synthesizing a signal (self-generated, correctness-oriented, distributionally aligned) that neither prior approach had access to.

Which research directions become more attractive. This work makes critic model design and training a first-class research problem for domain-specific LLM improvement. Previously, the RLHF/RLAIF literature focused primarily on training better reward models for general helpfulness and harmlessness. This paper suggests that domain-specific critics — trained from the base model itself, specialized to particular capability dimensions — can enable capability improvements that general reward models cannot. Research into critic architectures, critic training data efficiency, critic calibration, and methods for combining multiple specialized critics (e.g., a math critic + a code critic + a factuality critic) all become more attractive. The paper also makes self-training pipelines (where the model generates its own training data and its own evaluation signals) more credible as a practical deployment strategy, since it demonstrates that the approach works at 32B scale and has been deployed to a production system (ChatGLM).

Which directions become less attractive. The paper indirectly argues against the approach of "just train a bigger model to handle both capabilities." DeepSeek-67B-Chat, at more than 2× ChatGLM3-32B's parameters, still exhibits the alignment-math trade-off (-1.2% math under DPO alignment, Table 1). The self-critique pipeline at 32B achieves higher math scores (61.6 vs. 57.7–58.3) and comparable or better language scores (7.80 vs. 7.11–7.68) than these larger models. This suggests that investing compute in better training signals may be more efficient than investing it in larger models when the goal is joint improvement of multiple capabilities that have conflicting training signal requirements under standard methods. The paper also makes "accept the alignment tax as inevitable" a less tenable position — the tax was a property of the training signal, not of alignment itself.

Caveats on the magnitude of the shift. The paper does not introduce a new training algorithm, a new architecture, or a new theoretical framework. Its contribution is a pipeline design pattern (self-trained domain critic → rejection sampling → preference learning on hard cases) and an empirical demonstration that this pattern solves a specific documented trade-off. The shift is real but bounded: it applies to scenarios where (a) objective correctness can be defined and labeled for critic training, (b) the base model can learn to discriminate correct from incorrect outputs better than it can generate correct outputs, and (c) the domain of improvement does not require output patterns that are fundamentally incompatible with the model's general language distribution. These conditions likely hold for mathematical reasoning, code generation, and certain forms of factual QA, but may not hold for open-ended creative tasks, subjective evaluation domains, or capabilities that require fundamentally different reasoning architectures.

Follow-Up Research This Work Enables

1. Cross-family replication of the simultaneous improvement claim. The paper's central result — that the self-critique pipeline simultaneously improves math and language — is validated on exactly one model family (ChatGLM3-32B). A strong follow-up would apply the identical pipeline (Math-Critique model trained from the base LLM → RFT with critic-based rejection → DPO on hard problems) to at least three different model families (e.g., LLaMA-3-8B, Qwen-2-7B, DeepSeek-7B) and measure both math (GSM8k+MATH average) and language (AlignBench or equivalent) before and after. The key question: is the simultaneous improvement a property of the pipeline design, or is it specific to ChatGLM's training recipe, data mixture, or baseline capability profile? A negative result — e.g., LLaMA-3-8B improves math but degrades language under the same pipeline — would reveal that the self-distribution property is necessary but not sufficient, and that base model characteristics (pretraining data mixture, initial math-language balance, SFT recipe) mediate the pipeline's effect. This replication should also report the Math-Critique model's standalone accuracy for each base model to test whether critic reliability explains cross-model differences in pipeline effectiveness.

2. Direct ablation of self-critique versus external-critique versus ground-truth filtering. The paper's key design claim is that self-training the critic matters — that using a critic derived from the base LLM avoids distribution mismatch and enables language preservation. This claim is never directly tested. A strong follow-up would run a three-way comparison on the same base model (e.g., ChatGLM3-32B): (a) the full self-critique pipeline as described, (b) an identical pipeline but with GPT-4 (or another strong external model) replacing the Math-Critique model for RFT filtering and DPO pair construction, and (c) an identical pipeline but with ground-truth answer verification replacing the critic for filtering. All three variants would use the same training problems, the same number of candidate solutions per problem, and the same RFT and DPO hyperparameters. The comparison would measure: (1) math improvement (GSM8k+MATH average) — does self-critique match ground-truth filtering? How much does external critique lose? (2) language preservation (AlignBench) — does external critique cause language degradation similar to Math-InternLM2 (-14.8%)? Does ground-truth filtering cause degradation because ground-truth solutions may be out-of-distribution? (3) Critic accuracy — report precision/recall for both self-trained and external critics against ground truth. This experiment would definitively establish whether "self" matters, and if so, whether its benefit is in filtering accuracy, language preservation, or both.

3. Scaling the Math-Critique model's training data to understand the critic accuracy threshold. The paper never reports the Math-Critique model's accuracy or how much labeled data was needed to train it. A follow-up study would systematically vary the size of the labeled dataset used to train the Math-Critique model (e.g., 100, 500, 1,000, 5,000, 10,000 problem-solution pairs) and measure: (a) the critic's precision, recall, and F1 against ground-truth correctness on a held-out set, and (b) the downstream pipeline's math and language performance when that critic is used for RFT filtering and DPO pair construction. This would reveal the minimum critic accuracy needed for the pipeline to work — does the pipeline fail gracefully (smooth degradation) or catastrophically (cliff-edge drop) as critic accuracy decreases? If the pipeline tolerates critics with only 70–75% accuracy, that dramatically reduces the annotation burden for applying this method to new domains. If it requires >90% accuracy, then the one-time labeling cost for critic training is substantial and the method is less practical for niche domains. This experiment would also characterize the relationship between critic training data size and critic accuracy for mathematical reasoning, providing practical guidance for practitioners.

4. Single-stage versus two-stage ablation with difficulty-stratified analysis. The paper presents RFT → DPO as an integrated pipeline with DPO focused on hard problems, but never reports what happens if you stop after RFT, skip straight to DPO, or reverse the order. A follow-up would train and evaluate four variants on the same base model: (a) RFT-only, (b) DPO-only (preference pairs constructed from base model outputs, scored by Math-Critique), (c) RFT → DPO (the full pipeline), and (d) DPO → RFT (reverse order). Results should be reported stratified by problem difficulty (using the Math-Critique model's aggregate score as a difficulty proxy, binned into quintiles as in prior test-time compute work) to test the paper's implicit claim that RFT raises the floor (helps on easy/medium problems) while DPO targets the ceiling (helps on hard problems). If RFT-only matches the full pipeline on easy problems but substantially underperforms on hard problems, that validates the curriculum logic. If DPO-only performs poorly on hard problems (because preference pairs from the base model are noisy), that validates RFT as a necessary prerequisite. If DPO-only on all problems matches the full pipeline, the hard-problem filtering is unnecessary complexity. This experiment would transform the paper's conceptual rationale into an empirically validated design principle.

5. Continuous self-improvement loops using deployment data. The paper's most ambitious implicit claim is that the self-critique pipeline enables improvement from unlabeled user queries — the Math-Critique model replaces ground-truth verification, so new problems from production traffic can be incorporated into the training pipeline without manual labeling. A follow-up would simulate this by: (a) deploying the final ChatGLM3-32B model, (b) collecting real user math queries over some period, (c) running those queries through the self-critique pipeline (Math-Critique scoring → RFT on accepted solutions → DPO on hard cases) to produce an updated model, and (d) measuring whether the updated model improves on (1) the original academic benchmarks (GSM8k, MATH — to test for regression), (2) MathUserEval (to test for real-world generalization), and (3) a held-out set of recent user queries (to test for genuine improvement on the deployment distribution). The key risk to test: does the Math-Critique model's accuracy degrade on real user queries relative to academic problems? If user queries are more ambiguous, poorly specified, or have multiple valid approaches, the critic's judgments may become noisier, and the pipeline could drift or degrade. This experiment would either validate the pipeline's central value proposition (improvement without labels on real user data) or identify the conditions under which critic reliability breaks down in deployment.

6. Multi-domain critic composition for general-purpose deployed models. The paper demonstrates a math-specific critic, but a general-purpose deployed LLM like ChatGLM handles many domains — code, factual QA, translation, summarization, creative writing. A natural extension is to train multiple specialized critics (math critic, code critic, factuality critic, etc.) each derived from the same base LLM, and then apply domain-specific self-critique pipelines for each capability dimension. The follow-up research question is: do multiple sequential self-critique pipelines interfere with each other? Specifically, if you run the math self-critique pipeline (RFT + DPO on math), then run a code self-critique pipeline on the resulting model, does the code improvement degrade math? Does the order matter? Is there a way to combine the preference data from multiple critics into a single DPO stage that improves all domains simultaneously without interference? This experiment would test whether the self-critique approach scales to the multi-capability improvement problem that originally motivated the paper — the simultaneous improvement of all capabilities in a general-purpose deployed system, not just math and language.

Practical Applications and Downstream Use Cases

1. Deployed general-purpose LLM systems with math-capable and conversational users. The most direct application is the one the paper itself demonstrates: improving an online serving LLM (like ChatGLM at chatglm.cn) that receives a mixture of conversational and mathematical queries from users. The self-critique pipeline enables the operator to improve math accuracy (from 52.4 to 61.6 on GSM8k+MATH average, a +17.5% relative gain, Table 1) while simultaneously improving conversational quality (+5.85% on AlignBench, Table 1), without needing to route queries to separate models or accept degradation on either front. The concrete benefit: a single model serving all users gets better at math and better at language, improving the experience for both user segments without increasing serving infrastructure complexity. The pipeline can also be run periodically on accumulated user queries (after privacy review) to enable continuous improvement from production traffic, since the Math-Critique model eliminates the need for ground-truth labels on those queries.

2. Bootstrapping domain-specific capabilities in fine-tuned models without catastrophic forgetting. Organizations that fine-tune base LLMs for specific professional domains — legal reasoning, medical QA, financial analysis, scientific research — often face the same trade-off: domain-specific SFT on curated corpora improves target-domain accuracy but degrades the general language and instruction-following capabilities that make the model usable in interactive applications. The self-critique pattern provides an alternative: instead of fine-tuning on domain-specific corpora (which pulls the model toward out-of-distribution output patterns), train a domain-specific critic from the base model using a modest set of labeled examples, then use that critic to filter and contrast the model's own outputs on domain-relevant prompts. The domain capability improves through rejection of poor outputs and preference for good ones, while the model's output distribution stays anchored to its own generation patterns, preserving general capabilities. The paper's +17.5% math improvement with +5.85% language improvement (vs. Math-InternLM2's +5.1% math with -14.8% language, Table 1) provides a concrete magnitude comparison for the benefit of this approach over domain-specific SFT. The key practical requirement is a source of correctness labels for training the domain critic — which could come from expert annotations, existing labeled datasets, or automated verification tools (e.g., unit tests for code, fact-checking databases for factual claims).

3. Cost-efficient model improvement for organizations with limited annotation budgets. Training a Math-Critique model requires labeled problem-solution pairs, but this is a one-time cost to bootstrap the critic. Once the critic is trained, the RFT and DPO stages operate on unlabeled data — the model generates its own solutions, the critic scores them, and the pipeline filters and contrasts automatically. This means the marginal cost of improving the model on new data (e.g., newly collected user queries, expanded problem sets) is primarily generation and training compute, not human annotation. For an organization that has, say, 5,000 labeled math problems (enough to train a reasonable critic) but wants to improve on 100,000 unlabeled problems (from textbooks, user logs, or synthetic generation), the self-critique pipeline provides a way to leverage the larger unlabeled set without paying for 100,000 annotations. The paper doesn't quantify the annotation savings directly, but the structure of the pipeline makes this benefit clear: labels are needed only for critic training, not for each new training example. The practical constraint is that the critic's accuracy on the unlabeled data must remain sufficient — if the unlabeled data distribution drifts substantially from the critic's training distribution, the critic's judgments become less reliable, and the pipeline's effectiveness degrades. Monitoring critic confidence or periodically recalibrating the critic on a small labeled sample from the new distribution would be a necessary operational practice.

4. Improving mathematical reasoning in educational technology applications. The paper demonstrates improvement on both academic benchmarks (GSM8k, MATH) and a practical examination (Hungarian Exam, Figure 1), and introduces MathUserEval specifically for real-world mathematical queries. This combination makes the self-critique pipeline relevant for educational technology deployments where LLMs serve as math tutors, homework assistants, or automated grading systems. In these applications, both mathematical accuracy and the ability to explain reasoning in clear, natural language are critical — a tutor that produces correct answers but inscrutable explanations is as useless as one that produces fluent but incorrect reasoning. The self-critique pipeline's simultaneous improvement on both dimensions directly addresses this dual requirement. Furthermore, educational applications generate continuous streams of student queries that could be fed back into the pipeline (after appropriate anonymization and consent) to improve the model's handling of the specific kinds of mistakes, misconceptions, and problem types that real students encounter — which may differ substantially from academic benchmark distributions. The MathUserEval benchmark (which the paper releases publicly) provides a starting point for evaluating these real-world educational capabilities, though its specific alignment with student query distributions would need to be validated.