ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools

🎯 Pitch

GLM-4 outperforms GPT-4 on Chinese alignment and matches it on general benchmarks like MMLU, despite being trained on a predominantly Chinese-English corpus. Its 'All Tools' variant can autonomously decide when to call web browsers, Python, or image models—matching GPT-4's agentic capability. This report distills years of engineering lessons into a blueprint for building multilingual, tool-using models that rival much larger systems.

1. Executive Summary

This report introduces ChatGLM, an evolving family of large language models culminating in the GLM-4 series—GLM-4, GLM-4-Air, and GLM-4-9B—pre-trained on approximately ten trillion tokens of multilingual corpus and aligned through a multi-stage post-training process involving supervised fine-tuning and reinforcement learning from human feedback. GLM-4 demonstrates performance closely rivaling or surpassing GPT-4 across standard benchmarks (MMLU 83.3 vs. 86.4, and outperforming GPT-4 on GSM8K, MATH, BBH, GPQA, and HumanEval), matches GPT-4 Turbo in instruction following as measured by IFEval, and equals GPT-4 Turbo and Claude 3 Opus on long-context tasks via LongBench-Chat, while the GLM-4 All Tools variant is further aligned to autonomously understand user intent and call multiple tools—web browser, Python interpreter, text-to-image model, and user-defined functions—to complete complex tasks, matching or surpassing GPT-4 All Tools in web browsing and math problem-solving. GLM-4 achieves its strongest performance in Chinese language alignment, outperforming GPT-4 across all eight dimensions of AlignBench, establishing that bilingual pre-training with predominantly Chinese-English data and Chinese-focused alignment yields models competitive with or superior to state-of-the-art counterparts only when evaluated on tasks within their linguistic and cultural specialization.

2. Context and Motivation

The Core Problem: How to Build Truly Multilingual and Agentic LLMs

The fundamental question this report addresses is an engineering and scientific one: how do you build and continuously improve a family of large language models that is competitive with the best proprietary models, while also being practical enough for wide deployment? This is not a single paper proposing a novel algorithm, but rather a comprehensive technical report from the GLM team at Zhipu AI and Tsinghua University that synthesizes lessons learned across four generations of model development—from early explorations with GLM-10B and GLM-130B through the latest GLM-4 era. The challenge isn't "can we match GPT-4 on a single benchmark?" but rather how to systematically build capabilities across multiple dimensions (language understanding, instruction following, agentic tool use, long-context handling, safety) in an engineering pipeline that balances model capability, inference efficiency, and practical deployability.

This matters because the LLM landscape has evolved dramatically since the broader field coalesced around the decoder-only transformer + instruction tuning + RLHF recipe popularized by ChatGPT and GPT-4. While many research papers propose individual techniques—a new architecture variant, a new alignment method, a new agentic framework—the practical challenge of integrating these into a coherent model family that ships to millions of users remains under-documented. The GLM team has been in a unique position: they have been simultaneously releasing commercial API models (ChatGLM-130B, GLM-3-Turbo, GLM-4), open-sourcing smaller models for community use (ChatGLM-6B through GLM-4-9B, attracting over 10 million downloads), and conducting research on individual sub-problems. This report is their attempt to integrate these parallel streams of work into a coherent narrative about what works, what matters, and what tradeoffs arise when building LLMs at scale.

The report also addresses a gap in the linguistic and cultural representation of strong LLMs. By early 2024, the majority of state-of-the-art models were developed primarily with English data and English-centric alignment—GPT-4, Claude, Gemini, and Llama all skewed heavily toward English performance even when they had multilingual capabilities. The Chinese language ecosystem, with over a billion speakers and a distinct digital infrastructure (separate web platforms, social media, knowledge bases, and cultural references), was underserved by models that could genuinely match their English-language counterparts in Chinese understanding and generation. The paper positions GLM-4 as filling this gap: a model that is competitive with GPT-4 and Claude 3 Opus on general English benchmarks while being genuinely superior in Chinese, rather than treating Chinese as an afterthought.

The design of GLM-4 All Tools addresses yet another gap: the integration of tool use directly into the model's alignment rather than as a post-hoc extension. Prior to GLM-4 All Tools, many models offered "tool use" or "function calling" as a separate capability—often requiring explicit user configuration, external orchestration frameworks (like LangChain or AutoGPT), or specialized prompting patterns. The GLM team's approach was to align the model from scratch to understand user intent, plan multi-step instructions, and autonomously decide when and which tools to invoke, including web browsers, Python interpreters, text-to-image models, and user-defined functions. This represents an ambition to make agentic behavior a native capability rather than an external scaffold.

The Missing Piece: How to Get from a Research Prototype to a Shipped Product

One of the most underappreciated challenges in LLM development is the gap between a one-off training run that produces a single checkpoint with impressive benchmark numbers and a sustainable model development pipeline that produces progressively better models over years. The GLM team's timeline (Figure 1) shows this concretely: GLM-10B in June 2021 → GLM-130B in August 2022 → ChatGLM-6B in March 2023 → ChatGLM2-6B in June 2023 → ChatGLM3-6B in October 2023 → GLM-4 in January 2024 → GLM-4 (0520) and GLM-4-Air (0605) in mid-2024. Each step involved not just scaling up compute, but incorporating lessons from the previous generation into the pre-training data pipeline, architecture choices, alignment strategy, and tool integration.

Prior work in the LLM space typically takes one of two forms. Capability papers (GPT-4 Technical Report, Gemini Technical Report, PaLM 2 Technical Report) describe the capabilities of a specific model release but often provide limited detail on the engineering decisions, negative results, and iteration process that led to the final model. Method papers propose individual techniques (DeepNorm for training stability, GQA for inference efficiency, LongAlign for context extension) but don't show how these fit together in a complete system or what tradeoffs arise from combining them. The GLM-4 report attempts to bridge this divide: it references the individual method papers (ChatGLM-RLHF, LongAlign, ChatGLM-Math, Self-Contrast, AgentTuning, APAR) but describes how they were selected, integrated, and layered together, and—crucially—shows the cumulative effect across generations (Table 1: ChatGLM-6B at 25.2% MMLU → ChatGLM2-6B at 45.2% → ChatGLM3-6B at 61.4% → GLM-4-9B at 74.7%). This matters because it demonstrates that the path from a 25% to a 75% MMLU score isn't just "train on more data" but requires coordinated improvements across data quality, architecture, pre-training stability, alignment data, and RLHF techniques.

Where Existing Approaches Fall Short

The paper implicitly identifies several shortcomings in the prevailing LLM development paradigm that their approach aims to address:

1. English-centric development with multilingual performance as an afterthought. Most major LLMs are developed primarily for English, with multilingual capabilities emerging as a byproduct of accidentally including non-English data in web-scale training corpora. The GPT-4 technical report, for instance, discusses multilingual performance but doesn't structure the alignment pipeline around any particular non-English language. For Chinese in particular, this creates a gap: models perform well enough to be useful but make culturally insensitive errors, misunderstand Chinese-specific references, and struggle with the nuances of a language where context, idiom, and classical allusions carry significant meaning. The GLM team's approach is to make Chinese-English bilingualism a first-class design goal of both pre-training and alignment, resulting in a model that can genuinely outperform GPT-4 on Chinese-specific evaluations (AlignBench overall: GLM-4 8.00 vs. GPT-4 7.46, GPT-4 Turbo 7.90).

2. Agentic capabilities as separate systems, not integrated model capabilities. While frameworks like ReAct, Toolformer, and function-calling APIs allow LLMs to use tools, these typically require explicit orchestration: the user or a separate system must parse the model's output for tool calls, execute them, and feed results back. The model itself doesn't understand the context of "I should use a web browser here, and a Python interpreter there, and this is a multi-step plan that requires sequential tool use." The GLM-4 All Tools design (Figure 4: plan, analyze, recursive execute, tool call, feedback, memory) integrates this understanding directly into the model's training, so the model autonomously decides the tool-use strategy rather than relying on external control flow. This is a shift from tool use as an API feature to tool use as an agentic capability.

3. Long-context capabilities treated as a pre-training checkbox rather than an alignment challenge. Many models claim support for long contexts (32K, 128K, 1M tokens) but achieve this primarily through architectural changes (RoPE scaling, attention optimizations) and continued pre-training on long documents. The GLM team's approach, described in their LongAlign paper but summarized in Section 2, recognizes that the model also needs to be aligned for long-context tasks—it needs to learn through post-training how to effectively use all that context, how to retrieve relevant information from arbitrary positions, and how to handle the specific user intents that arise in long-document scenarios. This additional alignment step is what enables GLM-4 (128K) to match GPT-4 Turbo and Claude 3 Opus on LongBench-Chat (87.3 vs. 87.2 and 87.7 respectively).

4. Safety alignment evaluated primarily on English-language benchmarks. The paper evaluates GLM-4 on SafetyBench, which includes a Chinese subset, and shows competitive performance with GPT-4 models. This matters because safety failures are often language-specific: a model that refuses toxic requests in English might comply with them in Chinese if the safety training data was predominantly English. The inclusion of Chinese-specific safety evaluation acknowledges that safety alignment is not language-agnostic.

5. The "closed-source vs. open-source" tension. The GLM team has maintained a dual strategy throughout their development: the largest, most capable models (GLM-130B, ChatGLM-130B, GLM-4) are available as commercial APIs, while smaller but still capable models (ChatGLM-6B, ChatGLM2-6B, ChatGLM3-6B, GLM-4-9B) are open-sourced with permissive access. This addresses a genuine tension in the field: proprietary API models provide the best performance but limit researchers' ability to study, modify, or deploy the models locally, while open models provide freedom but often lag significantly in capability. The GLM team's approach of open-sourcing strong models (GLM-4-9B at 74.7% MMLU, outperforming Llama-3-8B) while maintaining a commercial tier represents a practical compromise that has driven their significant community adoption (10M+ downloads).

How This Paper Positions Itself

The paper positions itself not as a revolutionary new method but as a comprehensive system report—the synthesis of a multi-year, multi-generation effort to build competitive LLMs with a specific emphasis on Chinese-English bilingualism and integrated agentic capabilities. This is a different genre from most ML papers: it's closer to the GPT-4 or Gemini technical reports in scope, but with more detail on the iterative development process and the engineering decisions that drove the evolution from GLM-130B to GLM-4.

The theoretical contribution is the demonstration that the GLM architecture, originally proposed in 2021 with an autoregressive blank-infilling objective distinct from standard decoder-only transformers, can be evolved to match the state-of-the-art while incorporating modern efficiency improvements (GQA, SwiGLU, RMSNorm, 2D RoPE) without losing the benefits of the original design. This matters because the field has largely converged on a standard decoder-only recipe, and demonstrating that a slightly different architectural lineage can achieve competitive results validates that the specific architecture choices matter less than the quality of the overall training pipeline—a finding consistent with the broader observation that data quality and alignment matter more than architectural minutiae at scale.

The practical contribution is a detailed roadmap for other teams—particularly those working on non-English languages or in resource-constrained environments—showing that with careful attention to data quality, multi-stage alignment, and systematic iteration, it's possible to build models that compete with far more resource-intensive efforts from much larger organizations. The open-sourcing of the smaller models (ChatGLM-6B through GLM-4-9B) makes this roadmap concrete: other teams can study the artifacts at each stage of development and replicate or adapt the techniques.

The report also positions itself relative to the broader LLM literature through the "ChatGLM Techniques" section (Section 2), which catalogs the team's published research on specific sub-problems (emergent abilities from loss perspective, long-context alignment, math problem-solving via self-critique, RLHF practices, feedback-free alignment, agent tuning, parallel decoding). By referencing these papers explicitly, the report establishes that the GLM-4 system is not a single monolithic contribution but the integration of many incremental advances, each of which has been validated through the peer review process (or is under review) independently. This makes the report's claims more credible than if they were asserted without supporting methodological detail—the reader can trace each capability back to a specific technique with its own ablation studies and comparisons.

3. Technical Approach

3.1 Reader Orientation

This is a comprehensive system report describing a family of large language models rather than a paper proposing a single novel algorithm. The system is a pipeline for building progressively more capable models—from raw internet text to a final model that can understand user intent, follow instructions, use tools autonomously, and handle very long documents—with a specific focus on Chinese-English bilingualism and practical deployability across both commercial API and open-source channels. The core idea is that competitive LLMs are built through the accumulation and integration of many incremental improvements across pre-training data processing, model architecture, alignment, tool-use training, and safety, rather than through a single breakthrough; the report's contribution is making this integration process explicit and traceable across four generations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a sequential pipeline, plus a feedback loop that carries lessons from each generation into the next:

Pre-training Data Pipeline — takes raw multilingual documents (webpages, Wikipedia, books, code, research papers) and produces a deduplicated, filtered, tokenized corpus of ~10 trillion tokens. This is the foundation layer; all subsequent components build on its output.
Base Model Architecture and Pre-training — takes the tokenized corpus and trains a transformer model using specific architecture choices (DeepNorm/RMSNorm, RoPE, SwiGLU, GQA) to produce a base checkpoint that predicts next tokens. This base model has not yet been aligned to follow instructions; it only has raw language modeling capability.
Context Extension — takes the base checkpoint and extends its usable context length from 8K to 128K (and experimentally to 1M) through positional encoding extension and continued training on long documents. This produces a model capable of attending to very long inputs but not yet skilled at using that capability effectively.
Alignment Pipeline (SFT + RLHF) — takes the context-extended base model and refines it through supervised fine-tuning on human-annotated prompt-response pairs, followed by reinforcement learning from human feedback, producing a model that follows instructions, maintains multi-turn coherence, avoids generating unsafe content, and reasons step-by-step. For GLM-4 All Tools, this stage additionally integrates tool-use training.
Tool Integration (All Tools) — takes the aligned model and further trains it to autonomously plan multi-step tasks, decide when to invoke external tools (web browser, Python interpreter, text-to-image model, user-defined functions), and integrate the tools' outputs back into its response. This produces the final agentic model.

Information flows linearly through stages 1–5 for a single generation, but each generation (ChatGLM → ChatGLM2 → ChatGLM3 → GLM-4) feeds lessons back into the data pipeline, architecture choices, and alignment strategy for the next iteration.

3.3 Roadmap for the Deep Dive

First, the pre-training data pipeline (deduplication, filtering, tokenization, source weighting), since data quality and diversity are the foundation on which everything else depends.
Second, the model architecture choices and their justifications (no-bias-except-QKV, RMSNorm, SwiGLU, 2D RoPE, GQA), because these determine what the model can learn from the data and how efficiently it runs at inference.
Third, the context extension process, since it bridges the gap from a standard-length model to one that handles 128K–1M tokens and requires both architectural and alignment-level interventions.
Fourth, the alignment pipeline (SFT and RLHF), because this is where the base model acquires its practical utility—instruction following, safety, multi-turn coherence—and where the team's specific design choices (authentic human prompts, multi-dimensional scoring) have the most impact.
Fifth, the GLM-4 All Tools integration, since it builds on the aligned model to add autonomous tool use, planning, and execution capabilities.
Sixth, the evolution across generations (ChatGLM → ChatGLM2 → ChatGLM3 → GLM-4), showing how the accumulated changes compound and what specific techniques were added at each stage.

3.4 Detailed, Sentence-Based Technical Breakdown

Pre-Training Data Pipeline

The pre-training corpus is the foundation of the entire model, consisting primarily of multilingual documents with a heavy emphasis on Chinese and English, drawn from a mixture of sources: webpages, Wikipedia, books, code repositories, and research papers. The team reports a total corpus size of approximately ten trillion tokens, though the precise distribution across sources and languages is not given in the report.

The data processing pipeline follows three sequential stages: deduplication, filtering, and tokenization. Each stage addresses a specific quality or efficiency concern.

Deduplication improves data diversity by removing documents that are duplicated or near-duplicated within the corpus. The report mentions using both exact deduplication (where identical documents, identified by hash matching or equivalent, are removed) and fuzzy deduplication (where documents sharing a high fraction of overlapping n-grams or similar embeddings are treated as duplicates and one copy is retained). The rationale is straightforward: if the model sees the same text repeated many times during pre-training, it wastes compute memorizing exact strings rather than learning generalizable patterns, and it can bias the model toward over-represented sources. Fuzzy deduplication is particularly important for web-scale corpora because the same article often appears on multiple domains with slight formatting differences (different headers, footers, advertisements) but the same body text.

Filtering improves data quality by removing documents that contain noisy or harmful content. The report specifically lists removing documents with "offensive language, placeholder text, source code, etc." from the webpage portion of the corpus. Placeholder text (such as "lorem ipsum" or template fillers) contains no linguistic signal and wastes training compute. Source code is not inherently harmful but is separated into its own training stream rather than being mixed with natural language webpages, since code has a different syntactic structure and mixing it with prose degrades the model's ability to model natural language. Offensive language removal is both a quality and safety concern: toxic training data can lead to toxic model outputs, and filtering at the pre-training stage is one layer of defense (complemented by later alignment-stage safety interventions). The report does not provide details on the specific filtering heuristics or thresholds used.

Tokenization converts the filtered text into sequences of integer tokens that the transformer can process. The paper uses the byte-level Byte Pair Encoding (BPE) algorithm, which operates by starting with individual bytes as the initial vocabulary and iteratively merging the most frequent adjacent pairs to form subword units. The key design choice is how to handle the multilingual vocabulary, particularly the tension between Chinese characters and English subword tokens:

The team separately learns Chinese and multilingual token vocabularies using BPE.
These learned vocabularies are merged with the tokens from the cl100k_base tokenizer from OpenAI's tiktoken library (which is the tokenizer used by GPT-4 and GPT-3.5-turbo, with a vocabulary optimized for English and code).
The final unified vocabulary has a size of 150,000 tokens.

The motivation for this three-way merge is practical: starting from cl100k_base gives the tokenizer robust handling of English text, code, and common formats (URLs, whitespace, special characters) that are well-represented in that vocabulary. Adding separately trained Chinese and multilingual BPE tokens ensures that Chinese text—which uses a logographic writing system where individual characters carry semantic meaning, unlike the alphabetic English system—is tokenized efficiently rather than being broken into long sequences of subwords that are individually meaningless. The final vocabulary of 150,000 is a balance between coverage (enough tokens to represent many languages without excessive splitting into subwords) and efficiency (a larger vocabulary increases embedding matrix size and softmax computation cost). The number of tokens in the pre-training data directly affects training speed because each token requires a forward pass through the model; a better tokenizer (one that more often maps a character or word to a single token rather than multiple subword tokens) compresses the same text into fewer tokens, which directly speeds up training for a fixed computational budget.

After deduplication, filtering, and tokenization, the final training set is constructed by re-weighting different sources to increase the importance of high-quality and educational content. The report singles out books and Wikipedia as examples of sources that receive higher weight. Re-weighting means that the model sees tokens from these sources more frequently during training than their raw prevalence in the original data would dictate—for instance, if Wikipedia makes up 1% of the raw token count, it might be sampled as 5% of the training data by up-sampling or duplicating Wikipedia passages. The goal is to steer the model toward higher-quality, factually accurate, and well-structured text, under the assumption that language patterns learned from encyclopedia articles and published books are more useful for downstream tasks than patterns learned from social media posts or comment sections.

The report explicitly notes that the team has "yet to identify a fundamental principle that could guide the processes of data collection, cleaning, and selection," describing their approach as empirical and experience-driven rather than theoretically grounded. This is an honest acknowledgment of a broader field-wide challenge: while many papers have shown that data quality matters enormously, there is no rigorous framework for predicting which data sources will produce which model behaviors, and the process remains largely trial-and-error guided by downstream evaluation.

Model Architecture

The GLM family is built on the transformer architecture, specifically the General Language Model (GLM) variant originally proposed in 2021. Unlike standard decoder-only transformers (GPT) that are trained with a left-to-right autoregressive objective, or encoder-only transformers (BERT) trained with masked language modeling, or encoder-decoder transformers (T5) trained with span corruption, GLM uses an autoregressive blank infilling objective. The model is trained to generate text by filling in masked spans (blanks) in an autoregressive order—it sees the corrupted text with blanks and generates the missing spans one at a time, attending to both the surrounding context and previously filled spans.

However, the report focuses primarily on the architectural choices within the transformer blocks rather than on the pre-training objective itself, suggesting that the objective has been well-established in prior work (Du et al., 2022) and the recent innovations are in the architecture. The GLM-4 model adopts five specific architectural design decisions, each motivated by a combination of performance improvement and inference efficiency:

1. No Bias Except QKV. All bias terms in the model are removed—in the linear projections, layer norms, and feed-forward networks—with the sole exception of the biases in the Query, Key, and Value (QKV) projection matrices of the attention layers. Removing biases reduces the number of parameters that must be stored in memory and computed during both training and inference, which marginally increases training speed. The team also reports "a slight improvement in length extrapolation"—the model's ability to handle sequences longer than those seen during training is marginally better without biases everywhere. The QKV biases are retained because they are part of the attention computation that determines how much each token attends to every other token; the Q, K, and V projections are the three linear transformations applied to the input embeddings to produce the query, key, and value vectors that enter the scaled dot-product attention. The bias in these projections provides a learnable offset that can shift the attention patterns before the softmax is applied, and the team found this offset to be important enough to retain even when other biases were removed.

2. RMSNorm and SwiGLU. The team replaces LayerNorm with RMSNorm and ReLU with SwiGLU. LayerNorm normalizes activations by subtracting the mean and dividing by the standard deviation across the feature dimension, which stabilizes training by keeping activations in a consistent range. RMSNorm simplifies this by only dividing by the root mean square of the activations, omitting the mean subtraction; this is computationally cheaper (one fewer reduction operation per normalization) and has been shown to perform equally well in practice. SwiGLU replaces the ReLU activation function in the feed-forward networks. ReLU is the simple nonlinearity $\max(0, x)$ that passes through positive values unchanged and zeros out negative values, enabling the network to learn nonlinear transformations. SwiGLU is a more complex gating mechanism: the input is split into two linear projections, one is passed through a SiLU (Sigmoid Linear Unit, a smooth approximation to ReLU), and the two branches are multiplied element-wise. The report states these two changes "brought better model performance" without elaborating further, but the likely mechanism is that SwiGLU provides a smoother gradient landscape than ReLU (no hard zeroing that kills gradients for negative inputs), and RMSNorm's computational savings allow that compute to be reallocated elsewhere (e.g., to model depth or width) within the same training budget.

3. Two-Dimensional Rotary Positional Embeddings (RoPE). The original GLM architecture used Rotary Positional Encoding, which encodes position information by rotating the query and key vectors in the attention computation—rotations by different angles correspond to different positions, and the attention score between two tokens depends on their relative position through the difference in their rotation angles. Standard RoPE is one-dimensional: it encodes position along a single axis (token index in the sequence). The GLM architecture, because of its autoregressive blank infilling objective, operates in a two-dimensional space: each token has both a sentence position (where it appears in the full text) and a span position (its index within the current blank being filled). The team extended RoPE to a two-dimensional form to accommodate both positional signals simultaneously, with each dimension having its own set of rotation frequencies. This allows the attention mechanism to distinguish between "this token is at the beginning of its sentence but late in the span" versus "this token is early in the span but late in the sentence," which matters for the blank-infilling task where the model must attend to both the global context and the local generation position.

4. Group Query Attention (GQA). The team replaces Multi-Head Attention (MHA) with Group Query Attention. In standard MHA, the query, key, and value projections are split into $h$ independent heads, each with its own learned Q, K, V matrices, and attention is computed separately for each head. In GQA, the query projections remain independent across all $h$ heads, but the key and value projections are shared across groups of heads—for example, with $g$ groups, there are only $g$ distinct K and V projections, each shared by $h/g$ query heads. This reduces the size of the KV cache during inference: when generating text autoregressively, the model must store the key and value vectors for all previous tokens to avoid recomputing them at each generation step. In MHA, this cache stores $h$ sets of keys and values per token; in GQA with $g$ groups, it stores only $g$ sets—a factor of $h/g$ reduction. Since memory bandwidth is often the bottleneck for inference speed (loading the KV cache from memory dominates the computation), reducing the cache size directly increases tokens-per-second generation speed.

The report also notes a secondary design consequence of GQA: because GQA uses fewer parameters for the K and V projections than MHA (since keys and values are shared across heads), the team compensates by increasing the feed-forward network (FFN) parameter count to maintain the same total model size. Specifically, they set $d_{\text{ffn}}$ (the intermediate dimension of the FFN) to $10/3$ of the hidden size $d_{\text{model}}$ . The standard transformer FFN has two linear projections: an expansion from $d_{\text{model}}$ to $d_{\text{ffn}}$ followed by a nonlinearity, then a contraction from $d_{\text{ffn}}$ back to $d_{\text{model}}$ . By increasing $d_{\text{ffn}}$ , the FFN becomes wider and can represent more complex transformations, compensating for the reduced parameter count in the attention mechanism. The ratio $10/3 \approx 3.33$ is notably larger than the typical value of $4$ used in many transformer implementations (where $d_{\text{ffn}} = 4 \times d_{\text{model}}$ ), though the report does not explain why this specific value was chosen.

5. DeepNorm (from GLM-130B). Though not listed as a GLM-4-specific innovation, the report notes that GLM-130B used DeepNorm for training stability when scaling to 130B parameters. DeepNorm is a modified residual connection scheme that scales the residual branch differently from standard transformers to prevent the model's activations from growing unboundedly as depth increases. This was critical for training a 100B-scale model at a time (2022) when training stability at that scale was not yet well-understood. The report does not specify whether DeepNorm is still used in GLM-4 or has been replaced by RMSNorm, but given that RMSNorm is listed as a GLM-4-specific choice and DeepNorm is described in the context of GLM-130B, the two likely serve different normalization purposes (DeepNorm for residual scaling, RMSNorm for activation normalization) and may coexist.

Context Extension

The context length—how many tokens the model can attend to at once—has increased dramatically across ChatGLM generations: from 2K tokens (ChatGLM) to 32K (ChatGLM2 and ChatGLM3) to 128K and an experimental 1M tokens (GLM-4). The report describes this as achieved through two complementary processes, plus a third one specific to making the long context actually useful.

Position encoding extension addresses the fundamental challenge that models trained on sequences of length $L$ cannot directly process sequences longer than $L$ because the positional encodings for positions beyond $L$ have never been seen during training and are essentially random. The report references two approaches: positional interpolation and related techniques (Chen et al., 2023; Press et al., 2022). Positional interpolation works by taking a model trained on sequences up to length $L$ and, at inference time, scaling the position indices so that a sequence of length $kL$ is mapped back into the range $[0, L)$ that the model was trained on. For example, to handle a 128K sequence with a model trained on 8K, every position index $i$ in the input is replaced with $i \times (L_{\text{train}} / L_{\text{target}})$ . This means the model sees position embeddings it has been trained on, but the positional granularity is coarser—adjacent tokens in the long sequence are separated by sub-integer position distances. The alternative approach, referenced as "attention with linear biases" (ALiBi), replaces absolute positional encodings with a penalty term that is added to the attention scores based on the distance between tokens, naturally extending to any sequence length without special handling.

Continual training on long text further adapts the model to long sequences by fine-tuning (or continuing pre-training) on a corpus of long documents after the initial position encoding extension. This gives the model direct experience with long-range dependencies—learning to attend to information spread across tens of thousands of tokens—rather than relying solely on the interpolation trick. The reference to Xiong et al. (2023) suggests this uses a staged approach where the context length is gradually increased during training rather than jumping directly to the target length.

Long context alignment, described in the team's separate LongAlign paper (Bai et al., 2024), is the third and crucial component. Even if a model can technically attend to 128K tokens, it may not use that capability effectively—it might ignore the middle of long documents, fail to retrieve relevant information from arbitrary positions, or become confused by the sheer volume of context. Long context alignment uses instruction-tuning data specifically designed for long-context scenarios, training the model to follow instructions like "summarize this 50-page report" or "find the answer to this question in the provided document" using prompt-response pairs with long inputs. This teaches the model when and how to leverage its extended context rather than relying on short-context heuristics. The alignment focuses on practical use cases (document Q&A, summarization, coding with long codebases) rather than synthetic benchmarks, which is why GLM-4 can match GPT-4 Turbo and Claude 3 Opus on LongBench-Chat despite potentially having a shorter effective context during pre-training.

Alignment Pipeline (SFT and RLHF)

Alignment is the process of transforming a base model that only does next-token prediction into a model that follows instructions, engages in multi-turn dialogue, avoids harmful outputs, and respects human preferences. The report describes this as achieved primarily through two stages: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), with additional techniques layered on top.

Supervised Fine-Tuning (SFT) is the first stage. The base model is fine-tuned on a dataset of prompt-response pairs, where each pair consists of a user prompt and a high-quality response. The model is trained with standard next-token prediction loss on the response tokens (the prompt tokens are not used for loss computation, only for conditioning), which teaches the model the format and style of helpful assistant responses.

The critical design choice in SFT is the source and nature of the training data. The report emphasizes that "authentic human prompts and interactions instead of template-based or model-generated responses are vital to the alignment quality." For the first generation (ChatGLM-6B and ChatGLM-130B), the prompt-response pairs were annotated primarily by the model developers themselves—the team crafted conversations from scratch. For later models, the alignment data comes from a combination of in-house human annotation and proprietary data acquired from third parties, subject to strict quality control. This emphasis on human-generated data aligns with findings from the LIMA paper (Zhou et al., 2023) that a relatively small amount of high-quality, diverse human demonstrations can produce strong alignment, and contrasts with approaches that use LLM-generated outputs (from GPT-4 or otherwise) as synthetic training data, which can propagate subtle biases and stylistic quirks.

The report does not disclose the size of the SFT dataset, the number of annotators, or the exact annotation interface, but it does note that annotators score model responses along multiple dimensions: safety, factuality, relevance, helpfulness, and human preferences. This multi-dimensional scoring suggests that responses in the SFT data are not just "good examples" but have been explicitly evaluated and filtered across these criteria before inclusion.

Reinforcement Learning from Human Feedback (RLHF) is the second stage. After SFT, the model generally produces reasonable responses but still has issues: it may reject too many requests (over-cautiousness), mix languages within a single response (generating Chinese in the middle of an English answer or vice versa), lose coherence across multi-turn conversations, or produce subtle safety failures. RLHF addresses these by training a reward model on human preference comparisons (where annotators indicate which of two model responses they prefer) and then using that reward model to fine-tune the SFT model via reinforcement learning—typically Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO).

The team's specific RLHF practices are documented in their separate ChatGLM-RLHF paper (Hou et al., 2024). The report lists the issues that RLHF particularly helps with: "response rejection, safety, mixture of bilingual tokens generated, and multi-turn coherence." Response rejection refers to the model's tendency to say "I cannot help with that" for borderline-safe requests that a human would judge as acceptable to answer; RLHF tunes the model's refusal boundary. The bilingual token mixture suggests that the base model, having been trained on both Chinese and English, sometimes blends the two languages within a single response in unnatural ways—switching mid-sentence or using Chinese grammatical structures with English vocabulary—and RLHF provides a signal to keep languages separated or use them in contextually appropriate ways. Multi-turn coherence refers to the model maintaining consistent persona, knowledge state, and conversational goals across multiple turns of dialogue, rather than contradicting itself or forgetting the conversation history.

Self-Contrast is an additional alignment technique developed by the team (Liu et al., 2024) to address a practical bottleneck: human preference data for RLHF is expensive to collect at scale, especially across many languages, domains, and task types. Self-Contrast is a "feedback-free alignment strategy" that uses the target LLM itself to generate negative samples (incorrect or undesirable responses) for its own RLHF alignment. The core idea is that for a given prompt, the model can be prompted in different ways to produce both high-quality and low-quality responses—for example, by asking for a careful, step-by-step answer versus a rushed, sloppy answer, or by deliberately introducing errors into a correct answer. The contrast between the good and bad responses provides a training signal without human annotation, though the report does not detail how the model distinguishes which responses are good versus bad (this likely requires some form of automatic evaluation, such as consistency checks, self-verification, or comparison against a reference).

ChatGLM-Math is a technique specifically for improving mathematical reasoning (Xu et al., 2024). The key innovation is using self-critique—the model evaluating its own solutions—rather than external models or manual annotations for data selection. The pipeline works by generating multiple candidate solutions to math problems, having the model itself critique each solution (identifying errors, checking intermediate steps), and selecting the solutions that pass the self-critique for inclusion in the training data. This is a form of self-training or iterative refinement that is particularly effective for math because mathematical reasoning has clear correctness criteria (the final answer is either right or wrong, and intermediate steps can be checked for logical consistency) that make self-critique more reliable than in open-ended domains.

APAR (Auto-Parallel Auto-Regressive) is a technique for improving inference speed for responses with hierarchical structure (Liu et al., 2024b). Standard autoregressive generation produces one token at a time sequentially, which means generating a long, structured response (like a table, a JSON object, or a document with clear sections) can be slow even if parts of the output are independent and could be generated in parallel. APAR uses instruction tuning to train the model to plan its own generation process: it first generates a high-level outline or structure, identifies which sections are independent, and then generates those sections in parallel (auto-parallel) while still generating within each section autoregressively. This is analogous to a human writing an outline and then filling in sections independently, rather than writing every word in strict sequence.

AgentTuning is the team's framework for improving agent capabilities—the ability to interact with environments, use tools, and complete multi-step tasks (Zeng et al., 2023). The key component is the AgentInstruct dataset, which contains "high-quality interaction trajectories between agents and environment." These trajectories show the model step-by-step how to decompose a complex task, decide which actions to take, execute those actions, observe the results, and adjust the plan based on feedback. Training on these trajectories teaches the model the meta-skill of "being an agent": planning, tool selection, error recovery, and multi-step reasoning, rather than just the isolated skill of calling a single function.

GLM-4 All Tools Integration

The GLM-4 All Tools model is an aligned version of GLM-4 that is specifically trained to autonomously understand user intent, plan complex instructions, and call one or multiple external tools to complete tasks. This is the component that makes the model an agent rather than just a text generator.

The overall pipeline, illustrated in Figure 4, works as follows. When a user issues a complex request, the model first analyzes the task to understand its components, constraints, and goals. It then plans the problem-solving process step by step, determining which subtasks it can complete independently (using its own knowledge and reasoning) and which require external tools. When it decides that a tool is needed, it recursively executes a tool call: it generates the appropriate function call or action (e.g., a search query for the web browser, a Python script for the interpreter, an image generation prompt for CogView3), receives the tool's output (the search results, the Python execution result, the generated image), and integrates that feedback into its ongoing plan. This cycle of plan, execute, observe, replan can repeat for multiple tool calls within a single user request. The model maintains memory of what it has done so far—both in the conversation history and in its internal state—so that later steps can reference earlier results without repetition.

The available tools include:

Web Browser: The model can issue search queries, click on links, and extract information from web pages to answer questions requiring real-time or external knowledge. For example, in Figure 2, the model searches for "global population from 2000 to 2023," visits results from worldometers.info and macrotrends.net, and extracts the population figures to feed into the next step.
Python Interpreter: The model can write and execute Python code to perform calculations, analyze data, or solve problems that are better addressed algorithmically. In Figure 2, after retrieving the population data from the web, the model writes a Python script that calculates the Compound Annual Growth Rate (CAGR) using the formula (ending / starting) ** (1 / years) - 1 and converts the result to a percentage.
Text-to-Image Model (CogView3): The model can generate images based on text descriptions, useful for creative or illustrative tasks.
User-Defined Functions: Users can provide their own functions (via APIs) that the model can call, enabling integration with external services, databases, or custom workflows.

The key distinction from simpler function-calling setups is that the model learns when to use tools, not just how to write the function call syntax. This requires training data that includes multi-step trajectories where the model makes tool-use decisions, possibly making mistakes and recovering, and coordinating multiple tools for a single task. The report does not detail the specific training data or procedure for All Tools alignment, but it is built on the same alignment infrastructure (SFT + RLHF) with tool-use trajectories included in the training data.

Built on top of GLM-4 All Tools is the GLMs application platform (Figure 4, bottom), which allows users to create customized agents for specific tasks. These agents can combine the built-in tools (Python, web browser, CogView) with user-defined functions, external APIs, and custom knowledge bases, creating domain-specific AI assistants without requiring the user to modify the model itself.

Evolution Across Generations

The progression from ChatGLM-6B through GLM-4 is not just a story of scaling up compute; it's a cumulative integration of techniques, each adding capabilities and addressing limitations of the prior generation. Table 1 in the paper quantifies this progression:

Model	MMLU	GSM8K	MATH	BBH	HumanEval
ChatGLM-6B	25.2	1.5	3.1	0.0	0.0
ChatGLM2-6B	45.2	25.9	6.9	29.2	9.8
ChatGLM3-6B	61.4	72.3	25.7	66.1	58.5
GLM-4-9B	74.7	84.0	30.4	76.3	70.1

ChatGLM-6B (March 2023): Pre-trained on ~1T tokens, context length of 2K, post-trained primarily with SFT. This was the first generation—a proof of concept that a small bilingual model could be useful for conversational tasks, but it struggled on reasoning and code.

ChatGLM2-6B (June 2023): The major improvements came from better data quality (higher-quality pre-training corpus, more carefully curated alignment data), FlashAttention for extended context to 32K, and Multi-Query Attention (a precursor to GQA that shares keys and values across all heads) for faster inference. The 23% improvement on MMLU and the leap from 1.5% to 25.9% on GSM8K are primarily attributable to data quality improvements rather than architectural changes, since the model size remained the same.

ChatGLM3-6B (October 2023): This generation added native function call and code interpreter capabilities, plus complex agent tasks. The dramatic improvement in GSM8K (25.9% → 72.3%) and HumanEval (9.8% → 58.5%) reflects both improved pre-training (more code data, better data mix) and the addition of specialized training for reasoning and coding. The jump in BBH (a suite of 23 challenging reasoning tasks) from 29.2% to 66.1% suggests that chain-of-thought prompting and multi-step reasoning became more reliable.

GLM-4 (January–June 2024): The latest generation extends context to 128K (and 1M experimentally), adds the All Tools agentic capabilities, and achieves the strongest benchmark results. The 0520 version improves over the 0116 version on most metrics (MMLU 81.5 → 83.3, GSM8K 87.6 → 93.3, MATH 47.9 → 61.3), suggesting continuous improvement in both pre-training and alignment even between release checkpoints.

4. Key Insights and Innovations

Innovation 1: A Model Family Is a Scientific Instrument for Discovering What Matters

The report's most distinctive contribution is not any single technique, but the use of the multi-generation development process itself as a diagnostic tool to identify which factors drive LLM capability improvements. By open-sourcing and meticulously benchmarking each generation (ChatGLM-6B → ChatGLM2-6B → ChatGLM3-6B → GLM-4-9B) at the same model size (6B–9B parameters), the team creates a controlled experiment in LLM development. The dominant assumption in the field, following the GPT lineage, has been that capability improvements come primarily from scaling—more parameters, more data, more compute—with the specific techniques (architecture variants, alignment methods) contributing marginal gains on top of the scaling signal. The ChatGLM family challenges this framing by showing that a 9B parameter model can improve from 25.2% to 74.7% on MMLU (Table 1) over four generations without significant parameter scaling, purely through improvements in data quality, architecture, and alignment.

What makes this intellectually distinctive is that it allows causal attribution of capability gains to specific interventions, which is nearly impossible in one-shot models like GPT-4 or Gemini where the final checkpoint reflects many simultaneous changes with no intermediate checkpoints publicly available for comparison. When ChatGLM2-6B jumps from 1.5% to 25.9% on GSM8K (a 17× improvement on the same model size), the team can trace this primarily to higher-quality pre-training data and improved alignment, since the architecture changes between generations (FlashAttention for context extension, Multi-Query Attention for speed) wouldn't plausibly cause such a large reasoning improvement. This diagnostic power is unusual in the LLM literature, where capability papers typically present only the final model and method papers present isolated techniques in controlled settings that don't reflect the full integration challenge.

The report also uses this multi-generation lens to develop a scaling laws perspective specific to their architecture: models with 1.5B, 3B, 12B, 32B, 66B, and 130B parameters were trained across generations, allowing the team to validate observations across scales. This addresses a common failure mode where techniques that work at 6B parameters don't transfer to 100B+ scales, or vice versa—the family approach means techniques were tested at multiple scales before being adopted, making the final GLM-4 architecture more robust than if it had been designed solely from small-scale experimentation.

This contribution is fundamental in methodology rather than incremental: it demonstrates that systematic, well-documented iteration with public benchmarks at each stage can generate scientific knowledge about what matters in LLM development, rather than treating each model release as a proprietary black box whose internals can only be speculated about.

Innovation 2: Bilingualism as First-Class Architecture, Not Post-Hoc Adaptation

The report establishes that Chinese-English bilingual capability can be achieved natively through deliberate pre-training design rather than emerging incidentally from web-scale data or being added through post-hoc fine-tuning. This challenges the dominant field assumption, embodied by models like GPT-4 and Claude, that strong multilingual performance is a natural byproduct of training on sufficiently large and diverse web corpora—if you scrape enough of the internet, the model will eventually learn all languages represented in the data. The GLM team's approach argues instead that the architecture and pre-training pipeline must actively accommodate linguistic differences between English (alphabetic, whitespace-delimited, left-to-right) and Chinese (logographic, no natural word boundaries, deeply contextual) to achieve genuine parity rather than "English-native with Chinese as a second language."

The evidence for this being a genuine innovation rather than an obvious design choice lies in the specific architectural accommodations that distinguish GLM-4 from its English-centric competitors. The vocabulary design—separately learning Chinese and multilingual BPE tokenizers and merging them with the English-optimized cl100k_base vocabulary into a 150,000-token unified vocabulary—is not a standard multilingual tokenization approach. Standard practice (used by Llama, Mistral, and others) trains a single BPE tokenizer on the full multilingual corpus, which tends to over-fragment Chinese text into subword units that destroy the semantic information carried by individual characters. The GLM approach preserves character-level semantics for Chinese while maintaining efficient subword tokenization for English, a deliberate tradeoff that increases vocabulary size (and thus embedding parameters) in exchange for better Chinese representation.

The two-dimensional RoPE extension is similarly motivated by linguistic structure: the autoregressive blank infilling objective, which is core to the GLM architecture, requires the model to track both global position (where a token appears in the document) and local position (where a token appears within the span being generated). While this is described as an accommodation of the GLM objective rather than Chinese specifically, it reflects a broader design philosophy where the positional encoding adapts to the model's task structure rather than imposing a single one-dimensional sequential bias.

The results that anchor this claim are in Table 4 (AlignBench), where GLM-4 achieves an overall score of 8.00 versus GPT-4 Turbo at 7.90 and GPT-4 at 7.46, with particularly large advantages in Chinese Logic Reasoning (7.95 vs. 7.67 for GPT-4 Turbo) and Language Understanding (8.00 vs. 7.60). The fact that GLM-4 outperforms models with substantially more parameters and compute on Chinese-specific benchmarks, while being competitive on English benchmarks (Table 2), supports the interpretation that the bilingual-first design produces genuine parity rather than Chinese being an afterthought.

This is a fundamental reframing of the multilingual LLM problem: instead of asking "how much multilingual data accidentally leaked into our web crawl," the question becomes "how do we design the tokenizer, architecture, and training mix to treat each language as a first-class citizen?" The practical significance extends beyond Chinese—the approach provides a template for building LLMs that are genuinely strong in any language that is structurally different from English (Arabic, Japanese, Korean, Hindi) rather than treating multilingualism as a property that emerges from scale.

Innovation 3: Tool Use as a Native Model Capability, Not an External Framework

The report's approach to agentic capabilities represents a conceptual shift from tool use as infrastructure to tool use as alignment. Prior to GLM-4 All Tools, the dominant paradigm for giving LLMs access to external tools involved external orchestration: frameworks like LangChain, AutoGPT, or ReAct-style prompting where the model generates text that a separate system parses for tool calls, executes them, and feeds results back into the model's context. Even OpenAI's function calling API, while more integrated, still treats the model's function call generation and the external execution as separate steps managed by the API infrastructure rather than by the model itself. The model doesn't "know" it's using a tool—it just generates a specific JSON format that gets intercepted.

GLM-4 All Tools changes this by aligning the model to understand the semantics of tool use, not just the syntax. The model is trained to autonomously decide when a tool is needed, which tool is appropriate, how to sequence multiple tools, and how to integrate results—all as part of its learned behavior rather than relying on external control flow (Figure 4: Plan → Analyze → Recursive Execute → Tool Call → Feedback → Memory → Execution). This matters because external orchestration imposes a cognitive split: the "reasoning" happens in the orchestrator (which decides what to do next) while the model is merely a text generator that gets called repeatedly. In the All Tools approach, the reasoning about when and how to use tools is internal to the model's forward pass, which means the model can bring its full language understanding and world knowledge to bear on tool-use decisions—for instance, recognizing that a user's question about "average annual growth rate" requires both retrieving current data (web browser) and performing a mathematical calculation (Python interpreter) in sequence.

The evidence that this is a genuine capability rather than just a different API design comes from the integrated example in Figure 2 and the evaluation in Table 9, where GLM-4 All Tools matches GPT-4 All Tools on Python interpreter math (GSM8K: 91.59 vs. 92.72; MATH: 63.60 vs. 65.00) and surpasses it on web browsing for information seeking (78.08 vs. 67.12). The information-seeking gap is particularly notable because web browsing requires the model to make multiple sequential decisions—what to search for, which results to click, how to extract relevant information—that external orchestration frameworks often handle through brittle heuristics. The fact that an aligned model can outperform in this setting suggests that internalizing the tool-use decision process is genuinely more effective than external control flow.

This contribution is incremental in mechanism (it builds on the same SFT + RLHF alignment pipeline used for instruction following, extended with tool-use trajectories) but fundamental in conceptual framing: it asserts that agentic behavior is primarily an alignment and training problem, not an infrastructure problem. If this framing holds, it implies that future improvements in agentic capabilities will come from better training data and reward modeling rather than from more sophisticated orchestration frameworks—a significant reorientation of the LLM agents research agenda.

Innovation 4: Long-Context Capability Requires Alignment, Not Just Architecture

The report's treatment of long-context handling introduces a diagnostic distinction between technical context processing and functional context utilization. The field's dominant approach to long-context LLMs has focused on the engineering challenge: extending position encodings (RoPE scaling, ALiBi), optimizing attention for longer sequences (FlashAttention, sparse attention), and continuing pre-training on long documents. The implicit assumption has been that if the model can technically attend to 128K tokens without running out of memory or producing garbled outputs, it has long-context "capability."

The GLM team's LongAlign approach (referenced in Section 2 and described more fully in Bai et al., 2024) challenges this by demonstrating that technical processing and effective utilization are distinct problems. A model that can pass a "needle in a haystack" test—retrieving a specific fact from an arbitrary position in a long document—may still fail at practical long-context tasks like document summarization, multi-document synthesis, or long-form Q&A because it hasn't learned the meta-skill of how to use its extended context. The model might attend equally to all positions rather than learning to dynamically allocate attention to relevant sections, or it might default to short-context heuristics (paying attention primarily to the beginning and end of the document) that were learned during pre-training on predominantly short sequences.

The innovation is in making long-context alignment an explicit, named stage of the development pipeline, distinct from context extension. The LongAlign recipe involves creating instruction-tuning data specifically for long-context scenarios and training the model on these examples after the architectural context extension is complete. This teaches the model behaviors that purely architectural extension cannot provide: how to search through long contexts for relevant information, how to synthesize across distant sections, and how to produce coherent responses that reflect information from throughout the input rather than just the most recent or most prominent segments.

The evidence for this distinction comes from the LongBench-Chat results (Table 5), where GLM-4 (128K) achieves 87.3 on English prompts, matching GPT-4 Turbo (87.2) and Claude 3 Opus (87.7), and 84.0 on Chinese prompts, outperforming all baselines including GPT-4 Turbo (82.1) and Claude 3 Opus (82.7). These are not "find the random fact" tests—LongBench-Chat evaluates practical scenarios like document Q&A, summarization, and coding with long codebases, where effective context utilization matters more than raw retrieval accuracy. The fact that GLM-4, which was pre-trained with a base context of only 8K (versus GPT-4's 8K or potentially larger), can match or exceed models with substantially more pre-training length after alignment suggests that the alignment stage is doing significant work beyond what the architecture provides.

This contribution is incremental in technique (it adds a training stage to existing context extension methods) but diagnostically fundamental: it separates two concepts that the field had been conflating, and provides a concrete recipe for addressing the one that had been neglected. The practical implication is that teams working on long-context models should budget not just for architectural extension and continued pre-training, but also for long-context-specific alignment data and training, which has resource implications that were previously not accounted for in long-context development roadmaps.

5. Experimental Analysis

Evaluation Methodology

Dataset. The report evaluates GLM-4 across a wide range of benchmarks spanning different capabilities, rather than a single dataset. For general academic performance (Section 3.1), six benchmarks are used: MMLU (multi-choice questions from various examinations), GSM8K (8,500 grade school math word problems, 1,000 in test set), MATH (12,500 competition-level math problems, 5,000 in test set), BBH (23 challenging BIG-Bench tasks), GPQA (graduate-level multi-choice in biology, chemistry, physics), and HumanEval (code generation with automatic test-case checking). For instruction following (Section 3.2), IFEval is used (541 prompts from 25 distinct verifiable instructions), with the prompts translated into Chinese for cross-lingual evaluation (instructions not applicable in Chinese, such as capitalization, are omitted). For Chinese alignment (Section 3.3), AlignBench-v1.1 is used (683 queries across 8 categories, evaluated with GPT-4 as judge using multidimensional rule-calibrated scoring). For long-context (Section 3.4), LongBench-Chat is used (context lengths from 10–100K, with document Q&A, summarization, and coding scenarios), with results segregated into Chinese and English portions. For real-world coding (Section 3.5), NaturalCodeBench (NCB) is used (challenging bilingual coding problems derived from real user prompts, in Python and Java for both English and Chinese). For function calling (Section 3.6), the Berkeley Function Call Leaderboard is used (2,000 question-function-answer pairs evaluated by AST analysis, API execution, and relevance detection). For agent abilities (Section 3.7), AgentBench is used (7 out of 8 environments evaluated, excluding Digital Card Game due to interaction time). For safety (Section 4), SafetyBench is used (Chinese subset, 7 dimensions).
Base model(s). The primary models evaluated are GLM-4 (0520), GLM-4-Air (0605), and GLM-4-9B-Chat, all from the GLM family developed by Zhipu AI and Tsinghua University. GLM-4 (0520) is the latest and most capable API model; GLM-4-Air (0605) is a variant designed for lower latency and inference cost while maintaining comparable performance; GLM-4-9B-Chat is the open-source model with 9 billion parameters. The predecessor models ChatGLM-6B, ChatGLM2-6B, and ChatGLM3-6B are included in Table 1 to show the progression across generations. All GLM models are deployed with BFloat16 precision during evaluation.
Metrics. The primary metrics vary by benchmark and are computed as follows: For MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval, accuracy is reported—the fraction of problems for which the model's generated answer matches the ground truth, using chain-of-thought prompting for GSM8K, MATH, and BBH. For IFEval, prompt-level and instruction-level accuracy are computed in both strict mode (exact match of all constraints) and loose mode (partial credit for near-matches), following the methodology of Zhou et al. (2023). For AlignBench, a multidimensional rule-calibrated pointwise reference-based scoring method is used with GPT-4 as judge, producing scores on a scale that appears to range approximately from 0–10 based on the values in Table 4. For LongBench-Chat, outputs are scored by GPT-4 using a few-shot strategy, with multiple repeated evaluations to minimize variance, and the average is reported. For NCB, Pass@1 is reported (the fraction of problems where the first generated solution passes all test cases). For the Berkeley Function Call Leaderboard, accuracy is computed separately for AST summary (comparing generated function calls against function documents and possible answers via AST analysis), execution summary (checking response correctness by executing generated function calls), and relevance detection (evaluating the model's capacity to recognize unsuitable functions), with an overall accuracy combining these. For AgentBench, per-dataset scores are weighted according to the original per-dataset weights provided in the AgentBench paper (Liu et al., 2023) to produce an overall score. For SafetyBench, accuracy is reported as the fraction of multiple-choice safety questions answered correctly, aggregated across 7 dimensions and overall.
Baselines. The report compares against GPT-4 (0314 and 0613), GPT-4 Turbo (1106 and 2024-04-09), Claude 2, Claude 3 Opus, and Gemini 1.5 Pro. For code evaluation, Llama-3-8B-Instruct is additionally included as a comparison point for the function call evaluation (Table 7) and is referenced as a baseline for open-source comparisons (the report states GLM-4-9B "outperforms Llama-3-8B" in Section 1). For specific sub-evaluations, ChatGLM3-6B is included in the function call evaluation (Table 7). Baseline results are extracted from the corresponding technical reports or tested through public APIs, and the specific model versions are noted in each table (e.g., "GPT-4 (0613)", "GPT-4 Turbo (2024-04-09)"), acknowledging that API models may change over time.
Generation budget / compute accounting. The report does not explicitly measure test-time compute in FLOPs or generations as in a systematic scaling analysis; instead, it evaluates each model under its default or recommended inference configuration (chain-of-thought prompting where specified, BFloat16 precision for GLM models) and reports accuracy. The comparisons are thus primarily capability comparisons at deployment scale rather than compute-matched efficiency comparisons—the models are compared in terms of their final performance rather than the compute required to achieve that performance. The sizes of the GLM models are given (GLM-4's parameter count is not disclosed; GLM-4-9B has 9 billion parameters), and GPT-4's parameter count is not publicly known, so no parameter-matched or FLOPs-matched analysis is attempted. The lack of compute accounting is a deliberate choice reflecting the report's focus on end-user capability rather than training or inference efficiency.
Cross-validation / statistical protocol. For LongBench-Chat, evaluations are repeated multiple times to minimize score variations and reach more reliable statistical conclusions, and the average is reported (Section 3.4). For other benchmarks, no cross-validation, repeated sampling, or confidence interval computation is described, which means that the reported numbers are point estimates without quantification of variance. The number of evaluation examples varies by benchmark (e.g., 500 for MATH, 541 for IFEval, 683 for AlignBench, 500 for the MATH test set, 1,000 for GSM8K), and for some benchmarks with small test sets (like HumanEval with 164 problems), the statistical reliability of small absolute differences may be limited. The report does not discuss whether differences between models are statistically significant or provide error bars.

Main Quantitative Results

General Academic Benchmarks (Table 2)

GLM-4 (0520) achieves 83.3% on MMLU, representing 96.3% of GPT-4 (0314)'s 86.4% and closely approaching GPT-4 Turbo (2024-04-09) at 86.7% and Claude 3 Opus at 86.8%. On other benchmarks, GLM-4 (0520) surpasses GPT-4 (0314) across the board: GSM8K (93.3 vs. 92.0), MATH (61.3 vs. 52.9), BBH (84.7 vs. 83.1), GPQA (39.9 vs. 35.7), and HumanEval (78.5 vs. 67.0). The improvements over GLM-4 (0116) are notable—MMLU increased from 81.5 to 83.3, GSM8K from 87.6 to 93.3, and MATH from 47.9 to 61.3—indicating substantial gains from the updated pre-training and alignment between the January (0116) and May (0520) checkpoints. GLM-4-Air (0605) achieves 81.9% on MMLU, somewhat lower than GLM-4 (0520) but still competitive, and outperforms GLM-4 (0116) on GSM8K (90.9 vs. 87.6) and MATH (57.9 vs. 47.9), suggesting that the Air variant prioritizes efficiency while preserving reasonable capability. GLM-4-9B-Chat achieves 72.4% on MMLU and 79.6% on GSM8K, substantially below the larger GLM-4 models but with the advantage of being open-source and deployable on consumer hardware.

The comparison to the state-of-the-art is nuanced: GLM-4 (0520) outperforms Gemini 1.5 Pro on MMLU (83.3 vs. 85.9—actually lower), GSM8K (93.3 vs. 90.8), and BBH (84.7 vs. 89.2—actually lower), but lags behind Claude 3 Opus on GPQA (39.9 vs. 50.4) and HumanEval (78.5 vs. 84.9). The report's claim that GLM-4 "closely rivals or outperforms GPT-4" (from the abstract) is accurate for the original GPT-4 (0314) on five of six benchmarks where it outperforms, but is more qualified against the newer GPT-4 Turbo (2024-04-09), which leads GLM-4 on MMLU (86.7 vs. 83.3), MATH (73.4 vs. 61.3), GPQA (49.3 vs. 39.9), and HumanEval (88.2 vs. 78.5). The specific comparison point matters significantly—the results demonstrate that GLM-4 is competitive with GPT-4-class models from early 2024 but has not closed the gap to the latest GPT-4 iterations on several key benchmarks, particularly math and code.

Instruction Following (Table 3)

GLM-4 (0520) achieves 79.1% prompt-level strict accuracy in English, compared to 81.2% for GPT-4 Turbo (2024-04-09) and 77.1% for GPT-4 (0613). In instruction-level strict accuracy, GLM-4 reaches 85.0% versus 85.9% for GPT-4 Turbo, representing 99.0% of the GPT-4 Turbo's performance. In Chinese, GLM-4 achieves 71.9% prompt-level strict accuracy, slightly below GPT-4 Turbo (2024-04-09) at 72.6% but above GPT-4 (0613) at 68.9% and Claude 3 Opus at 73.3% (actually lower). The instruction-level strict accuracy in Chinese is 78.0% for GLM-4 versus 79.1% for GPT-4 Turbo. The gap between English and Chinese performance is consistent across all models (all models perform worse on the Chinese IFEval), which may reflect either the translation quality of the prompts or inherent differences in how instructions are expressed linguistically in Chinese versus English.

GLM-4-9B-Chat achieves identical scores across Chinese and English (73.0% loose prompt-level, 69.0% strict prompt-level), which is unusual and likely reflects the test methodology rather than identical capability—the report does not explain why these scores match. GLM-4-Air (0605) at 75.2% strict prompt-level English is somewhat behind GLM-4 (0520) at 79.1%, suggesting that the efficiency optimizations in Air come with a modest instruction-following cost. Overall, the IFEval results support the claim that GLM-4's instruction following "gets close to GPT-4-Turbo" (from the abstract), with the qualification that "close" means within 2–3 percentage points on most metrics rather than matching or exceeding.

Chinese Alignment (Table 4)

This is GLM-4's strongest area. GLM-4 (0520) achieves an overall score of 8.00 on AlignBench, matching GPT-4 Turbo (2024-04-09) at 8.00 and outperforming GPT-4 Turbo (1106) at 7.90, Claude 3 Opus at 7.53, GPT-4 (0613) at 7.46, and Gemini 1.5 Pro at 7.47. The category-level breakdown reveals where the advantage comes from: GLM-4 (0520) achieves its highest scores in Professional Knowledge (8.47), Writing (8.11), and Role Play (8.06), while GPT-4 Turbo (2024-04-09) leads in Math (8.32 vs. 7.89) and Writing (8.37 vs. 8.11). The largest gap favoring GLM-4 is in Chinese Logic Reasoning (7.95 vs. 7.67 for GPT-4 Turbo 2024-04-09) and Language Understanding (8.00 vs. 7.60), consistent with the claim that bilingual-first pre-training yields genuine advantages in Chinese-specific linguistic and reasoning tasks.

The comparison across GLM-4 versions shows that GLM-4-Air (0605) at 7.65 overall is competitive with GLM-4 (0116) at 7.66 and substantially above GLM-4-9B-Chat at 7.01. GLM-4 (0520) represents a meaningful improvement over GLM-4 (0116) on several categories, particularly Chinese Logic Reasoning (7.20 → 7.95), Math (7.20 → 7.89), and Language Understanding (7.60 → 8.00). The AlignBench results use GPT-4 as judge, which introduces a potential confound—if GPT-4 has biases in evaluating Chinese text, these could affect the relative rankings, though the report does not discuss this.

The claim that GLM-4 "outperforms GPT-4 in Chinese alignments" (from the abstract) is supported against the original GPT-4 (0613) at 7.46, but the comparison to GPT-4 Turbo is more accurately described as "matches" (both at 8.00 overall), and GPT-4 Turbo (2024-04-09) was released after some of the GLM-4 versions being compared. The specific model version matters significantly—the abstract's claim holds for GPT-4 but not necessarily for GPT-4 Turbo depending on which version is referenced.

Long Context Handling (Table 5)

GLM-4 (0520) achieves 87.3 on LongBench-Chat English prompts, compared to GPT-4 Turbo (1106) at 87.2, Claude 3 Opus at 87.7, and GPT-4 Turbo (2024-04-09) at 85.0. On Chinese prompts, GLM-4 (0520) achieves 84.0, substantially outperforming GPT-4 Turbo (1106) at 71.4, GPT-4 Turbo (2024-04-09) at 82.1, and Claude 3 Opus at 82.7. The pattern is consistent with the bilingualism thesis: GLM-4 is competitive with the best models in English while being clearly superior in Chinese, where the state-of-the-art models show more variance (GPT-4 Turbo 1106 at 71.4 vs. 2024-04-09 at 82.1—a 10-point difference that may reflect API updates or evaluation variance).

GLM-4-Air (0605) at 82.4 English and 81.0 Chinese is notably behind GLM-4 (0520), while GLM-4-9B-Chat at 76.8 English and 79.0 Chinese shows a reversal of the typical pattern (better Chinese than English), which could reflect the 9B model's training data composition or the specific alignment focus. The LongBench-Chat evaluation uses GPT-4 as the judge (few-shot strategy), which is standard for this benchmark but introduces the same potential judge-bias confound as AlignBench, particularly for Chinese-language evaluation where GPT-4's own Chinese capabilities may affect its judgment.

The report's claim that GLM-4 "matches GPT-4 Turbo (128K) and Claude 3 for long context tasks" (from the abstract) is supported on English prompts (87.3 vs. 87.2 and 87.7) and is clearly exceeded on Chinese prompts, though the abstract does not make the Chinese-specific claim for long context. The context length handling therefore represents one of GLM-4's strongest dimensions, with competitive English performance and clear Chinese leadership.

Real-World Coding (Table 6)

GLM-4 (0520) achieves an overall NCB score of 47.1%, compared to GPT-4 (0613) at 52.8%, GPT-4 Turbo (2024-04-09) at 53.8%, and Claude 3 Opus at 48.3%. The gap to GPT-4 models is 5–6 percentage points overall, which is larger than the gap on HumanEval (where GLM-4 at 78.5% is closer to GPT-4 Turbo at 88.2%, a gap of about 10 percentage points). The language-specific breakdown shows GLM-4 is stronger on Python (English) at 51.6% and weaker on Java (English) at 42.8%, consistent with the general observation that LLMs perform better on Python due to its prevalence in training data. On Chinese-language coding prompts, GLM-4 achieves 45.4% (Python) and 48.9% (Java), suggesting relatively balanced bilingual coding capability.

GLM-4-Air (0605) at 40.8% overall is substantially behind, and GLM-4-9B-Chat at 32.2% is at roughly the level of Claude 2 (34.4%). The report notes that HumanEval scores may be inflated due to training data contamination (referencing GPT-4 Technical Report, Li et al. 2023, and Yang et al. 2023), and that NCB is a more challenging and realistic benchmark—GLM-4's weaker performance on NCB versus HumanEval is consistent with this interpretation, since real-world coding prompts are harder than the introductory algorithm problems in HumanEval.

The claim in the abstract that GLM-4 "outperforms GPT-4 in ... HumanEval" is supported (78.5 vs. 67.0 for GPT-4 0314), but the NCB results show that this advantage does not necessarily generalize to practical coding tasks, where GPT-4 models maintain a lead. This highlights a common pitfall in LLM evaluation: benchmark-specific claims may not reflect real-world capability.

Function Call (Table 7)

GLM-4 (0520) achieves an overall accuracy of 81.76% on the Berkeley Function Call Leaderboard, compared to GPT-4 Turbo (2024-04-09) at 81.24%, GPT-4o at 82.94%, and GLM-4-9B-Chat at 81.00%. The most striking result is that GLM-4-9B-Chat (81.00%) nearly matches GLM-4 (0520) at 81.76% despite being a much smaller model, and substantially outperforms Llama-3-8B-Instruct at 58.88%. The execution summary metric, which measures the correctness of real-world API function calls by actually executing them, shows a clear scaling trend: GLM-4 (0520) at 87.78%, GLM-4-Air at 85.93%, GLM-4-9B-Chat at 84.40%, and ChatGLM3-6B at 69.78%. This monotonic improvement with model size on execution-based metrics contrasts with the overall accuracy, which does not improve monotonically (GLM-4-9B-Chat at 81.00% is higher than GLM-4-Air at 80.94%), suggesting that execution accuracy is a more reliable signal of genuine function-calling capability than AST-based matching.

The relevance detection metric shows interesting variation: GLM-4-9B-Chat achieves 87.92% (the highest of any model), while GLM-4-Air drops to 68.33%—a large discrepancy for models in the same family. The report does not explain this gap, but it could reflect differences in the alignment data or the precision-efficiency tradeoffs in the Air variant.

Agent Abilities (Table 8)

GLM-4 (0520) achieves an overall AgentBench score of 3.79, compared to GPT-4 Turbo (1106) at 3.77, GPT-4 (0613) at 3.69, and Claude 3 Opus at 3.62. The per-environment breakdown shows GLM-4's strengths in Database (52.7 vs. 52.7 for GPT-4 Turbo), House-Holding (82.0 vs. 70.0 for GPT-4 Turbo), and Web Shopping (68.3 vs. 55.1 for GPT-4 Turbo 2024-04-09). The weaknesses are in Operating System (36.8 vs. 42.4 for GPT-4 0613), Knowledge Graph (51.4 vs. 58.8 for GPT-4), and Lateral Thinking Puzzles (15.3 vs. 19.4 for GPT-4 Turbo 2024-04-09). The report interprets the gap on code-related agentic tasks and highly interactive language tasks as areas where there is still "room for GLM-4 to improve."

GLM-4-Air (0605) at 3.58 overall is competitive with Claude 3 Opus (3.62) and GPT-4 Turbo (2024-04-09) at 3.68, though the fine-grained scores show differences in which environments each model excels at. The overall AgentBench scores use the original per-dataset weights from the AgentBench paper, which means the summary statistic may not reflect the practical importance of different environments—a model that excels at Web Browsing and House-Holding (common use cases) might be practically more useful than one that excels at Lateral Thinking Puzzles, even if the weighted averages are similar.

All Tools (Table 9)

GLM-4 All Tools (Web, 0116) achieves 91.59% on GSM8K using the Python interpreter, compared to GPT-4 (Web, 0110) at 92.72%. On MATH with Python interpreter, GLM-4 All Tools scores 63.60% versus GPT-4's 65.00%. On the Chinese math benchmark Math23K, both models achieve similar scores (88.50% vs. 88.40%). For web browsing for information seeking, GLM-4 All Tools achieves 78.08%, substantially outperforming GPT-4's 67.12%. This 11 percentage point gap is the largest single advantage reported for GLM-4 in any capability dimension, and it supports the report's claim that GLM-4 All Tools "matches and even surpasses GPT-4 All Tools" for web browsing tasks.

The All Tools results are only reported for the 0116 version of GLM-4, not the later 0520 version that shows improvements on other benchmarks, so it is unclear whether the All Tools capabilities also improved between release checkpoints. Additionally, only one baseline (GPT-4 All Tools) is compared, without Claude or Gemini comparisons, making it difficult to assess whether the web browsing advantage is specific to GPT-4 or general across state-of-the-art models.

Ablation Studies and Robustness Checks

The report does not contain traditional ablation studies in the sense of removing or varying individual components and measuring the impact on performance. This is consistent with the report's nature as a comprehensive system description rather than a method paper proposing a specific technique. However, several implicit ablations can be extracted from the multi-generational and multi-variant comparisons:

Effect of pre-training data quality (generation-over-generation improvement at fixed model size): The progression from ChatGLM-6B to ChatGLM2-6B to ChatGLM3-6B to GLM-4-9B (Table 1) at roughly 6–9B parameters demonstrates the cumulative effect of improvements in data quality, architecture, and alignment. ChatGLM2-6B improved MMLU from 25.2% to 45.2% (a 20-point gain) over ChatGLM-6B, which the report attributes primarily to "high-quality data" and "more sufficient training steps." Given that the architectural changes between these generations (FlashAttention for context extension, Multi-Query Attention for speed) should not directly cause large reasoning improvements, the MMLU and GSM8K gains (1.5% → 25.9%) are strong circumstantial evidence that data quality is the dominant factor at this scale.

Effect of alignment quality (base model vs. chat model): The report presents results only for the aligned (Chat) versions of the models in most tables, but ChatGLM3-6B-Base is shown in Table 1 with MMLU of 61.4%, while the aligned ChatGLM3-6B presumably achieves higher scores on benchmarks that benefit from instruction tuning (though this is not directly shown). For GSM8K, ChatGLM3-6B achieves 72.3% (from the ChatGLM3-6B row, which is the chat version), while the base model's score is not separately reported. The absence of base model evaluations for the GLM-4 series makes it impossible to decompose the contributions of pre-training versus alignment to the final benchmark scores.

Effect of parameter scale (GLM-4 vs. GLM-4-Air vs. GLM-4-9B): Comparing GLM-4 (0520), GLM-4-Air (0605), and GLM-4-9B-Chat across Tables 2–8 provides an implicit scaling study, though the exact parameter counts of GLM-4 and GLM-4-Air are not disclosed. On MMLU (Table 2), the progression is 72.4% (9B) → 81.9% (Air) → 83.3% (0520). On GSM8K, the progression is 79.6% → 90.9% → 93.3%. On HumanEval, it is 71.8% → 75.7% → 78.5%. The diminishing returns from Air to 0520 (e.g., only 1.4 percentage points on MMLU) suggest that model scale beyond Air's size provides modest additional benefits on these benchmarks, with most of the gain coming from the jump from 9B to Air.

Effect of version updates (0116 vs. 0520): GLM-4 (0520) consistently outperforms GLM-4 (0116) across benchmarks where both are reported: MMLU (83.3 vs. 81.5), GSM8K (93.3 vs. 87.6), MATH (61.3 vs. 47.9), BBH (84.7 vs. 82.3), GPQA (39.9 vs. 35.7), HumanEval (78.5 vs. 72.0), and AlignBench overall (8.00 vs. 7.66). The MATH improvement is particularly large (+13.4 points), suggesting substantial improvements in math-specific training data or the application of the ChatGLM-Math self-critique pipeline between releases. These improvements demonstrate that the development pipeline continues to yield gains even after the initial GLM-4 release, though the specific changes between versions are not detailed.

Effect of prompting strategy (chain-of-thought vs. direct): The report notes that chain-of-thought prompting is used for GSM8K, MATH, and BBH evaluations (Section 3.1). No ablation is provided showing performance without chain-of-thought prompting, so the contribution of this prompting strategy to the reported scores cannot be separated from the model's base reasoning capability. This is a standard practice in the field, but it means that the reported numbers for these benchmarks reflect prompted performance rather than zero-shot reasoning.

Negative result: function call accuracy does not improve monotonically with model size: As noted earlier, GLM-4-9B-Chat achieves higher overall function call accuracy (81.00%, Table 7) than GLM-4-Air (80.94%), despite being a smaller model. The report does not explain this inversion, but it could indicate that the function call alignment data is more effective for the 9B model size, that the evaluation has high variance, or that the Air variant's efficiency optimizations (lower precision, architecture changes) disproportionately affect function call syntax generation.

Missing ablation: long-context alignment vs. architectural extension only: The report does not provide an ablation showing GLM-4 performance on LongBench-Chat without the LongAlign alignment stage, using only the architectural context extension and continued pre-training. Without this ablation, the report's claim that long-context alignment is crucial (Section 2) cannot be empirically validated from the data presented—it remains a methodological claim supported by the separate LongAlign paper but not directly demonstrated in this report's experiments.

Critical Assessment

Claim: "GLM-4 closely rivals or outperforms GPT-4 in terms of general metrics" (Abstract). This claim is supported against GPT-4 (0314), where GLM-4 (0520) outperforms on 5 of 6 benchmarks (Table 2: GSM8K 93.3 vs. 92.0, MATH 61.3 vs. 52.9, BBH 84.7 vs. 83.1, GPQA 39.9 vs. 35.7, HumanEval 78.5 vs. 67.0) and closely rivals on MMLU (83.3 vs. 86.4). However, the claim is substantially weaker against newer GPT-4 versions: GPT-4 Turbo (2024-04-09) leads on MMLU (86.7 vs. 83.3), MATH (73.4 vs. 61.3), GPQA (49.3 vs. 39.9), and HumanEval (88.2 vs. 78.5). The specific comparison point matters enormously—the claim holds for GPT-4 as originally released in early 2023 but does not hold for GPT-4 as of mid-2024. This is a genuine limitation of the evaluation: the models being compared are from different time periods, and the rapid pace of LLM development means that a claim about "outperforming GPT-4" depends heavily on which GPT-4 version is referenced.

Claim: "GLM-4 matches GPT-4 Turbo and Claude 3 for long context tasks" (Abstract). Strongly supported on English LongBench-Chat (Table 5: 87.3 vs. 87.2 and 87.7), and the claim is actually an understatement for Chinese (84.0 vs. 82.1 and 82.7). However, this claim is based on a single benchmark (LongBench-Chat), and long-context capability can manifest differently across different types of tasks (retrieval, summarization, multi-document synthesis, instruction following over long contexts). The reliance on GPT-4 as the judge for LongBench-Chat also introduces a potential confound: if GPT-4 has biases in evaluating model outputs (favoring outputs that resemble its own style), the relative rankings could be skewed. Multi-judge or human-evaluated long-context evaluations would strengthen this claim.

Claim: "GLM-4 outperforms GPT-4 in Chinese alignments as measured by AlignBench" (Abstract). Supported against GPT-4 (0613) at 7.46 overall (Table 4), but the comparison to GPT-4 Turbo (2024-04-09) is a tie at 8.00. The claim's strength depends on which GPT-4 version is referenced. Additionally, the AlignBench evaluation uses GPT-4 as judge, which raises the question of whether GPT-4 can impartially evaluate Chinese text, especially for models that may outperform it in Chinese. If GPT-4 has systematic weaknesses in Chinese (which the AlignBench results themselves suggest), using it as a judge for Chinese alignment may produce biased evaluations that favor models with similar error patterns to GPT-4 or disfavor models with different (but not necessarily worse) Chinese capabilities. Using human evaluators or multiple judges from different model families would address this concern.

Claim: "GLM-4 All Tools matches and even surpasses GPT-4 All Tools in tasks like accessing online information via web browsing and solving math problems using Python interpreter" (Abstract). The math claim is supported (Table 9: GSM8K 91.59 vs. 92.72, MATH 63.60 vs. 65.00—GLM-4 is slightly behind, not surpassing, on both), and the web browsing claim is strongly supported (78.08 vs. 67.12). However, the evaluation methodology for the All Tools comparison is not described in detail—the table simply reports numbers without specifying the number of test examples, whether the evaluation was automated or human-judged, or how partial credit was handled. For web browsing specifically, "information seeking" is a broad category that could encompass many different task types, and a single aggregate score may mask important variation. The report would be strengthened by a more detailed breakdown of the All Tools evaluation, including the number of tasks, the scoring rubric, and examples of failures.

Genuine weaknesses in the experimental design:

No statistical significance or confidence intervals. All reported numbers are point estimates without quantification of variance, despite some benchmarks having relatively small test sets (HumanEval: 164 problems; IFEval: 541 prompts; AlignBench: 683 queries). For benchmarks with test sets of a few hundred examples, differences of 1–2 percentage points between models may not be statistically meaningful. The repeated evaluation for LongBench-Chat is a positive exception, but the variance of those repetitions is not reported.
No compute-matched or parameter-matched comparisons. The comparisons are purely capability-based: the models are compared at their deployed scale without controlling for differences in parameter count or training compute. Since GPT-4's parameter count and training compute are not publicly known, this is not a fixable weakness, but it limits the conclusions that can be drawn—if GPT-4 is much larger than GLM-4, then matching or approaching its performance is an efficiency achievement; if it is similar in scale, the comparison is more straightforward. The absence of GLM-4's parameter count from the report makes this assessment impossible.
GPT-4 as judge for Chinese evaluations. Both AlignBench and LongBench-Chat (and possibly other evaluations) use GPT-4 as the automatic evaluator. This is standard practice but creates a circular evaluation problem when one of the claims is that GLM-4 matches or outperforms GPT-4 in Chinese. If GPT-4 is a flawed Chinese evaluator, the relative rankings could be distorted in ways that are difficult to detect without human evaluation or multi-judge baselines.
No base model evaluations for GLM-4. The report evaluates only the aligned (Chat) versions of GLM-4, making it impossible to determine how much of the benchmark performance comes from pre-training versus alignment. For a report that emphasizes the multi-stage post-training process, this is a notable omission—it would be informative to see, for example, GLM-4 base model MMLU to quantify the contribution of SFT and RLHF separately.
Single evaluation of All Tools capabilities. Table 9 provides only aggregate numbers for All Tools performance, with no breakdown by task type (for web browsing) or problem difficulty (for math). The claim that GLM-4 All Tools "matches and even surpasses GPT-4 All Tools" is based on essentially two data points (information seeking and Python math), which is a thin empirical foundation for a broad capability claim.
No safety evaluation for All Tools. While Section 4 evaluates GLM-4 on SafetyBench, the All Tools variant is not separately evaluated for safety. Since tool use introduces new safety risks (e.g., executing generated code, browsing the web and encountering harmful content, generating images with inappropriate content), the absence of All Tools-specific safety evaluation is a gap.
Potential benchmark contamination. The report acknowledges (Section 3.5) that HumanEval contamination has been reported in LLM training data, and that NCB is designed to address this. However, no contamination analysis is provided for the other benchmarks (MMLU, GSM8K, MATH, etc.), despite these being among the most commonly used LLM benchmarks with data publicly available on the internet. This is a field-wide issue, not specific to this report, but it means that the reported numbers may partially reflect memorization rather than generalization.

Missing experiments that would have strengthened the paper:

An ablation of long-context performance with and without the LongAlign alignment stage, to quantify the contribution of alignment-specific training versus architectural context extension alone.
Human evaluation of Chinese-language outputs (on AlignBench, LongBench-Chat, or IFEval) to validate the GPT-4-as-judge methodology and ensure that relative rankings are not distorted by evaluator bias.
A breakdown of All Tools evaluation by task difficulty or error type, to understand where the tool-use capability succeeds and fails.
Evaluation of the revision model's correct-to-incorrect reversion rate and its impact on multi-turn coherence—a phenomenon documented in other alignment papers but not quantified here.
A comparison of GLM-4 against open-source models beyond Llama-3-8B (e.g., Mistral, Qwen, Yi) to contextualize its position in the broader open-source landscape rather than only comparing to proprietary models.

Conditional nature of claims:

The report's central claims hold strongly when comparing against GPT-4 (0314) and Claude 2, hold conditionally against GPT-4 Turbo (1106) (GLM-4 is competitive but not superior on most English benchmarks), and hold weakly or not at all against GPT-4 Turbo (2024-04-09) (which leads on MMLU, MATH, GPQA, and HumanEval). The Chinese-specific claims hold more robustly across model versions, with GLM-4 consistently matching or exceeding the best available models on Chinese alignment and long-context Chinese tasks. The claim about All Tools surpassing GPT-4 All Tools holds for web browsing but not for math (where GLM-4 is slightly behind), and the evaluation is thin. Overall, the report demonstrates that GLM-4 is a competitive, state-of-the-art LLM with genuine strengths in Chinese-language tasks and specific agentic capabilities, but it does not establish universal superiority over the best proprietary models across all dimensions—and the rapid iteration of those proprietary models means that any static claim about "outperforming GPT-4" has a short shelf life that depends on which specific GPT-4 checkpoint is referenced.

6. Limitations and Trade-offs

Unresolved Cost of Difficulty Estimation for Compute-Adaptive Deployment

The assumption or constraint. The paper's compute-optimal framework (in the reference example) critically relies on knowing each prompt's difficulty before allocating inference compute. This difficulty is estimated by generating 2048 samples per prompt and scoring them—either by ground-truth correctness (oracle) or by the PRM's final-answer score (predicted). The authors explicitly acknowledge this gap:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2, reference example)

While this limitation is from the reference methodology rather than directly from the GLM paper, the broader principle applies: the GLM-4 report describes a sophisticated deployment pipeline involving multiple models (GLM-4, GLM-4-Air, GLM-4-9B) and capabilities (All Tools, function call, long-context) without any framework for adaptively routing user requests to the appropriate model variant or tool configuration based on task difficulty or complexity. The report evaluates each capability in isolation—the long-context model on long-context benchmarks, the All Tools model on tool-use tasks, the base model on academic benchmarks—but provides no mechanism for a production system to decide at runtime which configuration to use for an arbitrary user request, and no accounting for the overhead of making such decisions.

The consequence. In a deployment where all capabilities must be served (a user might ask a short factual question, upload a 100K-token document, or request a multi-step agentic task), the system must either: (1) always route to the full GLM-4 All Tools model with 128K context, paying the maximum inference cost on every query regardless of complexity, or (2) implement some form of request classification to route to lighter-weight configurations (GLM-4-Air, shorter context windows, tool-disabled mode), which itself adds latency, cost, and potential misrouting errors that the report does not quantify. The headline performance numbers for each capability assume that the system is already configured optimally for that particular task type—an assumption that does not hold in the mixed-workload reality of a production deployment. The efficiency claims for GLM-4-Air ("lower latency and inference cost") are presented as a static property of the model variant rather than as part of an adaptive system that would realize savings only if routing decisions are accurate. If 20% of tool-requiring requests are misrouted to GLM-4-Air (which has weaker agent performance as shown in Table 8: 3.58 vs. 3.79 overall AgentBench score), the apparent cost savings may be partially offset by degraded task completion rates.

What evidence exists in the paper. The report evaluates GLM-4, GLM-4-Air, and GLM-4-9B-Chat side-by-side across multiple benchmarks (Tables 2–8), quantifying the capability gap between model variants. For example, GLM-4-Air loses ~5 points on MMLU (81.9 vs. 83.3), ~2 points on GSM8K (90.9 vs. 93.3), and ~6 points on HumanEval (75.7 vs. 78.5) compared to GLM-4 (0520) (Table 2). On All Tools capabilities, GLM-4-Air scores substantially lower on function call overall (80.94% vs. 81.76%, Table 7), with a particularly large drop in relevance detection (68.33% vs. 84.17%)—a failure mode that would directly impact misrouting consequences. However, there is no evaluation of a routing system that combines these models, no measurement of routing accuracy or overhead, and no analysis of the Pareto frontier between cost and capability that a production deployment would need to navigate.

Mitigation status. Not addressed. The report presents model variants as independent products with different cost/capability tradeoffs rather than as components of an adaptive serving system. No routing classifier, difficulty estimation module, or dynamic configuration strategy is described. The reference methodology's proposed future work on "pretraining or finetuning models to directly predict difficulty" (Section 8, reference example) is not attempted or evaluated for the GLM family. A practitioner deploying GLM-4 would need to build this infrastructure from scratch, with no guidance from the report on expected performance or failure modes.

Hard Problems Remain Unsolved—No Capability Emergence Beyond Pre-Training Reach

The assumption or constraint. Like the reference methodology's finding that test-time compute provides essentially zero benefit on the hardest problems (difficulty bin 5, ~1–3% accuracy regardless of budget), the GLM-4 models exhibit a fundamental performance floor on the most challenging reasoning tasks that no amount of alignment, tool use, or context extension appears to overcome. The report does not explicitly frame this as a limitation, but the data reveals a consistent pattern: GLM-4's performance on the hardest benchmarks—MATH (61.3%, Table 2), GPQA (39.9%, Table 2), NCB (47.1% overall, Table 6), and Lateral Thinking Puzzles in AgentBench (15.3, Table 8)—remains substantially below ceiling despite outperforming earlier GPT-4 versions on several of these metrics. The model's pre-training determines a capability envelope, and post-training techniques do not create fundamentally new reasoning capabilities that were absent from the base model.

The consequence. For tasks that require novel problem-solving, deep multi-step reasoning, or integration of knowledge across disparate domains, GLM-4—like all current LLMs—has a non-trivial failure rate that limits its deployability in high-stakes or autonomy-critical settings. The GPQA result is particularly instructive: GPQA is designed to be "Google-proof," testing graduate-level domain knowledge that cannot be easily retrieved from the web. GLM-4's 39.9% accuracy (Table 2) means it fails on ~60% of these questions, despite having web browsing capability (which wouldn't help by design) and strong performance on knowledge-intensive benchmarks like MMLU (83.3%). This suggests a genuine reasoning gap—the model can retrieve and apply known patterns but struggles with the kind of novel synthesis that graduate-level science questions require. Similarly, the 15.3 score on Lateral Thinking Puzzles (Table 8) versus 19.4 for GPT-4 Turbo suggests that creative, counterintuitive reasoning remains a weakness across model families and is not substantially improved by the alignment techniques the report describes.

The practical implication is that GLM-4, despite matching or approaching GPT-4 on many benchmarks, should not be deployed in settings where correctness on hard reasoning tasks is required at high reliability—such as automated theorem proving, medical diagnosis support, or legal analysis where a single error has high cost. The All Tools capability does not solve this: a model that can use Python and web search is still limited by its base reasoning ability when those tools don't directly provide the answer (e.g., lateral thinking puzzles cannot be solved by executing code or searching the web).

What evidence exists in the paper. The evidence is distributed across multiple tables. Table 2 shows GLM-4 (0520) at 61.3% on MATH versus GPT-4 Turbo (2024-04-09) at 73.4%—a 12-point gap that has not closed despite the +13.4 point improvement from GLM-4 (0116) to (0520), suggesting diminishing returns from alignment improvements. Table 6 shows GLM-4 at 47.1% overall on NCB versus GPT-4 Turbo at 53.8%, with a particularly large gap on Java (English) at 42.8% versus 52.3%. Table 8 shows the Lateral Thinking Puzzles score of 15.3, the lowest of all evaluated environments, with even GPT-4 Turbo (2024-04-09) only reaching 19.4. The SafetyBench results (Table 10) show GLM-4 at 66.0 on Unfairness & Bias—a dimension requiring nuanced social reasoning—versus GPT-4 Turbo (2024-04-09) at 75.0. These are not failures of alignment or tool use; they reflect the base model's capacity to perform certain types of reasoning, and the report provides no evidence that post-training techniques systematically improve this capacity rather than making the model better at leveraging its existing capabilities.

Mitigation status. The report acknowledges these gaps implicitly by presenting the benchmark results transparently, but it does not discuss them as a fundamental limitation or propose architectural or training changes specifically targeting hard reasoning tasks. The ChatGLM-Math technique (self-critique for math problem solving) is referenced (Section 2) and likely contributed to the MATH improvement from 47.9% (0116) to 61.3% (0520), but the absolute level remains far from ceiling and the technique's returns appear to be diminishing. The report's forward-looking statement—"Currently, we are working on more capable models with everything we have learned to date" (Section 5)—is a generic statement of ongoing work rather than a specific plan to address the reasoning bottleneck. The report does not discuss whether scaling model size, changing the pre-training data mixture, or architectural innovations might address these gaps, leaving the practitioner to guess whether future GLM versions will show substantial improvements on the hardest tasks.

Undisclosed Model Scale and Training Compute Preclude Efficiency Claims Against Proprietary Baselines

The assumption or constraint. The report makes extensive performance comparisons between GLM-4 models and proprietary baselines (GPT-4, GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro) without disclosing the parameter counts or training compute of either the GLM-4 models or their proprietary competitors. The only disclosed scale is GLM-4-9B (9 billion parameters). The report states that the team trained "models with 1.5B, 3B, 12B, 32B, 66B, and 130B parameters" across generations (Section 1), implying that GLM-4 sits somewhere in this range—likely between 66B and 130B based on the progression—but no specific number is provided. The proprietary baselines similarly do not disclose their parameter counts. This means every capability comparison in the report is scale-uncontrolled: GLM-4 achieving 96.3% of GPT-4's MMLU score (83.3 vs. 86.4, Table 2) could represent either a genuine efficiency achievement (if GLM-4 is much smaller) or a scale disadvantage (if GLM-4 is comparable or larger). The report presents this comparison as a straightforward capability claim without acknowledging the interpretive ambiguity introduced by the missing scale information.

The consequence. The abstract's headline claim—"GLM-4 closely rivals or outperforms GPT-4 in terms of general metrics"—is fundamentally uninterpretable as an engineering achievement because the reader cannot assess the resource cost of achieving that performance. If GLM-4 achieves 83.3% MMLU at 70B parameters while GPT-4 achieves 86.4% at (speculatively) 1.8T parameters, then GLM-4 represents a dramatic efficiency breakthrough. If GLM-4 is similar in scale to GPT-4, then it's a more modest achievement of competitive capability with a slightly different architecture and training recipe. If GLM-4 is larger, then the comparison might actually be unfavorable. The absence of this information transforms the report from an engineering contribution (how to efficiently build a model of this capability) into a product announcement (our model achieves this capability level), where the reader cannot evaluate the methods' effectiveness independent of the resources consumed. This matters particularly for practitioners deciding whether to adopt the GLM architecture and training pipeline versus alternatives: without knowing the resource requirements, they cannot assess whether the techniques described (RMSNorm, SwiGLU, 2D RoPE, GQA, multi-stage alignment) actually improve efficiency or simply describe a well-resourced training run.

The same ambiguity applies to the open-source models. GLM-4-9B outperforms Llama-3-8B on MMLU (74.7 vs. numbers not shown in the report but stated as "outperforms Llama-3-8B" in Section 1), but the report does not compare training compute or data volume, making it unclear whether the performance difference reflects better architecture, better data, or simply more pre-training tokens (GLM-4-9B was trained on ~10T tokens, reported in the abstract; Llama-3-8B's training data volume is not discussed in the report).

What evidence exists in the paper. The report systematically avoids any compute-matched or parameter-matched analysis. The evaluation tables (2–8, 10) compare raw performance numbers without normalization by model scale. The one exception is the Berkeley Function Call Leaderboard results (Table 7), where GLM-4-9B-Chat (81.00% overall) approaches GLM-4 (0520, 81.76%) despite being much smaller, demonstrating that function-calling capability does not scale linearly with model size—but this is an incidental observation, not a systematic efficiency analysis. The report's acknowledgment of this gap is entirely implicit: the fact that GLM-4's parameter count is not mentioned anywhere in the 19-page report, despite detailed discussion of architectural choices and training data volumes, is itself a deliberate omission that the report does not explain or justify.

Mitigation status. Not addressed. The report does not discuss the rationale for withholding model scale, does not provide any compute-matched or scale-controlled comparisons, and does not frame its results in terms of efficiency or resource requirements. A practitioner reading the report cannot answer basic deployment-planning questions like: "How many GPUs do I need to serve GLM-4 at production throughput?" or "How does GLM-4's inference cost per token compare to GPT-4's API pricing?" The report's focus on end-user capability metrics (accuracy, benchmark scores) is appropriate for a product capability report but insufficient for a technical contribution that claims to describe "the insights and lessons gained from the preceding three generations"—insights about efficiency and scaling tradeoffs are among the most valuable lessons in LLM development, and their absence is a significant gap.

All Tools Capabilities Evaluated on Thin, Methodologically Under-Specified Benchmarks

The assumption or constraint. The GLM-4 All Tools evaluation (Section 3.8, Table 9) consists of exactly four numbers: Python interpreter performance on GSM8K, MATH, and Math23K, and web browser performance on "Information Seeking." No evaluation is provided for the text-to-image tool (CogView3), user-defined function calling in an agentic context, or multi-tool coordination (e.g., web browsing followed by Python computation, which is the signature All Tools scenario illustrated in Figure 2). The evaluation methodology for the existing numbers is not described: the number of test examples, the scoring rubric, whether evaluation was automated or human-judged, how partial credit was handled, and whether the web browsing evaluation was conducted in a controlled or live environment are all unspecified. The comparison baseline is a single model (GPT-4 All Tools, version 0110) with no Claude, Gemini, or open-source agent comparisons.

The consequence. The abstract's claim that GLM-4 All Tools "matches and even surpasses GPT-4 All Tools in tasks like accessing online information via web browsing and solving math problems using Python interpreter" rests on an empirically fragile foundation. The math interpreter comparison (Table 9) actually shows GLM-4 slightly behind on GSM8K (91.59 vs. 92.72) and MATH (63.60 vs. 65.00), making the "matches" claim accurate but the "surpasses" claim unsupported for math. The web browsing advantage (78.08 vs. 67.12) is substantial (+11 percentage points) but represents a single aggregate number from an unspecified evaluation. Without methodological detail, the reader cannot distinguish between: (1) a genuine capability improvement where GLM-4 more effectively browses the web to find and extract information; (2) a domain-specific advantage where the evaluation queries favor Chinese-language web sources that GLM-4 handles better; (3) a methodological artifact where the evaluation protocol or scoring was not identical between the two models; or (4) a transient result that would not replicate with a different set of queries, a different web environment, or a different version of the baseline model. The All Tools capability is the most commercially significant differentiator GLM-4 claims—it's the feature that transforms the model from "another GPT-4 competitor" to "an agentic platform with web browsing, coding, and image generation"—and the empirical support for this differentiator is the thinnest in the entire report.

The omission of multi-tool coordination evaluation is particularly notable because the paper's Figure 2 explicitly illustrates this as the signature use case: searching for population data (web browser) and then calculating growth rate (Python interpreter) in a single coordinated workflow. The report provides no quantitative evaluation of whether GLM-4 All Tools can reliably execute such multi-tool sequences, recover from tool failures (e.g., a search that returns no results, a Python script with a runtime error), or decide between alternative tool-use strategies (e.g., when a problem could be solved either via web search or via Python computation). These are the capabilities that distinguish an agent from a collection of isolated tool-use skills, and they are unevaluated.

What evidence exists in the paper. Table 9 is the entirety of the All Tools quantitative evaluation. Section 3.8 describes the evaluation in one paragraph of text, noting only the benchmark names and the comparison model version. No information is provided about the test set size, the evaluation protocol, or the scoring methodology. The separate function call evaluation (Section 3.6, Table 7) and agent evaluation (Section 3.7, Table 8) provide broader capability assessments, but these use standard benchmarks (Berkeley Function Call Leaderboard, AgentBench) that evaluate general function-calling and agentic capabilities rather than the specific All Tools integration of web browser, Python interpreter, and CogView3 that the commercial product offers. The custom GLMs platform (Figure 4) is described qualitatively with no evaluation at all.

Mitigation status. Not addressed. The report does not acknowledge the thinness of the All Tools evaluation as a limitation, does not provide methodological details, and does not discuss plans for more comprehensive agentic evaluation. The reference to the separate AgentBench evaluation (Table 8) partially mitigates the concern about general agentic capability, but AgentBench tests a different set of environments (Operating System, Database, Knowledge Graph, Web Shopping, Web Browsing, etc.) that do not correspond to the specific tool constellation (web browser + Python + CogView3) that defines the All Tools product. A practitioner deciding whether to build on GLM-4 All Tools cannot assess the reliability of multi-tool workflows, the model's error recovery behavior, or the consistency of tool-selection decisions from the data provided.

Safety Evaluation Does Not Cover Tool-Augmented Scenarios and May Not Transfer Across Deployment Contexts

The assumption or constraint. The safety evaluation in Section 4 uses SafetyBench, a multiple-choice question benchmark that assesses the model's knowledge of safety-related topics (e.g., "Is it ethical to do X?" with multiple-choice options). The evaluation is conducted on the Chinese subset of SafetyBench with "highly sensitive questions that tend to be censored" removed "to mitigate interference from different API safety policies" (Section 4). This evaluation methodology has three critical scope limitations: (1) it tests safety knowledge (can the model identify which of four options is the safest response?) rather than safety behavior (does the model actually refuse harmful requests in open-ended conversation?), (2) it evaluates only the base GLM-4 model, not the All Tools variant where new safety risks emerge (generating and executing code, browsing live web content, generating images), and (3) it evaluates a curated subset of safety scenarios that deliberately excludes the most sensitive content, potentially overstating real-world safety performance.

The consequence. SafetyBench accuracy (Table 10) cannot be interpreted as a measure of deployment safety. A model that achieves 92.3% on Ethics & Morality multiple-choice questions may still comply with 20% of unethical open-ended requests in practice, because multiple-choice evaluation tests a different capability (normative knowledge) than real-world safety (refusal behavior under adversarial prompting). This is a well-documented gap in the LLM safety literature that the report does not discuss. More critically, the All Tools variant introduces qualitatively new safety failure modes that are entirely unevaluated: the web browser could retrieve harmful content (instructions for illegal activities, hate speech, misinformation) and incorporate it into model responses; the Python interpreter could execute code that performs harmful operations (file system access, network requests, resource exhaustion); CogView3 could generate inappropriate or harmful images; and combinations of these tools could create complex harm pathways (e.g., searching for personal information about an individual and then using that information to craft a targeted harassment message) that are not captured by any single-tool safety metric. The removal of "highly sensitive questions" from the SafetyBench evaluation means the reported numbers may be higher than what would be observed in an unfiltered evaluation, and the gap between reported and actual safety performance is unquantified.

The Chinese-specific focus of the safety evaluation also limits generalizability. The SafetyBench results (87.2% overall, Table 10) show GLM-4 competitive with Claude 3 Opus (87.5%) and approaching GPT-4 models (87.9–89.7%). However, safety is highly culturally and legally context-dependent—what constitutes offensive content, what types of speech are restricted, and what safety norms apply vary significantly across jurisdictions. A model that is safe by Chinese content standards may not meet EU AI Act requirements or US harm reduction standards, and vice versa. The report's Chinese-focused safety evaluation does not provide evidence about GLM-4's safety characteristics in other linguistic or regulatory contexts, which limits the deployability assessment for international practitioners.

What evidence exists in the paper. Table 10 provides the only safety evaluation. The per-dimension breakdown shows GLM-4's strongest safety dimensions are Mental Health (93.3) and Ethics & Morality (92.3), while its weakest are Unfairness & Bias (66.0) and Physical Health (92.3—the report notes this dimension "demands robust common sense knowledge about the physical world to avoid potential risks"). The report acknowledges the Physical Health gap (Section 4: "GLM-4 slightly falls behind the GPT-4 family, especially on the Physical Health dimension") but does not discuss the broader limitations of the evaluation methodology. The red-teaming effort is described qualitatively ("We have a red team that constantly challenges the model with tricky questions") but no quantitative results from red-teaming are reported that could validate or contextualize the SafetyBench scores. The mitigation description—"We carefully cleaned data in the pre-training stage... In the alignment phase, we evaluate each training sample for safety... Harmlessness is also an important criteria for preference alignment"—describes processes but not outcomes.

Mitigation status. Partially acknowledged through the Physical Health gap discussion, but the fundamental mismatch between SafetyBench and deployment safety is not addressed. The report does not discuss open-ended safety evaluation (e.g., refusal rates on harmful prompt datasets like Anthropic's harmfulness benchmarks or the Do-Not-Answer dataset), does not evaluate All Tools safety, and does not provide cross-lingual or cross-jurisdictional safety comparisons. The statement "More efforts have been put into this direction to develop a more capable and safe GLM model" (Section 4) is a forward-looking commitment without specificity. A practitioner considering GLM-4 for deployment in a safety-sensitive application (education, healthcare, customer-facing services) would need to conduct their own comprehensive safety evaluation, since the report's safety evidence does not generalize to open-ended, tool-augmented, or cross-cultural deployment scenarios.

Cross-Generational Comparisons Confound Data Quality, Architecture, and Scale—No Clean Ablations

The assumption or constraint. The report presents the progression from ChatGLM-6B to GLM-4-9B (Table 1) as a narrative of cumulative improvement: "the progressive improvements of ChatGLM over time." However, each generation changed multiple factors simultaneously—pre-training data composition and quality, model architecture, alignment data and methodology, training duration, and (potentially) model scale—making it impossible to attribute the observed gains to any specific change. When ChatGLM2-6B improves MMLU from 25.2% to 45.2% over ChatGLM-6B (Table 1), the report attributes this to "more high-quality data" and "more sufficient training steps" (Section 1), but this attribution is post-hoc: no ablation is performed where the improved data is used with the original architecture, or the new architecture is used with the original data, or training steps are varied independently. The improvements could be driven primarily by data quality, primarily by architecture (FlashAttention for context extension, Multi-Query Attention), primarily by increased training compute (more tokens), or any interaction among these factors. The report's claim to have learned "insights and lessons" from each generation is undermined by the inability to causally attribute which insights actually drove the improvements.

The consequence. The report functions as a product evolution narrative rather than a scientific contribution about what matters in LLM development. A practitioner who wants to replicate the GLM team's success—building competitive bilingual LLMs with limited resources—needs to know which of the many changes between generations were actually important. Should they prioritize data quality and filtering (the report's emphasis in Section 2: "data quality and diversity are crucial"), or is the specific architecture recipe (no-bias-except-QKV, RMSNorm, SwiGLU, 2D RoPE, GQA) the key differentiator, or is the multi-stage alignment pipeline (SFT + RLHF + Self-Contrast + specific techniques like ChatGLM-Math and AgentTuning) doing most of the work? The report provides no evidence to discriminate among these hypotheses because every generation changed all of them simultaneously. This is a missed opportunity: the GLM team, with their multi-generation development history and models at multiple scales, is in a unique position to conduct controlled comparisons—training two models at the same scale with different data, or the same data with different architectures, or the same architecture and data with different alignment recipes—and the absence of such ablations means the report does not extract the scientific value that the development history could provide.

The architectural discussion in Section 2 compounds this issue by listing design choices (RMSNorm, SwiGLU, GQA, 2D RoPE) with brief justifications ("brought better model performance," "to cut down on the KV cache size") but without quantifying the contribution of each change. A practitioner wondering whether to adopt SwiGLU in their own model (which adds complexity and slightly increases parameter count for a given hidden size) cannot assess whether the performance gain justifies the cost because the gain is confounded with every other change between generations. The report's reference to separate technique papers (LongAlign, ChatGLM-Math, ChatGLM-RLHF, Self-Contrast, AgentTuning) partially mitigates this by providing detailed ablations for specific techniques in their original publications, but the interaction effects between techniques and the cumulative contribution of each to the final GLM-4 performance remain unknown.

What evidence exists in the paper. Table 1 is the primary quantitative evidence for the generation-over-generation narrative. The dramatic improvements on specific benchmarks—GSM8K from 1.5% to 25.9% to 72.3% to 84.0% across four generations—are presented as a cumulative benefit of all changes combined, with no variance decomposition. Section 1 provides qualitative attributions (e.g., "ChatGLM2-6B was pre-trained and aligned with more high-quality data, leading to substantial improvements"), and Section 2 lists the techniques used but does not map them to specific performance gains. The report's explicit acknowledgment of the field-wide knowledge gap—"we have to date yet to identify a fundamental principle that could guide the processes of data collection, cleaning, and selection" (Section 2)—is a candid admission that the data improvements driving inter-generational gains are empirically discovered rather than theoretically guided, but this candor does not substitute for controlled experiments that could identify which empirical discoveries actually mattered.

Mitigation status. Not addressed in the report. The separate technique papers (LongAlign, ChatGLM-Math, etc.) provide ablations for individual techniques, and the multi-scale model training (1.5B to 130B parameters mentioned in Section 1) suggests the team has infrastructure for controlled scaling experiments, but this infrastructure is not used to provide clean inter-generational ablations in this report. The report's framing as a comprehensive system description rather than a method paper makes this limitation somewhat inherent—a system report necessarily describes the final integrated product—but the scientific contribution ("insights and lessons gained") could have been strengthened by even a small number of targeted ablations, such as: training a ChatGLM3-scale model with GLM-4's data pipeline to isolate data quality effects, or training a GLM-4-scale model without the All Tools alignment to isolate agentic capability contributions. Without such ablations, the report's lessons remain suggestive rather than demonstrated.

7. Implications and Future Directions

How This Work Changes the Landscape

This report shifts the conversation around multilingual LLM development from a scale-centric narrative (where bilingual or multilingual capability is an emergent property of sufficiently large web crawls) to an architecture-and-alignment-centric narrative where linguistic parity is achieved through deliberate design of the tokenizer, pre-training data mixture, and alignment pipeline. The field's dominant assumption—embodied by GPT-4, Claude, and Gemini, all of which treat non-English performance as a byproduct of web-scale data ingestion—has been that you cannot realistically compete with English-first models on English benchmarks while simultaneously leading on a non-English language. The GLM-4 results falsify this assumption: a model can match GPT-4 Turbo on English general benchmarks (MMLU 83.3 vs. 86.7, Table 2) while outperforming it on Chinese alignment (AlignBench 8.00 vs. 7.90, Table 4) and Chinese long-context tasks (LongBench-Chat 84.0 vs. 82.1, Table 5), provided the pre-training and alignment treat the non-English language as a first-class design target rather than as incidental data.

This is not a paradigm shift in the sense of introducing a fundamentally new architecture or training objective—the GLM architecture's autoregressive blank infilling dates to 2021, and the individual techniques (RMSNorm, SwiGLU, GQA, RLHF) are standard in the field. Rather, it is a methodological reframing with practical consequences: it demonstrates that the path to strong multilingual models does not require out-scaling the largest proprietary efforts, but instead requires making linguistically informed choices about where to allocate fixed resources. The concrete decisions that distinguish GLM-4 from English-centric models with incidental multilingualism include: (1) the three-way tokenizer merge that separately learns Chinese and multilingual BPE vocabularies and combines them with the English-optimized cl100k_base vocabulary into a 150,000-token unified set (Section 2), (2) the two-dimensional RoPE that accommodates the GLM blank-infilling objective's dual sentence-and-span positional structure, and (3) the Chinese-focused alignment data and evaluation (AlignBench, SafetyBench Chinese subset) that treat Chinese-language quality as a primary metric rather than a secondary check. Each of these is individually modest, but collectively they produce a model that genuinely leads on Chinese while remaining competitive on English—a capability profile that no English-first model has demonstrated.

The report also reframes the agentic capabilities problem as primarily an alignment and training challenge rather than an infrastructure challenge. Prior to GLM-4 All Tools, the dominant approach to LLM tool use involved external orchestration frameworks (LangChain, AutoGPT) or API-level function calling where the model generates tool-use syntax that an external system parses and executes. The GLM-4 approach—aligning the model to autonomously understand user intent, plan multi-step instructions, and recursively call tools as part of its learned behavior rather than relying on external control flow (Figure 4)—represents a shift in where the "intelligence" of tool use lives. The evidence that this internalization works comes from the Table 9 web browsing comparison, where GLM-4 All Tools achieves 78.08% versus GPT-4's 67.12% on information seeking, an 11-percentage-point gap that is the largest single advantage reported in the entire evaluation. If this result is robust (acknowledging the thin evaluation noted in Section 6), it suggests that training a model to internalize tool-use decision-making genuinely outperforms relying on external orchestration, which would redirect research investment from better orchestration frameworks to better tool-use alignment data and training protocols.

The report partially reconciles a tension in the LLM agent literature between "agents as prompted behavior" (ReAct, Toolformer-style approaches where tool use is elicited through few-shot prompting of a general-purpose model) and "agents as trained capability" (approaches where the model is fine-tuned on tool-use trajectories). The GLM-4 results suggest that for complex, multi-tool scenarios—particularly those requiring sequential coordination like the web-search-then-Python-calculate workflow in Figure 2—training-based approaches substantially outperform prompting-based approaches, since the model needs to learn not just the syntax of tool calls but the meta-cognitive skill of recognizing when a tool is needed, which tool is appropriate, and how to recover from tool failures. The AgentTuning framework (Zeng et al., 2023) and the AgentInstruct dataset of "high-quality interaction trajectories between agents and environment" (Section 2) represent a concrete instantiation of this philosophy, and the strong AgentBench results (GLM-4 at 3.79 overall, Table 8, outperforming GPT-4 Turbo at 3.77) suggest the approach works for general agentic tasks beyond the specific All Tools constellation.

The report also makes research on Chinese-specific LLM evaluation more attractive and research on English-only evaluation less defensible. By developing and releasing multiple Chinese-language benchmarks (AlignBench, LongBench-Chat Chinese portion, the Chinese subset of IFEval, SafetyBench Chinese subset, NaturalCodeBench Chinese prompts), the GLM team has created infrastructure that makes it easier for subsequent researchers to evaluate Chinese-language performance rigorously. The fact that GLM-4's strongest results are on these Chinese-specific benchmarks—while being competitive but not leading on English benchmarks—serves as an existence proof that genuine Chinese-language leadership is achievable, which should increase the perceived value of Chinese-language evaluation in the broader LLM research community. Research directions that focus exclusively on English benchmarks, or that treat multilingual evaluation as a secondary table in the appendix, become harder to justify when a model family has demonstrated that Chinese-first development produces globally competitive models.

Follow-Up Research This Work Enables

Quantifying the contribution of each inter-generational change through controlled ablation. The most scientifically valuable follow-up to this report would decompose the ~50-point MMLU improvement from ChatGLM-6B (25.2%) to GLM-4-9B (74.7%, Table 1) into the contributions of data quality, architecture changes, alignment, and training duration. The GLM team is uniquely positioned to conduct this analysis because they have intermediate checkpoints, training data snapshots, and model variants at multiple scales. A strong follow-up would train a series of models at fixed 6–9B scale that vary one factor at a time—e.g., ChatGLM3 architecture with GLM-4 data, GLM-4 architecture with ChatGLM3 data, ChatGLM3 architecture with ChatGLM3 data but GLM-4 alignment—and report the MMLU, GSM8K, and BBH deltas attributable to each factor. This would transform the report's narrative from "we made many changes and things got better" to a causal decomposition that the field can use to prioritize investment: if data quality accounts for 60% of the gain and architecture only 10%, then practitioners building multilingual models should invest disproportionately in data curation over architectural experimentation. The existing report provides qualitative attributions (Section 1: "ChatGLM2-6B was pre-trained and aligned with more high-quality data, leading to substantial improvements") but no quantitative decomposition. A clean ablation study would be the single highest-impact follow-up, with implications for every team building LLMs under resource constraints.

Stress-testing the All Tools web browsing advantage against diverse information-seeking tasks and live-web environments. The 11-percentage-point gap between GLM-4 All Tools (78.08%) and GPT-4 All Tools (67.12%) on web browsing for information seeking (Table 9) is the report's most striking single result, but it comes from a methodologically opaque evaluation with unknown test set composition. A rigorous follow-up would replicate this comparison using a publicly documented benchmark of 200–500 diverse information-seeking tasks—ranging from factual lookup ("What was the closing price of stock X on date Y?") to multi-hop synthesis ("Compare the economic growth rates of countries A, B, and C over the last decade") to time-sensitive queries ("What are the current top 3 news stories about topic Z?")—and evaluate both models in a controlled web environment (e.g., a cached snapshot of web pages to eliminate temporal variation) with human evaluation of response accuracy, completeness, and source attribution. The key question is whether GLM-4's advantage generalizes across task types or is concentrated in specific categories (e.g., Chinese-language web sources, factual lookup, structured data extraction). A finding that the advantage persists across categories would strengthen the case that internalized tool-use training is genuinely more effective than external orchestration; a finding that it disappears or reverses for certain task types would reveal boundary conditions that inform when each approach should be preferred. Additionally, evaluation of multi-tool coordination sequences (the Figure 2 scenario where web browsing feeds into Python computation) with metrics for both final answer accuracy and intermediate tool-use correctness (did the model choose the right tool? did it execute the tool correctly? did it recover from tool failures?) would assess the capability that the All Tools branding promises but the current evaluation does not measure.

Cross-lingual transfer of Chinese-specific alignment techniques to other structurally distinct languages. The report demonstrates that Chinese-English bilingualism can be achieved through deliberate design (tokenizer, data mixture, alignment focus) rather than incidental emergence. A natural extension is to test whether the same recipe transfers to languages that share structural properties with Chinese (logographic writing, limited whitespace, complex character-level semantics)—specifically Japanese, Korean, and to a lesser extent Arabic and Hindi. A follow-up study would train models of comparable scale (~9B parameters) using the GLM-4 architecture and alignment pipeline but with the Chinese-specific components adapted: a separately trained tokenizer for the target language merged with cl100k_base; a pre-training data mixture that heavily weights the target language alongside English; and alignment data and evaluation benchmarks in the target language. The key measurements would be: (1) whether the target-language performance matches or exceeds English-centric models of similar scale (the equivalent of GLM-4's AlignBench leadership), (2) whether English performance is maintained or degraded (testing if the bilingual recipe genuinely preserves both languages rather than trading one for the other), and (3) which components of the recipe are language-specific (likely: tokenizer design, data mixture) versus language-agnostic (likely: RMSNorm, SwiGLU, GQA, multi-stage alignment). A negative result—where the recipe works for Chinese but fails for Japanese despite their structural similarities—would reveal that the approach depends on factors not captured by linguistic typology (e.g., the specific quality and volume of available training data for each language, cultural alignment challenges that differ across languages), which would be as informative as a positive result for guiding multilingual LLM development strategy.

Safety evaluation of tool-augmented models in open-ended adversarial settings. The report's safety evaluation (Table 10) uses SafetyBench, a multiple-choice benchmark that tests safety knowledge rather than safety behavior, and does not evaluate the All Tools variant at all. A critical follow-up would stress-test GLM-4 All Tools against a comprehensive adversarial safety benchmark specifically designed for tool-augmented models. This benchmark would include: (1) prompts that attempt to elicit harmful tool use ("Search for instructions on how to synthesize compound X and then write a Python script to calculate the required quantities"), (2) prompts that attempt to bypass safety filters through tool chaining (using one tool to retrieve content that another tool would refuse to process), (3) prompts that exploit the model's autonomous tool-selection capability to choose tools that are less safety-constrained (e.g., using the web browser when the Python interpreter correctly refuses a harmful computation), and (4) measurement of whether the model's safety behavior is consistent across tools and languages (does it refuse harmful requests in Chinese with the same reliability as in English?). The evaluation should use open-ended generation with human safety judgments rather than multiple-choice questions, and should report refusal rates, harmful completion rates, and the specific tool-use paths that lead to safety failures. This follow-up matters because GLM-4 All Tools is positioned as a commercial product (accessible via chatglm.cn and mobile applications), and the absence of tool-augmented safety evaluation represents a deployment risk that the current report does not address. Positive results (GLM-4 maintains safety across tools and languages) would strengthen the case for internalized tool use as a safe deployment paradigm; negative results (new failure modes emerge from tool integration) would identify specific safety alignment gaps that need to be closed before widespread deployment.

Scaling the open-source model to match closed-source performance with documented training recipes. GLM-4-9B achieves 74.7% MMLU and 84.0% GSM8K (Table 1), outperforming Llama-3-8B and supporting all the All Tools functionality of its larger siblings, but it remains substantially behind GLM-4 (0520) on most metrics (72.4% vs. 83.3% MMLU, 79.6% vs. 93.3% GSM8K). The open-source community's most pressing question is whether the gap between open and closed models can be closed purely through scale (more parameters, more training tokens) or whether the closed models benefit from qualitatively different training data, alignment, or post-processing that is not captured in the open recipe. A concrete follow-up would be to release intermediate-scale open models (e.g., 30B, 70B) trained with exactly the same pipeline and data as GLM-4-9B but with proportionally scaled compute, and to document the scaling trajectory for key benchmarks. If the performance gap closes smoothly with scale (suggesting the open recipe is fundamentally sound and the gap is primarily a resource gap), then the open-source community's path to matching closed-source performance is clear. If the gap persists at larger scales or shows diminishing returns (suggesting the closed models have qualitative advantages beyond scale—better data, more sophisticated alignment, or proprietary post-training), then the open-source community needs to identify and close those qualitative gaps. The GLM team, having trained models at 1.5B, 3B, 12B, 32B, 66B, and 130B parameters (Section 1), has the infrastructure to conduct this scaling analysis but has not published the results. Releasing these scaling curves would provide the most actionable guidance available for open-source LLM development.

Measuring and mitigating the correct-to-incorrect reversion rate in multi-turn agentic settings. The report does not quantify a phenomenon that is well-documented in other alignment work: models that have been trained to revise or self-correct often incorrectly "revise" correct answers into incorrect ones when the training data only contains incorrect-to-correct trajectories. For GLM-4 All Tools, this manifests as a potential failure mode in multi-tool sequences where the model correctly executes an early tool call but then, in a subsequent step, decides to "revise" the correct intermediate result—for example, correctly computing a population growth rate via Python but then, after a web search that retrieves conflicting information, overriding the correct computation with incorrect web data. A follow-up study would measure the reversion rate in multi-tool scenarios by constructing evaluation tasks where the correct answer can be obtained reliably from the first tool call (e.g., the Python interpreter on a deterministic math problem) and then measuring how often subsequent tool calls or model reasoning steps change the correct answer to an incorrect one. If the reversion rate is non-trivial (as the ~38% rate reported in the reference methodology's revision model suggests), then the All Tools model's effective accuracy on multi-tool tasks is lower than its single-tool accuracy would predict, and specific mitigation strategies—such as training the model to recognize when no revision is needed, or adding "do not override correct results" examples to the alignment data—would be necessary for reliable agentic deployment.

Practical Applications and Downstream Use Cases

Chinese-language enterprise search and knowledge management. Organizations with large repositories of Chinese-language documents—legal firms handling Chinese contract law, financial institutions analyzing Chinese market reports, academic institutions with Chinese-language research archives—can deploy GLM-4 with 128K context for document Q&A, summarization, and multi-document synthesis. The combination of competitive English long-context performance (LongBench-Chat 87.3, Table 5) and leading Chinese long-context performance (84.0 vs. 82.7 for Claude 3 Opus) means a single model can serve both English and Chinese document workflows without degradation in either language. The 1M context experimental model (GLM-4-9B-Chat-1M, mentioned in the abstract) would extend this to very long documents (up to ~2 million Chinese characters), though it is described as experimental and its performance on LongBench-Chat at 1M context is not reported. For organizations currently using separate English and Chinese models (with the Chinese model being substantially weaker due to the English-centric nature of most LLMs), consolidating on GLM-4 could reduce operational complexity while improving Chinese-language task accuracy.

Autonomous web research and data analysis pipelines. The All Tools capability, as illustrated in Figure 2 (web search for population data → Python CAGR calculation), can automate workflows where information must be gathered from live web sources and then analyzed computationally. Specific applications include: competitive intelligence (search for competitor pricing → compute price differentials), financial research (retrieve earnings reports → calculate financial ratios), academic literature review (search for papers on a topic → extract and compare reported effect sizes), and market research (search for product reviews → sentiment analysis via Python). The 78.08% information-seeking accuracy (Table 9) versus GPT-4's 67.12% suggests that for web-browsing-intensive tasks, GLM-4 All Tools has a meaningful practical advantage—processing ~11% more queries correctly per batch. For an organization processing thousands of research queries per day, this accuracy gap translates to hundreds of queries that would otherwise require manual researcher intervention. The Python interpreter performance on math (GSM8K 91.59%, MATH 63.60%, Table 9) makes this pipeline reliable for quantitative analysis tasks where computational accuracy matters, though the MATH score indicates that complex mathematical reasoning remains error-prone and should be verified for high-stakes applications.

Deployment of capable open-source bilingual agents on consumer hardware. GLM-4-9B-Chat supports all the All Tools functionality of the larger GLM-4 models (web browser, Python interpreter, text-to-image, user-defined functions) while being deployable on consumer GPUs via INT4 quantization (referenced in Section 1 for ChatGLM-6B). At 81.00% on the Berkeley Function Call Leaderboard (Table 7)—essentially matching GLM-4 (0520) at 81.76% and outperforming Llama-3-8B-Instruct at 58.88%—the 9B model provides near-state-of-the-art function-calling capability in an open-source package. This enables privacy-sensitive applications where data cannot be sent to cloud APIs (medical, legal, financial) to still benefit from agentic LLM capabilities: a hospital could deploy GLM-4-9B locally to help clinicians search internal medical databases and perform dosage calculations without patient data leaving the premises. The combination of strong function-calling (81.00%), competitive Chinese alignment (AlignBench 7.01, Table 4), and open-source availability (10M+ downloads across the ChatGLM family) makes GLM-4-9B the strongest available option for practitioners who need bilingual agentic capabilities with local deployment.

Chinese-language safety evaluation and red-teaming infrastructure. The report's development of Chinese-specific safety benchmarks (SafetyBench Chinese subset, Table 10) and its description of a red-teaming process ("We have a red team that constantly challenges the model with tricky questions," Section 4) provide a template for organizations that need to evaluate and improve LLM safety in Chinese-language contexts. Companies deploying LLMs in the Chinese market—which has distinct content regulations, cultural safety norms, and harm categories compared to English-language markets—can adopt the SafetyBench dimensions (Ethics & Morality, Illegal Activities, Mental Health, Offensiveness, Physical Health, Privacy & Property, Unfairness & Bias) as a starting framework for their own safety evaluation, extending it with open-ended red-teaming and tool-augmented safety testing as the follow-up research described above would recommend. The report's transparency about safety gaps ("GLM-4 slightly falls behind the GPT-4 family, especially on the Physical Health dimension," Section 4) provides specific targets for improvement that safety teams can prioritize.