GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

🎯 Pitch

Treating vision as a first-class reasoning component instead of a preprocessing step lets a multimodal model match its text-only sibling on code while dominating agent tasks—achieving 94.8 on Design2Code and 75.7 on AndroidWorld without sacrificing text performance. The key insight is that agent competence emerges better from hierarchical optimization across 30+ task categories than from monolithic end-to-end training.

1. Executive Summary

This report describes the development of GLM-5V-Turbo, a foundation model that integrates multimodal perception as a core component of reasoning, planning, tool use, and execution rather than treating it as an auxiliary interface to a language model. Built on top of the GLM-5-Turbo language model with a novel vision encoder (CogViT) and a Multimodal Multi-Token Prediction (MMTP) training objective (replacing visual embeddings with a shared learnable <|image|> placeholder token to improve training stability and system efficiency), the model undergoes broad joint reinforcement learning over more than 30 task categories spanning perception, reasoning, and agentic capabilities. GLM-5V-Turbo achieves strong results on multimodal agentic benchmarks — including 75.7 on AndroidWorld, 62.3 on OSWorld, 30.7 on the newly introduced vision-centric deep search benchmark ImageMining, and 94.8 on Design2Code — while preserving competitive text-only coding performance (22.8 on CC-Backend, 68.4 on CC-Frontend), establishing that native multimodal agentic capability can be built without sacrificing text-based reasoning when perception, hierarchical optimization, and reliable end-to-end verification are treated as first-class design priorities.

2. Context and Motivation

The Core Problem: Perception Is an Afterthought in Foundation Models, Not a Foundation

The fundamental gap this paper addresses can be stated simply: current foundation models treat visual perception as an accessory to language reasoning, not as a co-equal component of intelligence. The standard architecture for multimodal systems — a vision encoder bolted onto a pretrained language model via an adapter layer, with vision tokens fed as a prefix to text tokens — has enabled remarkable progress on benchmarks like VQA and captioning, but it embeds a subtle design assumption: that seeing is a preprocessing step that happens before thinking, rather than an ongoing process that is interleaved with thinking, planning, and acting.

This architectural choice has downstream consequences that become acute when models are deployed as agents in real environments. An agent navigating a website, for example, does not simply "look at the page once" and then reason from a fixed description. It must decide where to look next, what details matter, how to interpret partially occluded or ambiguous visual elements, and when to take a screenshot versus read the DOM. In current systems, these decisions are often handled by code outside the model — a harness or framework that manually captures screenshots, extracts structured representations, or reformats visual context between model calls. The model itself has no native capacity for visual attention, active perception, or perception-grounded deliberation. This creates a perception-execution gap: the model can reason, and it can see, but it cannot reason about what to see and when.

The paper frames this as the difference between a model that consumes visual inputs and one that natively integrates perception into its agentic loop. The former can answer questions about images; the latter can use vision as a tool for understanding, planning, and acting in environments where the state is constantly changing and partial observability is the norm. This is not merely a matter of adding more visual training data or a bigger vision encoder — it requires rethinking how perception relates to reasoning at the architectural, training, and evaluation levels.

Why This Matters: The Agentic Deployment Regime Changes the Requirements

The practical importance of this gap has grown rapidly as foundation models move from chat-oriented interfaces to agentic deployment in real digital environments. Three trends converge to make multimodal perception a bottleneck:

1. Agentic tasks are inherently visual. Tasks like GUI automation (navigating operating systems, filling forms, using desktop applications), web interaction (e-commerce, research, content creation), and software engineering (debugging UIs, reproducing designs from screenshots) all require the model to perceive, interpret, and act on visual interfaces. A text-only model operating through accessibility APIs or DOM parsing misses layout information, visual affordances, and graphical elements that are essential for correct action. As the paper notes in Section 3.1, many real-world tasks require the model to "first interpret the visual environment, decide what to do next, and then continue adapting its behavior based on the outcome of its actions." This is a fundamentally different capability from answering a question about a single image.

2. Long-horizon tasks amplify perception errors. In a single-turn VQA setting, a perception error (misreading a number, missing a visual detail) produces one wrong answer. In a multi-step agentic task, that error propagates: a misidentified button leads to a wrong action, which changes the environment state, which the model then misperceives, compounding the error across steps. The paper's "Lens 1" in Section 4 makes this explicit: "Many failures that appear high-level... begin with the model not seeing the environment accurately enough." The practical implication is that improving perception quality has an outsized effect on agentic task success because it breaks the compounding error chain.

3. The tool-use paradigm shifts what "seeing" means. When a model can call tools — search the web, take screenshots, crop images, run code — perception becomes active rather than passive. The model must decide not only what it sees in a given image, but which images to create, which regions to examine, and how to sequence visual operations to achieve a goal. This requires a tight coupling between reasoning (what should I do next?) and perception (what do I see now, and what do I need to see?), which is difficult to achieve when the vision system and the reasoning system are separate modules connected by a fixed interface.

The paper's introduction frames this in terms of "native foundation models for multimodal agents" — models where perception is not an auxiliary interface but a "core component of reasoning, planning, tool use, and execution." This is the unifying objective: close the perception-execution gap by building models that perceive as part of their agentic loop, not as input to it.

Where Prior Approaches Fall Short

The paper identifies several limitations in existing approaches, though it does so more through its design choices and results than through an explicit "prior work is insufficient" section. Reading between the lines, the criticisms are:

Vision encoders are optimized for static benchmarks, not agentic perception. Most vision encoders used in VLMs (CLIP, SigLIP, EVA-CLIP) are trained on image-text alignment tasks (contrastive learning) or fixed-resolution classification. They produce semantic representations that are good for answering "what is in this image?" but weaker for fine-grained spatial reasoning ("where exactly is element X relative to element Y?"), geometric understanding ("what is the 3D structure of this scene?"), and active perception ("what should I look at next?"). The paper's development of CogViT (Section 2.1) — with its two-stage training combining distillation-based masked image modeling (using dual teachers: SigLIP2 for semantics and DINOv3 for texture) followed by contrastive image-text pretraining — implicitly argues that existing encoders underinvest in the kind of dense, spatially precise representations needed for agentic tasks. The explicit comparison in Figure 1 showing CogViT outperforming other encoders on general and fine-grained tasks supports this.

Multimodal training treats images as static context, not as interactive environment. The standard paradigm for multimodal fine-tuning — supervised SFT on image-question-answer triples, followed by RLHF on preference pairs — teaches models to respond to images but not to interact with them. The paper's joint RL over 30+ task categories (Section 2.3) departs from this by training the model simultaneously on perception tasks (grounding, pointing, OCR), reasoning tasks (STEM, chart understanding), and agentic tasks (GUI agents, coding agents, tool use). This breadth is deliberate: the authors observe that "in domains with narrower distributions where single-task RL is often prone to oscillation, collaborative training can make optimization more stable." The implication is that prior approaches, by focusing on narrow task families, failed to build the kind of robust, transferable multimodal capabilities needed for open-ended agentic deployment.

The vision-to-text interface is a bottleneck for long sequences. In standard VLM architectures, visual tokens are projected into the language model's embedding space and prepended to the text sequence. For a single image, this might add 500-2000 tokens. For a video or a sequence of screenshots (as in GUI agent trajectories), this explodes: hundreds of thousands of visual tokens can consume the entire context window, forcing the system to drop earlier visual observations. The paper identifies this in Section 6 as a core challenge: "images and especially videos consume context budget much more aggressively, making them expensive to retain over long trajectories." The MMTP design (Section 2.2) doesn't directly solve this, but its use of a shared <|image|> placeholder token — rather than propagating full visual embeddings — is motivated partly by practical concerns about "substantially reducing communication complexity while improving system scalability." The deeper issue, which the paper flags as unresolved, is that "most current memory mechanisms remain fundamentally text-centric: they are better at compressing what was said than what was seen."

Agent capabilities are studied in isolation, not as a hierarchy. Prior work on agents tends to focus on specific task types: GUI grounding (identifying clickable elements), web navigation (following instructions across pages), or code generation from screenshots. These are typically treated as separate capabilities, optimized separately. The paper's "Lens 2" (Section 4) argues that this is inefficient: "agent capability is developed more effectively when optimization is distributed across multiple levels of the capability hierarchy, rather than concentrated primarily on high-level long-horizon tasks." The hierarchical optimization strategy they describe — spanning element perception, GUI grounding, single-step action prediction, and trajectory-level action prediction — is a direct response to the observation that training only on end-to-end tasks is both data-inefficient and unstable. The paper positions this as a practical insight from their development process, but it is implicitly a critique of prior work that attempted to learn agentic behavior directly from demonstrations without building up the lower-level perceptual and action primitives first.

Verifier design and task specification are afterthoughts. The paper's "Lens 3" (Section 4) argues that for end-to-end agent tasks, "the real challenge is often not extending tasks to longer horizons, but making end-to-end tasks stable enough to serve as meaningful targets for evaluation and optimization." This is a critique of the broader agent evaluation landscape: many benchmarks have underspecified goals, ambiguous success criteria, or evaluation procedures that are not reproducible. The paper's Vision2Web benchmark (Section 4) is positioned as a response — using workflow-based verification where "execution is assessed through a controlled sequence of dependent steps rather than a single final state." This enables more reliable attribution of failures and more stable optimization signals. The broader point is that progress on agentic capabilities has been held back not just by model limitations, but by inadequate task construction and verification methodology.

How This Paper Positions Itself

The paper positions its contribution not as a single architectural innovation or training trick, but as a coordinated set of advances across model design, training methodology, and infrastructure that collectively enable more "native" multimodal agentic capability. This is an important rhetorical and intellectual choice: the paper is arguing that the gap between current models and capable multimodal agents cannot be closed by any single improvement — better vision encoders, or more RL, or better benchmarks — but requires simultaneous progress on multiple fronts.

The key positioning claims are:

Multimodal perception is not separate from reasoning — it is foundational to it. The paper repeatedly returns to the theme that perception errors are the root cause of many higher-level failures. This is not a claim that perception is sufficient (the model also needs strong reasoning, planning, and code generation), but that it is necessary in a way that prior work has underappreciated. The CogViT encoder design, the broad multimodal pretraining, and the inclusion of perception tasks in the RL mix all follow from this premise.

Hierarchical optimization is more efficient than end-to-end training for agentic capabilities. Rather than training primarily on long-horizon agent trajectories (which are expensive to collect, hard to verify, and prone to instability), the paper advocates for building up capabilities from lower-level perceptual and action primitives to higher-level planning and execution. This is a practical engineering insight, but it has theoretical implications: it suggests that agentic competence has a compositional structure that can be learned more efficiently when the learning signal is distributed across the hierarchy rather than concentrated at the top level.

Test-time compute and verification design are first-class concerns, not implementation details. The paper's extensive discussion of infrastructure for multimodal RL at scale (Section 2.4), toolchain expansion (Section 3.1), and the Vision2Web benchmark design all reflect a view that model capability is not separable from the systems around it. The paper is explicit in Section 6: "The effective capability boundary is no longer determined by the model alone, but jointly shaped by the model and the harness around it." This positions the work at the intersection of model development and systems engineering, arguing that the two must co-evolve.

Generalization across modalities is achievable without sacrificing text-only performance. A persistent concern in multimodal model development is that adding visual capabilities degrades text-only performance — a form of catastrophic interference. The paper explicitly claims that GLM-5V-Turbo "preserves the coding capability of its language-only base model GLM-5-Turbo and even surpasses it" on certain benchmarks (Section 5). This is not just a results claim but a positioning claim: native multimodal capability does not require trading off text-based reasoning. The MMTP design (which uses a shared placeholder token rather than full visual embeddings, reducing interference between modalities) and the broad joint RL training (which the authors claim "tends to show weaker interference across domains" compared to SFT) are both motivated by this goal.

Benchmarks must evaluate the full perception-planning-execution loop, not isolated capabilities. The introduction of ImageMining (Section 3.3) — a benchmark that requires models to "actively mine visual inputs through agentic behaviors" including "localized cropping or magnification of minute details to refine search queries" — positions the paper as pushing evaluation beyond standard VQA toward tasks that test the integration of perception, search, and reasoning. This is not just another benchmark; it is an argument about what "multimodal capability" should mean in the agent era.

In the broader landscape, this paper sits at the convergence of several trends: the shift from language models to multimodal models (spurred by GPT-4V, Gemini, Claude 3.5, and others), the rise of agentic frameworks (Claude Code, OpenClaw, AutoClaw), and the growing recognition that real-world deployment requires models that can interact with visual environments autonomously. The paper's contribution is less about advancing the state of the art on any single dimension and more about demonstrating that a carefully coordinated design — spanning model architecture, training objectives, RL task breadth, infrastructure, toolchain, and evaluation — can produce a model that is genuinely more capable as a multimodal agent without the typical tradeoffs. This holistic approach is, in itself, the paper's primary intellectual contribution.

3. Technical Approach

3.1 Reader Orientation

GLM-5V-Turbo is a multimodal foundation model designed to serve as the cognitive core of an autonomous agent — it takes in heterogeneous inputs (images, videos, webpages, documents, text) and produces actions, code, and structured outputs while natively reasoning about what it sees. The problem it solves is the perception-execution gap in current multimodal systems: rather than treating vision as a preprocessing step that happens before reasoning, GLM-5V-Turbo integrates perception directly into its reasoning, planning, and tool-use loops so that the model can actively decide what to look at, how to interpret it, and what to do next based on what it sees.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major architectural and training components:

CogViT Vision Encoder — a parameter-efficient vision transformer trained in two stages (masked image modeling with dual teacher distillation, then contrastive image-text alignment) to produce dense visual representations optimized for fine-grained, spatial, and geometric understanding rather than just semantic categorization.
Multimodal Multi-Token Prediction (MMTP) Module — a training objective and architectural extension that appends lightweight prediction heads to the transformer layers, where visual information is represented by a shared learnable <|image|> placeholder token rather than full visual embeddings, enabling the model to predict future tokens from multimodal prefixes while maintaining training stability and system scalability.
Large Language Model Backbone (GLM-5-Turbo) — the pretrained text model that serves as the reasoning engine, extended to accept interleaved visual and text token sequences through a projector (MLP adapter) that maps CogViT outputs into the LLM's embedding space.
Joint Multi-Task Reinforcement Learning Pipeline — a training framework that simultaneously optimizes the model over more than 30 task categories spanning perception (grounding, pointing, OCR, video understanding), reasoning (STEM, math, logic), and agentic execution (GUI agents, coding agents, tool use), using a unified RL Gym and independent reward system.
Toolchain and Agent Framework Integration — a set of multimodal tools (search, browser, image processing, creation, deep research) that the model can invoke, plus integration layers for external agent frameworks (Claude Code, AutoClaw, OpenClaw) that enable the model to operate as a "vision-language controller" in real digital environments.

Information flows as follows: raw visual input (image, video, screenshot) enters CogViT → visual tokens are projected through the MLP adapter → they are interleaved with text tokens and fed to the transformer backbone → the MMTP heads provide auxiliary training signals at intermediate layers → the model generates text, tool calls, or structured actions → during RL training, outputs are evaluated by a reward system that orchestrates multiple verifiers → the model updates to improve across all task categories simultaneously.

3.3 Roadmap for the Deep Dive

First, the CogViT vision encoder (Section 2.1): how it is trained, why the two-stage recipe, what makes it different from standard CLIP-style encoders, and how it connects to the LLM backbone. This is the perceptual foundation — everything else depends on the quality of visual representations.
Second, the Multimodal Multi-Token Prediction (MMTP) design (Section 2.2): the three alternatives considered, why the <|image|> placeholder was chosen, and how it balances multimodal modeling capability with training stability and infrastructure efficiency. This is the architectural innovation that enables efficient multimodal training at scale.
Third, the broad joint training strategy (Section 2.3): how vision and language are integrated starting from pretraining, what data mixtures are used, and — critically — how the multi-task RL optimization over 30+ categories works, including the observed properties (weaker cross-domain interference, transfer of thinking patterns, capacity concentration effects). This is where the model acquires its agentic capabilities.
Fourth, the multimodal RL infrastructure (Section 2.4): the four systematic redesigns needed to make large-scale multimodal RL practical — unified task/reward abstraction, asynchronous pipeline parallelism, fine-grained memory management, and topology-aware load balancing. This is the engineering foundation that makes the training strategy possible.
Fifth, the toolchain and agent framework integration (Sections 3.1–3.5): how the model's capabilities are extended through multimodal tools and how it connects to external agent frameworks. This is the deployment layer that transforms the trained model into a functional agent.
Sixth, the ImageMining benchmark and Vision2Web design philosophy (Sections 3.3 and 4): how the paper constructs evaluation tasks that test the integration of perception, search, and reasoning, and the broader principles for building verifiable agent benchmarks. This is the evaluation methodology that validates the approach.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and engineering paper whose core idea is that building a capable multimodal agent requires simultaneous, coordinated advances across model architecture, training methodology, infrastructure, and evaluation — that no single improvement suffices, and that the integration of these components is itself the primary intellectual contribution. The paper presents a set of design choices, each motivated by specific limitations of prior approaches, and validates them through a combination of benchmark results and qualitative analysis of development insights.

CogViT Vision Encoder: Two-Stage Training for Dense Visual Representations

The CogViT encoder is the perceptual backbone of GLM-5V-Turbo. Its design reflects a specific hypothesis: that vision encoders trained primarily for semantic image-text alignment (as in CLIP or SigLIP) produce representations that are good for answering "what is in this image?" but insufficient for the fine-grained spatial reasoning, geometric understanding, and dense localization required by agentic tasks. The encoder must support not just object recognition but also grounding (where exactly is element X?), pointing (what are the pixel coordinates of region Y?), spatial reasoning (what is the relative position of A and B?), and 3D understanding (what is the structure and depth of this scene?).

Stage 1: Distillation-based Masked Image Modeling. The first training stage uses a teacher-student distillation setup to build strong visual representations without any text. The architecture is a Vision Transformer (ViT) that receives images at 224 × 224 resolution with a 35% masking ratio. Instead of training the student to reconstruct raw pixels — which tends to produce representations that capture low-level texture but miss semantic structure — the student is trained to reconstruct the feature representations of two frozen teacher models in the masked regions. The two teachers are:

SigLIP2: provides semantic representations that capture what objects and concepts are present in the image. SigLIP2 is a multilingual vision-language encoder trained with a sigmoid-based contrastive loss, chosen because it produces semantically rich features that generalize across languages (relevant for the bilingual Chinese-English training later).
DINOv3: provides texture and structural features that capture fine-grained visual patterns, object boundaries, and spatial relationships. DINOv3 is a self-supervised vision model trained with a student-teacher distillation approach that excels at producing dense, locally structured representations useful for tasks like segmentation and depth estimation.

The dual-teacher design is deliberate: SigLIP2 alone would miss the fine-grained spatial structure needed for grounding; DINOv3 alone would miss the semantic categorization needed for recognition. By distilling from both simultaneously, the student ViT learns representations that are both semantically meaningful and spatially precise. This is a form of representation fusion — the two teachers capture complementary aspects of visual information, and the student learns to combine them into a single representation.

The training data follows a "quality-aware mixture strategy": 80% high-quality natural images, 10% instruction-following data (images paired with task descriptions, likely to prepare the encoder for downstream agentic tasks), and 10% scientific imagery (charts, diagrams, microscopy images — relevant for document understanding and STEM reasoning). This mixture ensures the encoder sees both natural scenes and the kinds of structured visual content common in agentic workflows (webpages, documents, UI screenshots).

The optimizer is Muon (an optimizer designed specifically for hidden layers in neural networks, proposed by Jordan et al., 2024) with a cosine decay schedule. Muon is chosen over AdamW likely because it provides better training stability and convergence properties for large vision transformers, though the paper does not provide an explicit ablation comparing optimizers for this stage.

A critical architectural detail is the introduction of QK-Norm (Query-Key Normalization). Before computing attention scores, the query and key vectors are normalized:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{\text{Norm}(Q) \cdot \text{Norm}(K)^T}{\sqrt{d_k}}\right) \cdot V$

where $Q$ , $K$ , and $V$ are the query, key, and value matrices from the self-attention mechanism, $\text{Norm}(\cdot)$ applies layer normalization to each vector independently, and $\sqrt{d_k}$ is the standard scaling factor with $d_k$ being the dimension of the key vectors.

What this computes: rather than using raw query and key vectors for computing attention scores, each vector is first normalized to unit norm (or near-unit norm, depending on the normalization type). This constrains the magnitude of the dot products $Q \cdot K^T$ , preventing individual attention scores from growing unboundedly large.

Why this form: in large vision transformers, the logits (pre-softmax attention scores) can "explode" during training — individual query-key dot products become extremely large in magnitude, leading to near-one-hot attention distributions that are unstable to optimize. QK-Norm prevents this by bounding the norm of each query and key vector, which bounds the maximum possible dot product. The paper explicitly states this "effectively mitigates logit explosion and ensures stability at scale." Without QK-Norm, training large ViTs often requires careful learning rate tuning, gradient clipping, or specialized initialization schemes; with QK-Norm, these become less sensitive. This is a known technique from the NLP literature (Henry et al., 2020) applied here to vision.

Stage 2: Contrastive Image-Text Pretraining. The second stage shifts from pure visual representation learning to cross-modal alignment. The goal is to map visual features and text features into a shared embedding space so that the LLM backbone can process them jointly. This stage introduces three key upgrades over Stage 1:

Upgrade 1: Variable-resolution processing with NaFlex. Instead of the fixed 224 × 224 resolution used in Stage 1, the encoder now uses the NaFlex scheme (introduced in SigLIP2, Tschannen et al., 2025) to process images at their native aspect ratios and variable resolutions. Concretely, NaFlex divides the image into a grid of patches (as in standard ViT), but the number of patches along each dimension adapts to the image's actual dimensions rather than forcing a square crop. An image that is 800 pixels wide and 400 pixels tall would produce (800/P) × (400/P) patches (where P is the patch size) rather than being distorted into a (224/P) × (224/P) square.

Why this matters for agentic tasks: many visual inputs in agent workflows have extreme aspect ratios — a screenshot of a full webpage might be very tall and narrow, a spreadsheet might be very wide, a document might be portrait-oriented. Forcing these into a square crop either discards information (if cropping) or introduces distortion (if resizing). NaFlex preserves the original spatial relationships, which is critical for tasks like GUI grounding where the model needs to output pixel coordinates that correspond to the original image dimensions.

Upgrade 2: Large-scale SigLIP training. The global batch size is scaled to 64K (64,000 image-text pairs per optimization step), using the sigmoid-based SigLIP loss. The SigLIP loss is an alternative to the standard softmax-based contrastive loss used in CLIP. The standard CLIP loss computes:

$\mathcal{L}_{\text{CLIP}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(\text{sim}(I_i, T_i) / \tau)}{\sum_{j=1}^N \exp(\text{sim}(I_i, T_j) / \tau)}$

where $N$ is the batch size, $\text{sim}(I_i, T_j)$ is the cosine similarity between image $i$ and text $j$ , and $\tau$ is a learned temperature parameter. This loss compares each image to all texts in the batch (and vice versa), treating correct pairs as positives and all other pairs as negatives.

The SigLIP loss replaces the softmax with independent sigmoid activations:

$\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N} \sum_{i=1}^N \sum_{j=1}^N \left[ \mathbb{1}_{i=j} \log \sigma(\text{sim}(I_i, T_j) / \tau + b) + \mathbb{1}_{i \neq j} \log (1 - \sigma(\text{sim}(I_i, T_j) / \tau + b)) \right]$

where $\sigma(\cdot)$ is the sigmoid function and $b$ is a learned bias term. Each image-text pair is classified independently as a match or non-match, rather than competing against all other pairs in the batch.

What this computes: for every pair of image $I_i$ and text $T_j$ in the batch, the model predicts whether they correspond to each other (positive pair, $i = j$ ) or not (negative pair, $i \neq j$ ). The loss sums binary cross-entropy terms over all $N^2$ possible pairs.

Why this form: the SigLIP loss is more amenable to very large batch sizes because the gradient signal from each pair is independent — there is no normalization over the entire batch as in softmax. This makes distributed training more efficient: you can shard the batch across devices and compute per-pair losses locally without needing to gather all similarities for normalization. The paper mentions a "bidirectional distributed implementation for efficiency" which likely refers to computing image-to-text and text-to-image similarities in parallel across devices. At 64K batch size, the standard CLIP loss would require gathering an N × N similarity matrix across all devices (64K × 64K = 4 billion entries), which is prohibitively expensive in communication; SigLIP avoids this bottleneck.

Upgrade 3: Bilingual training corpus. The contrastive training uses an "8-billion bilingual (Chinese-English) image-text corpus." This is substantially larger than typical contrastive pretraining datasets and explicitly includes both languages to support cross-lingual visual understanding. The motivation is practical: the model will be deployed in both Chinese and English environments, and visual concepts should be equally accessible regardless of the language used to query them.

Optimization details. Stage 2 continues to use the Muon optimizer but with "module-specific learning rates and decay schedules to the vision, text, and projection components." This is a form of differential learning rate scheduling: the vision encoder (already well-trained from Stage 1) likely uses a lower learning rate to avoid catastrophic forgetting, while the text encoder and projection layers (newly introduced) use higher learning rates to learn the alignment. The specific learning rates are not provided in the paper.

Connection to the LLM backbone. After Stage 2, the CogViT encoder produces visual features that are mapped to the LLM's embedding space through an MLP adapter (a small multi-layer perceptron that projects from the vision encoder's output dimension to the LLM's hidden dimension). This is the standard "vision encoder → projector → LLM" architecture seen in LLaVA, MiniGPT-4, and most open-source VLMs. The novel contribution is not the adapter architecture itself but the quality of the representations feeding into it — by training CogViT with the two-stage recipe (distillation for dense features, then contrastive alignment), the adapter receives representations that are already rich in both semantic and spatial information, reducing the burden on the LLM to extract this information from weaker visual features.

Performance validation (Figure 1). The paper presents a performance comparison of CogViT against other state-of-the-art vision encoders across "general and fine-grained multimodal tasks." While the specific numbers are not detailed in the text, the figure shows CogViT achieving competitive or superior performance, validating the two-stage training recipe. The comparison is important because it establishes that the perceptual foundation is solid before any agentic capabilities are built on top.

Multimodal Multi-Token Prediction (MMTP): Efficient Multimodal Training with Placeholder Tokens

Multi-token prediction (MTP) is a training technique originally proposed for text-only language models (Gloeckle et al., 2024) where the model predicts not just the next token but the next k tokens simultaneously, using lightweight prediction heads attached at intermediate transformer layers. The intuition is that predicting multiple future tokens forces the model to learn more structured, longer-range representations because it cannot succeed by memorizing local token co-occurrence patterns — it must anticipate content that is several tokens ahead.

GLM-5V-Turbo extends this to the multimodal setting, which introduces a fundamental design question: when the input contains both image and text tokens, what information should be passed to the MTP prediction heads? The standard text-only approach is straightforward — prefix tokens are embedded through the word embedding layer and passed to the MTP head. But image tokens are not discrete symbols with embedding table entries; they are continuous vectors produced by the vision encoder and projector. The paper systematically compares three alternatives (illustrated in Figure 2):

Option 1: Direct Vision Embeddings. Pass the full visual embeddings (the output of the CogViT encoder plus the MLP adapter) directly to the MTP heads, exactly as they are passed to the main transformer layers. The MTP head receives the same multimodal sequence as the backbone.

Option 2: Masked Vision Tokens. Mask out (remove) all visual tokens at the MTP head input, reducing the MTP head to a text-only predictor. The model still sees visual information in the backbone (which conditions the representations at the MTP head layers), but the MTP head itself only receives text tokens.

Option 3: <|image|> Placeholder (Adopted). Replace all visual tokens with a shared learnable <|image|> special token at the MTP head input. The spatial structure of the image is discarded (one placeholder token replaces potentially hundreds of visual tokens), but the presence of visual information is signaled through the learnable embedding of the placeholder token.

The choice: Option 3 is adopted. The decision is based on two criteria: optimization behavior and system efficiency.

Optimization behavior. The paper reports an ablation study on a 0.5B parameter model showing that Option 3 achieves "lower training loss and more stable convergence than directly using visual embeddings" (Option 1). The proposed explanation is that "the MTP head is typically lightweight, and may not have sufficient modeling capacity to effectively absorb visual representations whose distribution differs substantially from that of text embeddings." In other words, the MTP heads are small neural networks (likely a few linear layers) designed to predict text tokens from text-token representations. Visual embeddings have very different statistical properties — different mean, variance, and correlation structure — and a lightweight head may struggle to process them effectively. The <|image|> placeholder, by presenting a uniform token type with a learned embedding in the same space as text embeddings, "alleviates this optimization difficulty."

Option 2 (masked vision tokens) would also avoid the distribution mismatch problem, but the paper notes it is less effective because the MTP head loses the signal that visual information exists in the prefix. The placeholder token preserves this signal (the model knows "there was an image here") without burdening the lightweight head with the full complexity of visual representations.

System efficiency. The efficiency argument is about communication overhead in distributed training. The LLM backbone is trained with pipeline parallelism, where different layers of the transformer are placed on different devices (GPUs). If the MTP head receives full visual embeddings, those embeddings must be propagated across pipeline stages — every device that hosts an MTP head needs access to the visual embeddings from earlier pipeline stages. This "substantially increases communication complexity while reducing system scalability and engineering maintainability." The <|image|> placeholder avoids this because it is a single token that can be looked up from the embedding table on each device, requiring no cross-device communication for visual features.

Additionally, the placeholder design "remains naturally compatible with existing partitioning strategies such as sequence parallelism and context parallelism, without requiring additional handling for visual-embedding partitioning, alignment, or offset mapping." Standard parallelism strategies assume text tokens with consistent embedding sizes; visual embeddings may have different dimensions or require different partitioning logic. The placeholder token hides this complexity behind a simple text-like interface.

Architecture (Figure 2). The full MMTP architecture works as follows:

Visual inputs pass through CogViT and the MLP adapter, producing a sequence of visual embeddings $v_1^1, v_1^2, ..., v_n^m$ (the superscript indices in the figure denote different images, the subscripts denote token positions within each image).
Text inputs are converted to token IDs and embedded through the standard word embedding layer, producing text embeddings $t_1, t_2, ..., t_k$ .
The concatenated sequence $[v_1^1, v_1^2, t_1, t_2, v_2^1, v_2^2, t_3, t_4, t_5]$ (interleaved visual and text tokens) is fed to the transformer backbone.
At each of the top L transformer layers (where L is the number of MTP modules), the hidden state at that layer is passed to a corresponding MTP head. However, before passing to the MTP head, all visual token positions are replaced with the <|image|> placeholder embedding.
Each MTP head predicts the next token (or next k tokens, depending on the configuration) given the modified prefix. The MTP heads share parameters across layers (indicated by "Shared Parameters" in Figure 2), meaning the same prediction module is reused at each depth.

What this computes operationally: the MTP heads take the hidden representations at intermediate layers, replace visual token positions with a shared placeholder, and predict future text tokens. This provides an auxiliary training signal: the model is penalized not only for the final-layer predictions (standard language modeling loss) but also for intermediate-layer predictions (MTP loss). This encourages each layer to build representations that are useful for prediction, rather than relying on later layers to refine ambiguous representations.

Why this form over alternatives: Option 1 would preserve all visual information but introduces two problems: (a) optimization difficulty because lightweight MTP heads cannot effectively process visual embeddings with different statistical properties than text, and (b) communication overhead from propagating visual embeddings across pipeline stages. Option 2 would avoid both problems but loses the signal that an image is present, which could be important for certain predictions (e.g., predicting that the next token should be a description of the image). Option 3 balances these concerns: it preserves the presence signal through the learnable placeholder while avoiding both the optimization difficulty and the communication overhead. The shared parameter design further reduces memory usage and encourages the MTP head to learn a general "predict from prefix" function that works across different layer depths.

Training loss curves (Figure 2, bottom-left). The paper shows training loss curves comparing Option 1 and Option 3, with Option 3 achieving lower loss. This is presented as empirical validation of the optimization stability hypothesis, though the figure only shows one run and the paper does not report confidence intervals or multiple seeds.

Broad Joint Training: Integrating Vision and Language from Pretraining through RL

The training strategy for GLM-5V-Turbo is built on the premise that multimodal agentic capability cannot be achieved by training on narrow task families in isolation — it requires exposure to a wide distribution of tasks that jointly develop perception, reasoning, planning, and execution. This section describes the two-phase training process: pretraining (which integrates vision and language data) and post-training (which uses joint reinforcement learning over 30+ task categories).

Pretraining Phase: Multimodal Data Integration from the Start

The pretraining phase uses "a mixture of plain text and multimodal data to foster a balanced development of diverse capabilities." This is an important design choice: rather than pretraining a text-only model and then "bolting on" vision capabilities through fine-tuning (the approach used in LLaVA and many other VLMs), GLM-5V-Turbo interleaves text and multimodal data throughout pretraining. The goal is for the model to develop native multimodal representations — representations that encode visual and textual information in a shared, integrated way — rather than representations where visual information is mapped into a text-centric space as an afterthought.

The multimodal data categories include "world knowledge, interleaved image-text, OCR, coding, GUI, video, multimodal tool-use, spatial perception, grounding, and academic problem-solving." This is a deliberately broad set designed to cover the full spectrum of tasks the model will encounter as an agent:

World knowledge and interleaved image-text: standard web documents with images and captions, building general visual-semantic knowledge.
OCR: documents, screenshots, and images containing text that the model must read and understand.
Coding: source code paired with visual context (e.g., UI screenshots with corresponding HTML/CSS, diagrams with corresponding LaTeX or SVG).
GUI: screenshots of user interfaces paired with descriptions of their structure and functionality.
Video: temporal sequences of frames requiring the model to track objects and events over time.
Multimodal tool-use: examples of the model calling tools (search, crop, annotate) in response to visual inputs.
Spatial perception and grounding: images with bounding box annotations, point coordinates, or depth information.
Academic problem-solving: science and math problems presented with diagrams, charts, or figures.

The paper places "particular emphasis on multimodal coding data to better align visual understanding with code generation and to improve the model's performance in multimodal agentic tasks." This is strategically important because many agentic tasks involve generating code from visual specifications: reproducing a website from a screenshot (Design2Code), writing SVG from a diagram, or generating plotting code from a chart description. By overrepresenting these tasks in pretraining, the model learns to connect visual structure to code structure — a capability that transfers to GUI automation, web development, and document generation.

Post-Training Phase: Joint Reinforcement Learning over 30+ Task Categories

The core innovation in the training strategy is the post-training phase, which uses "joint RL optimization over more than 30 task categories." This is not the standard RLHF pipeline (train a reward model on preference data, then optimize against it) but rather a multi-task RL setup where each task provides its own reward signal through rule-based verifiers or model-based judges. The key properties of this setup are:

1. Task breadth prevents over-specialization. The paper explicitly contrasts this with narrow RL: "compared with the cross-domain trade-offs often seen in SFT, RL tends to show weaker interference across domains, allowing multiple domains to improve together with stable gains." This is a counterintuitive claim — one might expect that training on 30+ diverse tasks would create interference, where improvements on one task degrade performance on another. The paper argues the opposite: multi-task RL reduces interference compared to single-task fine-tuning, because the model is forced to learn representations and strategies that generalize across tasks rather than overfitting to the idiosyncrasies of any single task.

2. Collaborative training stabilizes narrow-domain optimization. The paper observes that "in domains with narrower distributions where single-task RL is often prone to oscillation, collaborative training can make optimization more stable by exposing the model to a richer distribution of strategies and steering it toward more robust solutions." When a model is trained only on a narrow task (e.g., a specific GUI environment), it may discover a strategy that works well on the training distribution but is brittle — changing a single environment parameter causes it to fail. By training simultaneously on many tasks, the model is exposed to diverse strategies and converges to solutions that are more robust because they work across varied conditions.

3. Thinking patterns transfer across domains. The paper reports observing "some transfer of thinking patterns across tasks: reasoning behaviors acquired in one domain can sometimes carry over to another and produce measurable benefits there as well." For example, the step-by-step debugging behavior learned in code generation tasks might transfer to GUI tasks where the model needs to identify why a particular action failed. This suggests that the value of multi-task RL is "not only in covering a broader range of tasks, but also in inducing deeper sharing at the level of strategy patterns."

Specific RL improvements reported (Section 2.3). The paper provides quantitative gains from the RL stage compared to the SFT baseline. These are reported as improvements on specific benchmarks:

Perception improvements:

2D image grounding: +4.8% on RefCOCO-avg (a referring expression comprehension benchmark where the model must locate objects described in natural language).
Pointing: +3.2% on PointBench (a benchmark for language-guided pointing, where the model must output pixel coordinates).
Video understanding: +5.6% on MVBench (a comprehensive video understanding benchmark).
3D grounding: +7.7% on SUNRGBD (an RGB-D scene understanding benchmark requiring 3D bounding box prediction).
OCR: +4.2% on OCRBench (a benchmark for optical character recognition in multimodal models).
Chart understanding: +7.7% on CharXiv (a benchmark for chart comprehension).

Reasoning improvements:

STEM: +1.8% averaged over MMMU_Val, MMMU_Pro, MathVista, and LogicVista (benchmarks for multimodal understanding, mathematical reasoning, and logical reasoning in visual contexts). The paper notes this improvement is modest but shows "greater stability in problem solving" — the model makes fewer catastrophic errors even when the accuracy gain is small.

Agentic improvements:

GUI agents: +4.9% on OSWorld (a benchmark for open-ended tasks in real computer environments).
Coding agents: +0.2% on CC-Backend (a benchmark for backend coding tasks in the Claude Code framework). The small improvement here may reflect the fact that coding capability was already strong from pretraining.
General tool use: +3.5% on MMSearch (a multimodal search benchmark), which "demonstrates improved planning and execution."

A critical caveat: capabilities not covered during RL can degrade. The paper acknowledges a limitation of this approach: "capabilities left uncovered during RL can sometimes decline after post-training, especially those more orthogonal to the trained task distribution." The proposed explanation is that "as RL proceeds, both model capacity and learned thinking patterns become increasingly concentrated around the sampled task distribution, weakening the model's ability to retain performance in under-represented domains." This is a form of catastrophic forgetting in RL — the model's representations optimize for the tasks it is being rewarded on, and capabilities irrelevant to those tasks may atrophy.

This has an important practical implication: "the scope of task coverage during RL is itself an important factor shaping the model's eventual generalization boundary." You cannot simply train on a subset of tasks and expect the model to maintain performance on others; the task distribution during RL defines the model's capability frontier. The paper suggests a mitigation strategy: "even when a target capability cannot be easily formulated directly as an RL task, semantically or structurally related proxy tasks may provide useful optimization signals." For example, "RL on single-turn UI-to-code generation can support more complex multi-turn coding ability" — training on a simpler related task can indirectly improve performance on a more complex target task through transfer of learned patterns.

A specific technical improvement: Relative Visual Policy Optimization. The paper mentions adopting "relative visual policy optimization in UI-to-code tasks" (citing Yang et al., 2025). This is a technique for UI-to-code generation where the model optimizes its output relative to a reference implementation, rather than optimizing against an absolute correctness criterion. The motivation is that UI-to-code tasks have many valid solutions — different HTML/CSS implementations can produce visually identical results — so an absolute reward signal (does the output exactly match the reference?) is too strict. Relative optimization compares the model's output to its own previous outputs and rewards improvement, which provides a more informative learning signal for tasks with non-unique solutions. The specific mechanism is not detailed in this paper, but it illustrates the kind of task-specific reward design needed for effective RL in agentic domains.

On-policy distillation. The paper mentions "on-policy distillation" as part of the multi-task collaborative RL approach, though the details are sparse. In standard RL for language models, on-policy distillation means generating outputs from the current model (the "policy"), scoring them, and using the best outputs as training targets for the next iteration — essentially distillation from the model's own top-performing samples. This is distinct from off-policy distillation, where the training targets come from a different (typically stronger) model. On-policy distillation creates a self-improvement loop: the model generates, the best generations become targets, the model trains on those targets, and the cycle repeats. The paper frames this as "not merely a tool for improving individual capabilities, but a central path toward shaping a more unified multimodal capability structure over a broader agentic distribution."

Multimodal RL at Scale: Infrastructure for Large-Scale Agent Training

The training strategy described above would be infeasible without substantial infrastructure work to handle the unique demands of large-scale multimodal RL. The paper identifies four systematic redesigns (Section 2.4):

1. Unified Task and Reward Abstraction.

The VLM RL Gym provides a consistent environment interface that handles both single-step tasks (e.g., answer a question about an image) and multi-step tasks (e.g., navigate a website over multiple actions) within the same training framework. This is non-trivial because these task types have fundamentally different structures: single-step tasks produce one observation and one reward, while multi-step tasks produce sequences of observations, actions, and rewards. The unified interface abstracts over this difference, allowing the training loop to treat all tasks uniformly.

In parallel, an independent reward system centrally orchestrates multiple verifiers:

Rule-based verifiers execute locally and synchronously — these are deterministic programs that check correctness (e.g., string matching for QA, unit test execution for code).
Model-based judges are invoked asynchronously through APIs — these are separate LLMs that evaluate the quality of model outputs on tasks where rule-based verification is impossible (e.g., is this generated website visually similar to the reference?).

The reward system combines outputs from multiple verifiers through "configurable aggregation strategies" — for example, a task might be considered successful only if both rule-based and model-based verifiers agree, or the final reward might be a weighted average of multiple scores. This decouples verifier logic from the main training code, making it easier to add new tasks or modify reward functions without touching the training loop.

For observability, each sample carries a data-source tag that tracks which task it came from. This allows per-task metrics (reward, pass@k) to be aggregated and reported separately, making it possible to monitor whether some tasks are improving while others are degrading.

2. Full-Pipeline Decoupling, Asynchrony, and Stage Overlap.

The training pipeline is restructured into four decoupled stages: rollout inference, reward evaluation, batch construction, and weight transfer. The goal is to overlap these stages so that no component sits idle waiting for another. The key mechanisms:

Completion callbacks for reward evaluation. Each inference request is registered with a callback, so reward computation begins as soon as that request finishes — rather than waiting for the entire rollout batch to complete. This is particularly important because rollout lengths vary widely (a simple VQA task might produce 10 tokens, a complex agent trajectory might produce 10,000), and waiting for the longest request would leave other workers idle.
Parallel batch construction and CPU-GPU transfer. While old-policy weights are being transferred from GPU to CPU (for reference model computation), batch construction executes in parallel. This hides the latency of weight transfer behind useful computation.
Reference model CPU residency. The reference model's parameters remain on CPU memory and are asynchronously prefetched to GPU immediately before the reference forward pass, then released right after. This allows reference computation (needed for KL penalty or PPO-style updates) to overlap with the main training step without permanently consuming GPU memory.
Two early-abort modes. To handle long-tail latency from particularly long trajectories, the system supports aborting based on either a completion count threshold (stop after N rollouts have finished) or a time threshold (stop after T seconds). Aborted prompts are cached and can be reused in subsequent training steps, so "this helps control long-tail latency without materially reducing data utilization."

3. Fine-Grained Runtime Memory Management for Multimodal Workloads.

Standard activation recomputation schemes (where intermediate activations are discarded during the forward pass and recomputed during the backward pass to save memory) are designed for text-only training. Multimodal inputs introduce memory bottlenecks that these schemes do not adequately address:

Vision encoder activations are large. The ViT forward pass produces activations for every patch of every image. With high-resolution images or videos, this can dwarf the LLM's activation memory.
The projector (MLP adapter) adds another activation layer. The visual embeddings pass through the projector, creating additional intermediate tensors.

The paper's solution is "separate memory-management strategies for the vision-side ViT and projector modules, combining targeted recomputation with CPU offloading." Specifically:

Some ViT activations are recomputed during the backward pass rather than stored (reducing memory at the cost of extra computation).
Other activations are offloaded to CPU memory and fetched back when needed (reducing GPU memory at the cost of PCIe transfers).
The combination is targeted — not all activations are treated the same way; the paper analyzes which activations are memory-intensive but cheap to recompute, and which are expensive to recompute but can tolerate CPU offloading latency.

This "prevents activation memory from scaling linearly with the number of images in the naïve way, and substantially reduces runtime memory pressure while preserving overall computational efficiency." In a naïve implementation, adding a second image would double the vision-side activation memory; with targeted recomputation and offloading, the increase is sublinear.

4. Topology-Aware Partitioning and Dynamic Load Balancing for Visual Inputs.

Visual inputs — especially videos — have highly variable sequence lengths. A video might produce anywhere from hundreds to tens of thousands of visual tokens, depending on its resolution and frame count. This creates load imbalance in distributed training: some GPUs get batches with many visual tokens, others get batches with few, and the overall throughput is limited by the slowest GPU.

The paper addresses this with several techniques:

Upstream partitioning during data loading. In conventional implementations, data partitioning (deciding which tokens go to which device) happens during the forward pass, meaning each rank must first hold the full patch tensor, then redistribute. The paper moves "CP and TP partitioning upstream into the data-loading stage," meaning the partitioning is determined before the data reaches the GPU. This eliminates the need for cross-rank patch aggregation (combining tensors from different GPUs before partitioning) and the associated memory and communication overhead.

Aligning partition boundaries with downsample groups. Vision transformers often use downsampling (e.g., patch merging) to reduce sequence length. The paper aligns the partition boundaries with these downsample groups, so that no partition splits a group across two devices. This avoids the need for cross-device communication within a downsample group.

Asynchronous all-to-all dispatch with CPU-side buffering. After load balancing across data-parallel groups, tokens are dispatched to the appropriate devices through asynchronous all-to-all communication. The paper moves "large Python objects off the GPU communication path and onto the CPU path, which reduces GPU communication buffer overhead by about 7 GB in practice." This is a concrete engineering optimization: Python objects used for communication bookkeeping are kept on the CPU rather than the GPU, freeing up GPU memory for actual computation.

Joint bin-packing over sequence length and ViT token count. During rollout, variable-length sequences are packed into micro-batches by considering both the text sequence length and the number of visual tokens. This produces micro-batches that are balanced in both dimensions — avoiding cases where one micro-batch has long text sequences and another has many visual tokens, both of which would create memory or compute bottlenecks for different reasons.

Multimodal Toolchain and Agent Framework Integration

Once the model is trained, it needs to be deployed in environments where it can act. Sections 3.1–3.5 describe the toolchain and framework integration that transforms GLM-5V-Turbo from a model that can answer questions about images into a model that can autonomously complete tasks in digital environments.

Multimodal Tool Categories (Section 3.1, Table 1). The tools are organized into categories:

General Recognition Tools: zai_recognize_plant, zai_recognize_location, zai_recognize_person — specialized vision models wrapped as tools that the LLM can invoke for fine-grained recognition tasks beyond its native capability.
Multimodal Search: zai_search_web_text, zai_search_web_by_image, zai_search_similar_images, zai_search_web_images, zai_search_scholar — tools for retrieving information from the web using either text queries or images as search keys (reverse image search).
Browser Tools: zai_load_image_from_url, zai_read_webpage — tools for fetching and parsing web content, including extracting both text and visual elements from pages.
Image Processing: zai_crop_image, zai_draw_image_bounding_boxes, zai_draw_image_point_markers, zai_draw_image_geometry, zai_draw_image_3d_bounding_boxes, zai_draw_video_objects_tracking — tools for manipulating and annotating images, including cropping regions, drawing boxes/points, and tracking objects across video frames.
Creation Tools: Web creation (submit_plan, apply_edits, zai_generate_web_html, zai_generate_web_outline) and slide creation (zai_generate_slide_html, zai_generate_outline_ppt) — tools for generating structured content (websites, presentations) from specifications.
Deep Research Tools: zai_dr_python, zai_dr_open_url_mm, zai_dr_visit_img, zai_dr_search, zai_dr_images_search, zai_dr_images_lens — tools specifically designed for the multimodal deep research workflow, including Python execution, URL opening with multimodal parsing, image visitation for detailed analysis, and multiple search modes.

The critical design principle is that these tools are not just text-based APIs — many of them take images as input or return images as output. For example, zai_search_web_by_image takes an image and finds similar images on the web; zai_crop_image takes an image and bounding box coordinates and returns the cropped region; zai_draw_image_bounding_boxes takes an image and a list of coordinates and returns the annotated image. This means the model's interaction with tools is visually grounded — it can search for something it sees, annotate what it finds, and use the annotated result in further reasoning.

Toolchain expansion purpose (Section 3.1). The paper emphasizes that this toolchain enables "a fuller perception–planning–execution loop in more realistic environments." The model can:

Perceive the environment (take a screenshot, load a webpage, search for an image).
Plan based on what it perceives (decide which region to crop, what to search for next).
Execute actions (crop the image, run code, generate a website).
Then perceive the outcome and adapt (did the generated website match the reference? should I refine it?).

This is demonstrated concretely in the website reproduction example: "when reproducing a real website, the model can first use a multimodal GUI agent to explore the site through screenshots, interaction with page elements, and navigation across pages, building a richer understanding of layout, functionality, and interaction flow. It can then rely on its native UI-to-code capability to reproduce the site more faithfully." The model is not just generating code from a single screenshot — it is actively exploring the target website, collecting information, and using that accumulated understanding to produce a higher-quality reproduction.

Integration with External Agent Frameworks (Section 3.2).

GLM-5V-Turbo is designed to serve as the "cognitive core" for external agent frameworks, specifically Claude Code (Anthropic's agentic coding framework) and AutoClaw/OpenClaw (browser-based and GUI-centric automation frameworks). The integration works as follows:

With Claude Code: GLM-5V-Turbo serves as the vision-language controller that can navigate terminal environments and local file systems while perceiving visual context. Claude Code handles the execution environment (running commands, managing files, interacting with the operating system), and GLM-5V-Turbo provides the multimodal reasoning — it can look at terminal output, screenshots of GUIs, and file contents, then decide what commands to run or what code to write.
With AutoClaw: AutoClaw provides "the 'hands' for browser-based and GUI-centric automation." It can click buttons, fill forms, navigate pages, and interact with desktop applications. GLM-5V-Turbo acts as the vision-language controller that decides what actions to take based on what it sees on screen.

This division of labor — the model handles perception and high-level reasoning, the framework handles low-level execution — is the paper's answer to the challenge that "model and harness increasingly co-shape the system's capability boundary" (Section 6, Lens 3). Rather than trying to build execution capabilities into the model itself (which would be expensive and fragile), the model specializes in what it does best (understanding visual context and deciding what to do) and delegates execution to specialized frameworks.

Official Skills (Section 3.5, Table 2).

To make it easier for users to deploy GLM-5V-Turbo within agent systems, the paper provides a set of "official skills" — pre-configured workflows that combine model capabilities, tool invocations, and framework interactions. These fall into three categories:

Native skills (5 skills): Built on the native capabilities of GLM-5V-Turbo without external model dependencies. Examples include PDF-to-Web (convert a PDF document to an interactive website), Web Replication (reproduce a target website from exploration), PRD-to-App (generate an application from a product requirements document), and Stock Analyst (gather information and produce stock analysis reports). These leverage the model's combined perception, coding, and tool-use capabilities.
External Tool skills (5 skills): Wrap GLM-5V-Turbo as an external tool (via MaaS API) that other agents can invoke. Examples include Image Captioning, Visual Grounding, Doc-based Writing, Resume Screening, and Prompt Generation. These expose specific model capabilities as callable services.
Specialized Model skills (5 skills): Use previously released specialized models (GLM-OCR for optical character recognition, GLM-Image for image generation) to handle specific modalities. Examples include General OCR, Table Recognition, Handwriting Recognition, Formula Recognition, and Image Generation. These offload specific perceptual or generative tasks to models optimized for those tasks.

A "unified master skill" (https://clawhub.ai/jaredforreal/glm-master-skill) helps users discover, install, and use all the official skills.

ImageMining: A Vision-Centric Deep Search Benchmark (Section 3.3)

The paper introduces ImageMining as a benchmark specifically designed to evaluate the integration of visual understanding and autonomous search — what the paper calls "think with image, deep search with image." Unlike standard VQA where the model answers a question about a given image, ImageMining requires the model to actively use visual inputs as starting points for multi-step search and reasoning.

Benchmark composition. ImageMining comprises 217 curated test cases derived from manually collected trace samples, spanning seven domains (Social, Entertainment, Products, Places, Rich Text, Nature, and Science) and five reasoning categories:

Universal Recognition: Fine-grained identification of flora, fauna, and artifacts — tasks where the model must identify specific species, models, or variants from visual cues.
Spatio-Temporal Reasoning: Geographic deduction grounded in visual cues — for example, "where was this photo taken based on the architecture, vegetation, and signage visible in the image?"
Event Reasoning: Comprehension of news events and product launches from visual evidence — piecing together what happened from images of the event.
Text-based Reasoning: Reasoning over embedded rich text such as academic papers, reports, or documents where key information is in figures, tables, or formatted layouts.
Visual Search: Cross-referencing visual inputs to retrieve specific artworks or imagery — for example, "find the original source of this artwork" by searching for visually similar images and tracing attribution.

The "Visual Jump" constraint. A key design element is the WEB_VISUAL constraint: during the data discovery and construction process, "intermediate reasoning hops must involve visual transitions, forcing the model to parse images rather than relying on textual shortcuts or parametric knowledge." This means that to solve an ImageMining task, the model cannot simply recognize the image content from its training data and answer from memory — it must actually perform visual search operations (reverse image search, cropping, magnification) and reason about the results. This constraint ensures the benchmark measures active visual reasoning rather than passive visual recognition.

The "Deep-Wide-Search" spectrum. The paper describes ImageMining as evaluating models on both search breadth (how widely they search across sources) and search depth (how precisely they reason about visual details). Success "correlates strongly with the precision of on-image tool usage" — models that can crop precisely to isolate relevant details, magnify to read small text, and use these processed images as search queries perform better than models that only use whole-image search or text-based queries.

OCR Search data. The benchmark includes "specialized OCR Search data for charts, maps, and posters" that requires models to "perform entity isolation and localized cropping before initiating search chains." For example, a chart might contain multiple data series, axis labels, and annotations; the model must crop to the specific element it wants to search for rather than searching the entire chart image. This "transforms images from static inputs into interactive environments for deep exploration."

Results on ImageMining (Section 5, Figure 4). GLM-5V-Turbo achieves 30.7 on ImageMining, compared to 30.0 on MMSearch-Plus (a related benchmark). The paper highlights this as representing "nearly an eightfold improvement over the previous generation" (GLM-4.6V), though the absolute scores are modest, indicating that vision-centric deep search remains a challenging capability even for state-of-the-art models.

Vision2Web: Task Specification, Verification, and Controlled Evaluation (Section 4, Lens 3)

While Vision2Web is primarily discussed in the "Design Lenses" section and the evaluation, its design methodology is a technical contribution that shapes how the paper thinks about agent evaluation. The benchmark is for "end-to-end visual website development" — the task of reproducing a target website from a specification that may include PRDs (product requirements documents), mockups, reference pages, and resource assets.

Task specification. Each task is "grounded not just in a textual instruction, but in a richer specification." This multi-source grounding makes the task better specified than a simple "reproduce this website" prompt: the model receives structured information about what the website should do (PRD), what it should look like (mockups), what existing implementations look like (reference pages), and what assets to use (images, fonts, logos).

Workflow-based verification. Rather than evaluating the generated website as a single final state, Vision2Web uses "workflow-based verification so that execution is assessed through a controlled sequence of dependent steps rather than a single final state." This means the evaluation checks not just "does the final website look right?" but also "did the model follow the correct process?" — for example, did it correctly parse the PRD, identify the required components, implement them in the right order, and validate each step?

Why this matters for optimization. The paper argues that workflow-based verification "makes it easier to compare systems, attribute failures, and model different forms of signal separately — for example, functional correctness during interactive execution and visual consistency in a more isolated comparison setting." By decomposing the evaluation into steps, you can identify where a model fails (e.g., it parsed the PRD correctly but generated incorrect CSS) rather than just that it fails. This provides more informative feedback for both evaluation and training.

The broader principle (Lens 3) is that "the value of an end-to-end task depends not only on how realistic it is, but also on whether it can be specified clearly enough, verified reliably enough, and evaluated under sufficient procedural control to produce stable and reusable feedback." This is presented as a design philosophy for agent evaluation: realistic tasks are valuable, but only if they can be reliably scored. An unverifiable task — no matter how realistic — provides no useful signal for optimization.

Summary of Key Design Choices and Their Justifications

Two-stage CogViT training (distillation + contrastive alignment) over standard CLIP-style training: produces dense, spatially precise representations needed for grounding and spatial reasoning, not just semantic categorization. The dual-teacher design (SigLIP2 for semantics, DINOv3 for texture) captures complementary aspects of visual information. QK-Norm prevents attention logit explosion during large-scale training.
MMTP with <|image|> placeholder over direct visual embeddings or masked visual tokens: balances multimodal modeling capability (preserves the signal that an image is present) with training stability (lightweight MTP heads cannot effectively process visual embeddings with different statistics) and system efficiency (avoids propagating visual embeddings across pipeline stages). Empirically validated with lower training loss on a 0.5B model.
Broad joint RL over 30+ task categories over narrow single-task or few-task RL: reduces cross-domain interference, stabilizes optimization in narrow domains, enables transfer of thinking patterns across tasks. The tradeoff is that uncovered capabilities may degrade — task coverage during RL defines the model's generalization boundary.
Decoupled, asynchronous RL infrastructure over synchronous pipeline: maximizes hardware utilization by overlapping rollout inference, reward evaluation, batch construction, and weight transfer. Completion callbacks prevent idle time from long-tail requests. Early-abort with caching prevents data waste.
Topology-aware visual input partitioning over standard data parallelism: addresses the unique challenges of variable-length visual sequences through upstream partitioning (before GPU), alignment with downsample groups, CPU-side communication buffering, and joint bin-packing over both text length and visual token count.
Multimodal toolchain over text-only tool interfaces: enables the model to interact with visual environments using visual operations (crop, annotate, reverse image search) rather than only text-based queries. This closes the perception-execution gap by letting the model act on what it sees.
Hierarchical optimization over end-to-end agent training: building up capabilities from perception and single-step actions to multi-step trajectories is more data-efficient and training-stable than directly optimizing long-horizon tasks.
Workflow-based verification in benchmarks over final-state-only evaluation: enables failure attribution, provides more informative feedback signals, and makes optimization more stable by decomposing complex tasks into verifiable steps.

4. Key Insights and Innovations

Innovation 1: Multimodal Agentic Capability Is a Systems Integration Problem, Not a Single-Model Problem

The dominant paradigm in foundation model development treats each model as a self-contained artifact: you design an architecture, train it on data, benchmark its outputs, and declare victory. The paper makes a fundamentally different argument — that building a capable multimodal agent requires simultaneous, coordinated advances across model architecture, training methodology, inference infrastructure, tool design, agent framework integration, and evaluation methodology, and that the integration of these components is itself the primary intellectual contribution, not any individual piece.

This is not merely a "we worked hard on engineering" claim. It is a reframing of what it means to build an agentic model. The field has largely proceeded by improving components in isolation — better vision encoders (SigLIP, DINOv3), better RL algorithms (PPO, DPO, GRPO), better benchmarks (MMMU, OSWorld) — under the implicit assumption that combining best-of-breed components would yield best-of-breed agents. The paper's experience contradicts this: the model cannot be separated from the harness, the training objective cannot be separated from the infrastructure that makes it feasible, and the evaluation cannot be separated from the verification design that makes it reliable.

Concretely, the paper shows that MMTP's <|image|> placeholder design (Section 2.2) was chosen not purely for accuracy but because it eliminates cross-pipeline-stage communication of visual embeddings — a systems consideration that became a model architecture decision. The joint RL over 30+ task categories (Section 2.3) is enabled only by the four-way infrastructure redesign (Section 2.4) that decouples rollout, reward, batch construction, and weight transfer. The toolchain (Section 3.1) is not an afterthought bolted onto a trained model but a design target that shaped what capabilities the model was trained to have. The Vision2Web benchmark's workflow-based verification (Section 4, Lens 3) is not just an evaluation choice but a feedback design that makes downstream optimization possible.

What makes this framing distinctive is that it inverts the usual relationship between model and system. In most papers, the model is the innovation and the system is the implementation detail. Here, the system shapes the model — infrastructure constraints drive architectural choices, verification design drives training objectives, and framework integration drives capability requirements. This is a fundamentally different development philosophy, and the paper argues (implicitly, through its structure) that it is necessary because agentic capability is an emergent property of the model-harness-tool-evaluation system, not a property of the model alone.

This is a fundamental shift, not incremental. Prior work has recognized that agents need tools and frameworks (e.g., Toolformer, WebGPT, SWE-Agent), but the model and the framework were developed separately — the model was trained to use tools, but the tools did not influence the model architecture. Here, the co-design is deeper: the model's perceptual training (CogViT's two-stage recipe), its training objective (MMTP's placeholder token), its RL task distribution (30+ categories spanning perception to execution), and its deployment toolchain (tools that consume and produce images) are all designed to reinforce each other. The evidence that this works is not a single ablation or metric but the pattern across Section 5: GLM-5V-Turbo performs strongly on multimodal coding, tool use, GUI agents, and text-only coding simultaneously — a breadth that single-component improvements rarely achieve.

Innovation 2: Hierarchical Optimization as a Training Strategy for Agentic Capabilities

Prior work on training agents has gravitated toward one of two extremes: either train on narrow, well-defined subtasks (element detection, action classification) and hope they compose, or train end-to-end on long-horizon trajectories and hope the model discovers the right intermediate behaviors. The paper identifies a third path — hierarchical optimization — and argues it is both more data-efficient and more training-stable than either extreme.

The concept is introduced in Lens 2 (Section 4) through the GUI-agent development example. Instead of training primarily on multi-step GUI trajectories (expensive to collect, hard to verify, prone to compounding errors in RL), the authors build a "multi-level task hierarchy spanning element perception, GUI grounding, single-step action prediction, and trajectory-level action prediction, and... use it in both SFT and RL." The hierarchy creates a curriculum: the model first learns to see GUI elements accurately, then to ground them in coordinates, then to predict individual actions, then to chain actions into trajectories. Each level provides learning signal for the next — accurate perception makes action prediction easier, reliable action prediction makes trajectory optimization more stable.

What makes this distinctive is that it is not just curriculum learning (train on easy tasks first, then hard ones) — it is simultaneous optimization across the hierarchy during RL. The 30+ task categories in the joint RL phase (Section 2.3) span the full hierarchy, from low-level perception (RefCOCO grounding, PointBench pointing, OCR) through mid-level reasoning (chart understanding, video understanding, STEM problem solving) to high-level agentic execution (OSWorld GUI tasks, MMSearch tool use, CC-Backend coding). The model is concurrently receiving reward signals at every level of the capability stack.

The paper reports three non-obvious benefits of this approach. First, multi-task RL "tends to show weaker interference across domains" compared to SFT, meaning improvements on one task category are less likely to degrade performance on others — a direct challenge to the intuition that training on many tasks creates destructive interference. Second, "in domains with narrower distributions where single-task RL is often prone to oscillation, collaborative training can make optimization more stable" — the diversity of tasks acts as a regularizer that prevents the model from overfitting to narrow task-specific strategies. Third, "some transfer of thinking patterns across tasks" occurs, where reasoning behaviors learned in one domain produce measurable benefits in another, suggesting the hierarchy induces shared representations at the strategy level, not just the feature level.

These observations matter because they contradict a commonly held view that agentic capabilities are best learned through immersion in realistic, end-to-end task environments (the "just let the agent explore" approach). The paper's counterargument is that without reliable lower-level capabilities, end-to-end optimization is both inefficient (the model spends most of its learning budget on basic perception and action, not on higher-level planning) and unstable (errors at lower levels compound, creating noisy reward signals for higher-level decisions). Hierarchical optimization solves this by ensuring that when the model attempts a complex trajectory, it already has reliable perception and action primitives to build on — the RL signal at the trajectory level can focus on strategy and planning rather than also needing to teach the model to see and click.

This is an incremental contribution in the sense that hierarchical training is not a new idea (curriculum learning, progressive networks, and skill chaining all have precedents), but it is fundamental in its implications for how agent training should be organized. The evidence is the breadth of improvements in Section 2.3: the RL stage produces gains across perception (+4.8% grounding, +7.7% chart understanding), reasoning (+1.8% STEM), and agentic execution (+4.9% OSWorld, +3.5% MMSearch) from a single joint optimization — gains that would typically require separate training runs for each capability.

Innovation 3: Perception as the Dominant Bottleneck for Higher-Level Multimodal Agentic Capability

The field has increasingly focused on "higher-level" capabilities — planning, reasoning, reflection, multi-step deliberation — as the frontier for agentic models, often treating perception as a largely solved problem (after all, VLMs can answer VQA questions with high accuracy). The paper makes a counterargument that is both simple and deeply consequential: perception errors are the root cause of many failures that appear high-level, and further gains in agentic capability depend critically on improving perception quality.

Lens 1 (Section 4) states this explicitly: "Even among the strongest current VLMs, errors in fine-grained perception and spatial understanding remain common, and these often propagate into downstream reasoning, decision-making, and execution. Many failures that appear high-level... begin with the model not seeing the environment accurately enough." This is not a claim about benchmark accuracy — it is a claim about failure mode attribution. When a GUI agent clicks the wrong button, is it because it made a bad planning decision, or because it misidentified the button's location? When a coding agent generates incorrect CSS from a screenshot, is it because it doesn't understand layout, or because it misread the pixel values? The paper argues that in many cases, the root cause is perceptual, not cognitive.

What makes this framing distinctive is that it inverts the usual prioritization. The standard narrative is: perception is good enough, now we need better reasoning. The paper's counter-narrative is: perception is still the limiting factor, and improving it unlocks gains across the entire capability stack. This is supported by several concrete observations:

Multimodal coding and grounding are "useful proxy tasks for perceptual learning." Tasks like frontend or SVG coding require the model to capture layout, structure, relative position, and local detail — exactly the perceptual capabilities that are most error-prone. The paper reports that "adding paired data between subject-specific images and their SVG representations during pretraining contributed positively to downstream STEM problem solving," and "strengthening grounding-related training during RL also improved GUI-agent performance." The causal chain is: perceptual training → better perceptual representations → better performance on tasks that depend on those representations, even when the tasks seem superficially unrelated.

Critic training on perception errors reduces hallucination. In GUI-agent instruction tuning, the authors include "a subset of critic data that targets errors in the reasoning process, such as misreading interface details, misidentifying target elements, and making incorrect decisions about the next action." This improves "the model's observation quality on GUI details and reduces several recurring perception failure modes." The mechanism is meta-cognitive: the model learns to detect its own perceptual errors, which prevents those errors from propagating into downstream actions. This is not just better perception — it is better self-monitoring of perception.

The significance of this insight extends beyond the paper's immediate results. If perception is the dominant bottleneck, then research priorities should shift: rather than investing primarily in better planning algorithms or longer reasoning chains, the highest-return investment is in better vision encoders, more diverse perceptual training data, and perceptual self-critique mechanisms. The paper's development of CogViT — with its two-stage training, dual teachers (SigLIP2 for semantics, DINOv3 for texture), QK-Norm for stability, and NaFlex for resolution flexibility — is a direct response to this diagnosis. The performance comparison in Figure 1, showing CogViT outperforming other encoders on both general and fine-grained tasks, provides supporting evidence that encoder quality matters in ways not captured by standard VLM benchmarks.

This is a reframing insight, not a new technique. The individual components (dual-teacher distillation, QK-Norm, NaFlex) are known techniques applied in a new combination. The intellectual contribution is the diagnosis that perception is the binding constraint and that investment in perceptual quality propagates through the entire agentic capability stack — a claim that, if correct, should change how the field allocates research effort.

Innovation 4: The Capability Boundary Is Co-Determined by Model and Harness, Not Separable

The final insight is the most philosophically ambitious and perhaps the most forward-looking. The paper argues that for agentic systems, "the effective capability boundary is no longer determined by the model alone, but jointly shaped by the model and the harness around it" (Section 6). This is not a claim about engineering convenience — it is a claim about the ontology of agentic capability. What the system can do is not a property of the model; it is a property of the model-harness interaction, and the two co-evolve.

The paper traces the implications of this observation through three consequences:

1. The harness is not a stable external layer. "The usefulness of a harness often depends on the model's capability regime, and designs that are ineffective at one stage may become critical once the model crosses a threshold in reasoning, planning, or feedback utilization." A tool that is useless when the model has weak planning (because the model cannot figure out when to invoke it) may become essential when the model's planning reaches a threshold. A memory mechanism that works for short trajectories may break for long ones. The harness must evolve with the model, and vice versa — a model upgrade may require harness redesign, and a harness improvement may unlock capabilities that were latent in the model.

2. Evaluation becomes inseparable from the system. "Task definition, verification design, and feedback structure should be considered together rather than in isolation." The Vision2Web benchmark (Section 4, Lens 3) is presented as a concrete instantiation: the task specification (PRD + mockups + reference pages + assets), the verification design (workflow-based with dependent steps), and the feedback structure (separate signals for functional correctness and visual consistency) are co-designed. You cannot evaluate the model independently of how the task is specified and verified, because the specification and verification determine what "success" means and what feedback the model receives.

3. Development can no longer be framed as model improvement alone. "Agentic model development can no longer be framed as model improvement alone: the effective capability boundary is increasingly co-shaped by the model and the harness, and so too are the objectives by which progress is optimized and evaluated." This is a challenge to the standard ML development paradigm: train a model, evaluate on a fixed benchmark, iterate. If the benchmark is part of the harness and the harness co-evolves with the model, then static benchmarks become misleading. Progress requires simultaneous innovation in models, harnesses, tools, and evaluation — which is, of course, exactly what the paper presents.

This insight is the deepest of the four and the hardest to operationalize. It is more a diagnosis of a structural challenge than a solution, and the paper acknowledges this in Section 6: "This greatly expands the design space... [and] makes the development path substantially complex." The evidence is not a single result but the paper's entire structure — the fact that a "model report" spends significant space on infrastructure (Section 2.4), toolchain (Section 3.1), framework integration (Section 3.2), benchmark design methodology (Sections 3.3 and 4), and skills (Section 3.5) is itself evidence that the boundary between model and system has dissolved. The strong results on framework-based evaluations (87.0/80.7 on PinchBench, 57.7/75.0 on ClawEval, 75.7 on AndroidWorld) validate that this integrated approach produces gains in realistic deployment settings, not just isolated benchmarks.

This is a fundamental conceptual contribution, not an incremental refinement. It challenges a core assumption of the field — that models and evaluation benchmarks are separable, stable entities — and argues that agentic capability requires a different development paradigm where the model, the harness, the tools, and the evaluation co-evolve. Whether the field adopts this view remains to be seen, but the paper makes a compelling case that ignoring it leads to models that benchmark well but fail in deployment because the deployment harness was not part of the design process.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates across four benchmark categories (Section 5): (1) Multimodal Coding: Design2Code [31], Flame-VLM-Code [9], Vision2Web [14]; (2) Multimodal ToolUse: ImageMining (newly introduced, Section 3.3), BrowseComp-VL [10], MMSearch [18], MMSearch-Plus [35], SimpleVQA [7], Facts [17], V* [41]; (3) GUI Agent: OSWorld [44], AndroidWorld [30], WebVoyager [13]; (4) Text-only Coding and Claw: CC-Bench-V2 [49], PinchBench [1], ClawEval [46], ZClawBench [2]. The ImageMining benchmark, introduced in this paper, contains 217 curated test cases across seven domains and five reasoning categories (Section 3.3). No single unified test set size is reported — each benchmark has its own standard split as defined by its originating publication.

Base model. GLM-5V-Turbo is built on the GLM-5-Turbo language model (Zeng et al., 2026), extended with the CogViT vision encoder and MMTP training. The language backbone serves as the reasoning engine. The paper positions this model as representing a capability range where test-time compute strategies can make a meaningful difference — neither trivial nor saturated on the target benchmarks. For comparison, the paper references Claude Opus 4.6 (Anthropic), Kimi K-2.5 (Kimi Team), and GLM-4.6V (the immediate predecessor from the same team, Team et al., 2025) as baselines in specific evaluations.

Metrics. The paper reports standard accuracy metrics appropriate to each benchmark: for multimodal coding, the metric is typically functional correctness or visual fidelity (Design2Code reports a pass rate; Vision2Web uses workflow-based verification with separate functional and visual consistency signals); for tool use and QA benchmarks, accuracy or F1 against ground-truth answers; for GUI agents, task success rate (the fraction of tasks completed successfully in the interactive environment); for text coding, pass@1 or task completion rates as defined by CC-Bench-V2 and the Claw framework evaluations. The paper does not provide a detailed formula for each metric, deferring to the original benchmark publications for exact computation.

Baselines. The paper compares against:

Claude Opus 4.6 [4]: on Design2Code (94.8 for GLM-5V-Turbo vs. the Opus 4.6 baseline, Figure 4) and multimodal tool use tasks.
Kimi K-2.5 [36]: on BrowseComp-VL and ImageMining, where GLM-5V-Turbo "matches or exceeds" performance (Section 3.1).
GLM-4.6V [37]: the immediate predecessor, with an ~8× improvement on MMSearch-Plus (30.0 vs. prior generation, Section 3.1) cited as evidence of the toolchain expansion's impact.
GLM-5-Turbo [49]: the text-only base model, used to verify that adding multimodal capability does not degrade text-only coding performance (Section 5, comparing CC-Backend, CC-Frontend, CC-RepoExploration scores).

For the RL stage impact (Section 2.3), the SFT-only version of the model serves as the baseline, with improvements reported on RefCOCO-avg (+4.8%), PointBench (+3.2%), MVBench (+5.6%), SUNRGBD (+7.7%), OCRBench (+4.2%), CharXiv (+7.7%), MMMU_Val/MMMU_Pro/MathVista/LogicVista averaged (+1.8%), OSWorld (+4.9%), CC-Backend (+0.2%), and MMSearch (+3.5%).

Generation budget / compute accounting. The paper does not report a standardized compute budget (e.g., FLOPs or number of generations) for evaluation, unlike the example paper which provides detailed FLOPs-matched comparisons. Instead, each benchmark is evaluated under its standard protocol — the model receives the test input and produces the output, with no explicit test-time compute scaling (no best-of-N, beam search, or iterative revision loops at test time). The "compute" for training is described qualitatively (two-stage CogViT training, joint RL over 30+ tasks) but no FLOP counts or GPU-hours are reported. This is a significant departure from the example paper's careful compute accounting and represents a limitation in the experimental methodology — the reader cannot assess the computational cost of achieving the reported performance.

Cross-validation / statistical protocol. The paper does not describe any cross-validation protocol, statistical significance testing, or confidence intervals for the benchmark results. Section 2.3 reports RL improvements as point estimates (+4.8%, +3.2%, etc.) without error bars, standard deviations, or multiple seeds. The ImageMining benchmark (Section 3.3) uses "217 curated test cases derived from manually collected trace samples" but does not describe a held-out validation split or how hyperparameters were tuned. For Vision2Web (Section 4), workflow-based verification is described qualitatively but no statistical protocol for aggregating scores across tasks is provided. The RL stage (Section 2.3) is described as "joint RL optimization over more than 30 task categories" but the paper does not specify how the mixture weights across tasks were determined, whether they were tuned on a validation set, or whether the reported improvements are robust to different task weightings.

Main Quantitative Results

Multimodal Coding Performance

GLM-5V-Turbo achieves 94.8 on Design2Code, outperforming Claude Opus 4.6 (Section 5, Figure 4). The task requires generating functional web code (HTML/CSS) from visual design mockups — testing the model's ability to perceive layout, structure, spatial relationships, and translate them into executable code. The paper also reports results on Flame-VLM-Code and Vision2Web (Figure 4), though specific numbers for these benchmarks are not quoted in the text. The Design2Code result is the headline claim for this category and is shown in the bar chart in Figure 4 alongside other multimodal coding benchmarks.

The significance of the Design2Code result is that it demonstrates the model's ability to bridge perception and code generation — a core agentic capability. When reproducing a website, the model must not only recognize visual elements (buttons, text, images, layout) but also infer their functional behavior and generate semantically correct, executable code. The paper frames this (Section 3.1) as a capability that enables the model to "first use a multimodal GUI agent to explore the site through screenshots... building a richer understanding of layout, functionality, and interaction flow" and then "rely on its native UI-to-code capability to reproduce the site more faithfully." The 94.8 score provides quantitative evidence that this perception-to-code pipeline works.

Multimodal Tool Use and Search

The paper reports results across seven tool-use benchmarks (Section 5, Figure 4):

ImageMining: 30.7 (this paper's newly introduced benchmark)
BrowseComp-VL: 51.9
MMSearch: 72.9
MMSearch-Plus: 30.0 (described as "nearly an eightfold improvement over the previous generation" GLM-4.6V, Section 3.1)
SimpleVQA: 78.2
Facts: (score visible in Figure 4 but not quoted in text)
V*: (score visible in Figure 4 but not quoted in text)

The BrowseComp-VL result (51.9) is highlighted alongside the Kimi K-2.5 comparison (Section 3.1): GLM-5V-Turbo "matches or exceeds" comparable models in these categories. BrowseComp-VL tests the model's ability to navigate web interfaces and extract information, making it a direct test of the perception-planning-execution loop that the paper claims as its core capability.

The ImageMining score (30.7) is particularly informative because the benchmark was specifically designed (Section 3.3) to test the integration of visual understanding and autonomous search. The paper explicitly states that "task performance correlates strongly with the precision of on-image tool usage" — meaning that the score reflects not just visual recognition accuracy but the model's ability to use tools like cropping, magnification, and reverse image search strategically. An absolute score of 30.7 on 217 test cases indicates that the task remains challenging (the model fails on roughly 70% of cases), which the paper acknowledges implicitly by positioning ImageMining as a "vision-centric deep search" benchmark that pushes beyond current capabilities. No breakdown by the five reasoning categories (Universal Recognition, Spatio-Temporal Reasoning, Event Reasoning, Text-based Reasoning, Visual Search) or seven domains is provided, which would have been informative for understanding where the model's strengths and weaknesses lie.

GUI Agent Performance

GLM-5V-Turbo achieves 75.7 on AndroidWorld and 62.3 on OSWorld (Section 5, Figure 4). These are interactive benchmarks where the model must complete real tasks in simulated mobile (AndroidWorld) and desktop (OSWorld) environments — installing apps, filling forms, navigating settings, and performing multi-step workflows. The tasks require the model to perceive the current screen state, decide on an action (tap, swipe, type, etc.), execute it, observe the result, and adapt.

The 75.7 on AndroidWorld is the strongest GUI agent result reported and is a key validation of the paper's claim that multimodal perception transfers effectively into grounded interaction and action (Section 5, text). The 62.3 on OSWorld is somewhat lower, which is consistent with OSWorld being a more challenging benchmark — it involves real computer environments with more complex interfaces, longer task horizons, and greater variability in valid solution paths.

The paper also reports results on WebVoyager (Figure 4), though the specific score is not quoted in the text. WebVoyager tests web navigation capabilities, complementing the OS-level benchmarks.

The RL stage contribution to GUI agent performance is quantified in Section 2.3: +4.9% on OSWorld from the RL stage compared to the SFT baseline. This is a substantial gain — approximately an 8-9% relative improvement assuming a base rate around 55-57% — and provides evidence that the joint multi-task RL training improves agentic execution, not just perception and reasoning.

Text-Only Coding Performance (Preservation of Base Model Capability)

A critical claim in the paper is that adding multimodal capabilities does not degrade text-only performance. The results on CC-Bench-V2 (Section 5, Figure 5) support this:

CC-Backend: 22.8 (compared to GLM-5-Turbo, the text-only base model)
CC-Frontend: 68.4 (the paper states GLM-5V-Turbo "even surpasses" the base model on this and CC-RepoExploration)
CC-RepoExploration: 72.2 (also surpassing the base model)

These benchmarks evaluate model performance within the Claude Code framework [3] on backend coding tasks (server-side logic, database operations), frontend coding (UI implementation, styling), and repository exploration (understanding and navigating large codebases). The fact that GLM-5V-Turbo preserves or exceeds the base model's performance is non-trivial — it demonstrates that the multimodal training (CogViT integration, MMTP objective, broad multimodal pretraining, joint RL over 30+ tasks) did not cause catastrophic interference with the model's text-only coding capabilities.

The small RL improvement on CC-Backend (+0.2%, Section 2.3) suggests that coding capability was already strong from pretraining and that the RL stage primarily benefits perceptual and agentic tasks rather than pure coding. This is consistent with the paper's framing — the model preserves text coding while adding multimodal agentic capabilities, rather than trading one for the other.

Framework-Based Agent Evaluations (Claw Integration)

GLM-5V-Turbo was evaluated when integrated into Claw agent frameworks (Section 5, Figure 5):

PinchBench: 87.0 / 80.7 (two scores reported, likely representing different settings or task subsets)
ClawEval: 57.7 / 75.0 (two scores)
ZClawBench: 57.6

These benchmarks evaluate end-to-end agent execution in realistic digital environments — the model must perceive on-screen content, reason about what actions to take, and execute them through the Claw framework's execution layer. The results show strong performance on PinchBench (mid-to-high 80s) and more moderate performance on ClawEval and ZClawBench (mid-50s to mid-70s). The paper presents these as evidence that "the model's multimodal capability is not limited to isolated benchmark gains, but carries over to realistic end-to-end agent execution" (Section 5).

The dual scores on PinchBench and ClawEval raise a question: what do the two numbers represent? The paper does not explain this in the evaluation section. One plausible interpretation is that they represent different settings — perhaps with and without certain tools, or with different Claude Code integration configurations. Without clarification, the dual scores are ambiguous.

RL Stage Improvements Across Task Categories

Section 2.3 reports quantitative improvements from the RL stage (joint optimization over 30+ task categories) compared to the SFT baseline. The numbers are organized by capability category:

Perception: +4.8% RefCOCO-avg (2D grounding), +3.2% PointBench (pointing), +5.6% MVBench (video understanding), +7.7% SUNRGBD (3D grounding), +4.2% OCRBench (OCR), +7.7% CharXiv (chart understanding).

Reasoning: +1.8% average across MMMU_Val, MMMU_Pro, MathVista, and LogicVista (STEM and logical reasoning in visual contexts).

Agentic execution: +4.9% OSWorld (GUI agents), +0.2% CC-Backend (coding agents), +3.5% MMSearch (general tool use).

These results support the paper's hierarchical optimization claim (Lens 2, Section 4): improvements are distributed across the capability stack, from low-level perception (grounding, OCR) through mid-level reasoning (chart understanding, STEM) to high-level agentic execution (GUI, tool use). The pattern is consistent with the paper's argument that multi-task RL yields "stable gains" across domains (Section 2.3).

However, the paper reports only point estimates without error bars, significance tests, or information about the number of evaluation runs. The +0.2% on CC-Backend is notably small and may not be statistically distinguishable from zero — the paper does not discuss whether this represents a genuine (if tiny) improvement or noise.

Ablation Studies and Robustness Checks

MMTP design alternatives (Section 2.2, Figure 2): The paper compares three options for passing visual information to the multi-token prediction heads: (1) direct vision embeddings, (2) masked vision tokens, and (3) <|image|> placeholder. The ablation on a 0.5B model shows that Option 3 achieves "lower training loss and more stable convergence" than Option 1 (direct embeddings). The paper hypothesizes this is because "the MTP head is typically lightweight, and may not have sufficient modeling capacity to effectively absorb visual representations whose distribution differs substantially from that of text embeddings." Figure 2 (bottom-left) shows the training loss curves, with Option 3 lower throughout training. No ablation on the full-scale model is reported, and no downstream task performance comparison is provided — the ablation is purely on training loss for a small-scale model.

Dual-teacher distillation for CogViT (Section 2.1): The CogViT encoder uses two teacher models in its first training stage — SigLIP2 (semantic) and DINOv3 (texture). This is a design choice, but the paper does not provide an ablation comparing single-teacher vs. dual-teacher training. We do not know whether both teachers are necessary, or whether one dominates. Figure 1 shows performance comparisons of CogViT against other vision encoders (presumably CLIP, SigLIP, etc.), but this compares the final encoder to alternatives, not the contribution of each teacher.

QK-Norm stability (Section 2.1): The introduction of query-key normalization before attention computation is described as "effectively mitigating logit explosion and ensuring stability at scale." However, no ablation is provided — we do not see training curves with and without QK-Norm, nor any quantitative measure of stability improvement. The claim is based on known behavior from prior work (Henry et al., 2020) rather than an empirical demonstration in this specific setting.

NaFlex variable resolution vs. fixed resolution (Section 2.1): The transition from fixed 224 × 224 resolution (Stage 1) to NaFlex variable-resolution processing (Stage 2) is described as important for preserving original aspect ratios in agentic tasks. No ablation comparing fixed-resolution and variable-resolution inputs on downstream agentic benchmarks is provided, so the practical impact on agentic task performance is unknown.

Impact of multimodal training on text-only performance: The paper claims that "GLM-5V-Turbo preserves the coding capability of its language-only base model GLM-5-Turbo and even surpasses it" on certain benchmarks (Section 5). The results in Figure 5 show CC-Frontend (68.4 vs. base model) and CC-RepoExploration (72.2 vs. base model) where GLM-5V-Turbo surpasses the base model. This is a robustness check for the claim that multimodal capabilities do not degrade text performance. However, the paper does not show a comprehensive text-only evaluation (e.g., standard NLP benchmarks like MMLU, HellaSwag, GSM8K) to verify that general language capabilities are preserved — only coding benchmarks within the Claude Code framework are reported.

Task coverage during RL and capability degradation (Section 2.3): The paper reports a negative result: "capabilities left uncovered during RL can sometimes decline after post-training, especially those more orthogonal to the trained task distribution." This is not documented with specific before-and-after numbers (e.g., "performance on task X dropped from Y to Z after RL"), making it difficult to assess the magnitude of the degradation. The paper frames this as a motivation for broad task coverage during RL but does not provide the quantitative evidence that would allow readers to gauge the severity of the problem.

On-policy distillation vs. ReST^EM (Section 2.3 and Appendix K in the example paper's context): The paper mentions "on-policy distillation" as part of the multi-task RL approach but does not compare it against alternative RL formulations (e.g., off-policy distillation, standard PPO without distillation). There is no ablation showing that on-policy distillation provides benefits over simpler RL approaches.

Revision model not ablated: Unlike the example paper, GLM-5V-Turbo does not include a revision model component, so no revision-related ablations are present.

Verifier design not ablated: The paper describes an "independent reward system that centrally orchestrates multiple verifiers" (Section 2.4) but does not ablate different verifier configurations — e.g., rule-based only vs. model-based only vs. combined, or different aggregation strategies. The sentence "their outputs are then combined into rewards through configurable aggregation strategies" suggests flexibility but provides no empirical comparison of strategies.

Toolchain expansion impact: The paper reports an ~8× improvement on MMSearch-Plus from GLM-4.6V to GLM-5V-Turbo (30.0 vs. "previous generation," Section 3.1), attributing this partly to toolchain expansion. However, this improvement conflates model upgrades (CogViT, MMTP, broader RL), toolchain improvements, and framework integration changes — the paper does not isolate the contribution of toolchain expansion alone.

Hierarchical optimization vs. end-to-end training (Lens 2): The paper argues that hierarchical optimization (distributing training across perception, grounding, single-step actions, and trajectories) is more efficient than end-to-end training. This is a central design claim. However, no ablation compares the adopted hierarchical approach against a baseline trained only on end-to-end trajectories with equivalent total data. The claim is supported by the qualitative observation that "when lower-level capabilities are still underdeveloped, pushing only on high-level tasks often fails to yield reliable gains and can instead make training less stable" (Section 4), but no quantitative stability comparison (e.g., reward variance, training loss curves, success rate variance) is provided.

Infrastructure optimizations performance impact: Section 2.4 describes four systematic infrastructure redesigns for multimodal RL at scale. The paper mentions that moving "large Python objects off the GPU communication path... reduces GPU communication buffer overhead by about 7 GB in practice." This is a concrete number. However, no end-to-end throughput comparison is provided (e.g., "training throughput improved by X% with these optimizations"). The 7 GB figure quantifies memory savings but not speed improvements.

Critical Assessment

Claim 1: "Multimodal perception is integrated as a core component of reasoning, planning, tool use, and execution, rather than as an auxiliary interface to a language model."

What was tested: The paper evaluates GLM-5V-Turbo on benchmarks that require perception to be integrated with action — GUI agents (AndroidWorld, OSWorld), multimodal coding (Design2Code), and tool use (ImageMining, BrowseComp-VL, MMSearch). These benchmarks inherently require the model to perceive visual inputs and then act on them, which is consistent with the claim.

What was not tested: The paper does not provide a head-to-head comparison against a baseline model that treats perception "as an auxiliary interface" — e.g., a standard VLM architecture (CLIP encoder + LLM) trained on the same data and evaluated on the same agentic benchmarks. Without this comparison, we cannot attribute the strong agentic performance specifically to the "native integration" design choices (two-stage CogViT training, MMTP placeholder, broad joint RL) rather than to scale, data quality, or other factors. The comparison to GLM-4.6V shows improvement, but GLM-4.6V was also a multimodal model — the paper does not demonstrate that a "non-native" multimodal approach with equivalent resources would underperform.

The Design2Code result (94.8 vs. Claude Opus 4.6) is the closest to such a comparison, but it is a single benchmark. We do not know whether Claude Opus 4.6 uses a "native" or "auxiliary" perception approach, and the paper does not characterize it as such.

Verdict: The claim is plausible and consistent with the results, but the experimental design does not isolate the "native integration" design choice. The strong agentic results could be due to better data, more compute, better RL infrastructure, or other unobserved factors.

Claim 2: "GLM-5V-Turbo achieves strong results on multimodal agentic benchmarks... while preserving competitive text-only coding capability."

What was tested: The paper reports CC-Bench-V2 results (CC-Backend 22.8, CC-Frontend 68.4, CC-RepoExploration 72.2) and states these preserve or exceed the base GLM-5-Turbo model's performance. This is direct evidence for preserved text coding capability.

What was not tested: The paper only reports coding benchmarks — not general language understanding benchmarks (MMLU, HellaSwag, ARC, GSM8K, HumanEval). "Text-only coding capability" is a subset of text-only capability. A reader interested in whether the model can still perform standard NLP tasks (summarization, translation, question answering at base-model levels) would not find evidence here. The claim "preserving competitive text-only coding capability" is narrower than "preserving text-only capability" and is supported. But the broader implication — that multimodal training does not interfere with text capabilities — is tested only on coding.

Additionally, the paper reports that GLM-5V-Turbo "even surpasses" the base model on CC-Frontend and CC-RepoExploration. This is surprising: adding multimodal tasks during training improved text-only coding. Possible explanations include (a) positive transfer from multimodal coding data (e.g., UI-to-code training improved general frontend coding skill), (b) the RL stage providing beneficial regularization, or (c) statistical noise. The paper does not investigate which explanation is correct.

Verdict: Supported for coding specifically, but the evidence base is narrow. A claim about "preserving competitive text-only capability" would require broader NLP evaluation.

Claim 3: "Multimodal coding and grounding proved to be useful proxy tasks for perceptual learning" — with downstream benefits to STEM and GUI-agent performance.

What was tested: Section 4 (Lens 1) reports two specific observations: (1) "adding paired data between subject-specific images and their SVG representations during pretraining contributed positively to downstream STEM problem solving," and (2) "strengthening grounding-related training during RL also improved GUI-agent performance." These are causal claims: perceptual training → better STEM/GUI performance.

What was not tested: The paper does not provide quantitative evidence for either claim. For the SVG-to-STEM transfer: we do not see a table comparing STEM performance with and without the SVG paired data, nor a regression showing the relationship between SVG data quantity and STEM improvement. For the grounding-to-GUI transfer: the RL stage improves both grounding (+4.8% on RefCOCO) and GUI performance (+4.9% on OSWorld) simultaneously, but this does not demonstrate that the grounding improvement caused the GUI improvement — both could be independent effects of the joint RL training. A proper test would require, for example, training a model variant with grounding tasks removed from the RL mixture and showing that GUI performance degrades specifically because perceptual capabilities are weaker.

The claim that critic training on perception errors "improves the model's observation quality on GUI details and reduces several recurring perception failure modes" is similarly unquantified. No error-type breakdown or failure mode frequency comparison with and without critic training is provided.

Verdict: These are plausible hypotheses consistent with the observed simultaneous improvements, but the paper presents them as empirical findings without the controlled experiments that would establish causality. The claims are weakly supported by the evidence provided.

Claim 4: "Agent capability can be more efficiently built through hierarchical optimization."

What was tested: The joint RL stage simultaneously trains on tasks spanning perception through execution and produces improvements across all levels (Section 2.3). The paper argues this is more efficient than training only on high-level tasks.

What was not tested: The paper does not compare the hierarchical approach against a "flat" approach that trains only on end-to-end agent trajectories with equivalent total data and compute. Without this comparison, "more efficient" is unverified. The observation that single-task RL is "prone to oscillation" while collaborative training is "more stable" is qualitative — no stability metric (e.g., reward variance over training steps, success rate variance across seeds) is reported for either approach.

The Vision2Web benchmark (Section 4, Lens 3) is described as an instantiation of hierarchical verification (workflow-based with dependent steps), but the paper does not show that models trained with this verification perform better than models trained with final-state-only verification. Vision2Web results appear in Figure 4 but are not discussed in the context of verifying the hierarchical optimization claim.

Verdict: The hierarchical optimization claim is presented as a "design lens" — an insight from the development process — rather than as a rigorously tested empirical finding. The paper provides suggestive evidence (simultaneous improvements across capability levels, qualitative stability observations) but no controlled experiment that would establish the efficiency advantage of hierarchical over flat training.

Claim 5: "The key to constructing, evaluating, and optimizing end-to-end long-horizon tasks lies in clear task specification, reliable outcome verification, and controlled evaluation procedures."

What was tested: The Vision2Web benchmark is presented as an instantiation of these principles. The ImageMining benchmark's "Visual Jump" constraint and multi-step tool-use requirements also reflect careful task specification.

What was not tested: The paper does not compare Vision2Web's workflow-based verification against simpler verification schemes (e.g., final-state screenshot comparison, or human evaluation) in terms of reliability (inter-rater agreement, score stability across runs) or usefulness for optimization (does training with workflow-based rewards produce better models than training with final-state rewards?). The claim is presented as a design philosophy, not as an empirically validated finding.

The broader challenge here is that evaluating evaluation methodologies requires meta-evaluation — comparing how different verification designs affect downstream model quality or measurement reliability — which is expensive and rarely done. The paper does not attempt this.

Verdict: This is a methodological position, not an empirical claim, and the paper does not provide the kind of evidence that would validate it. Vision2Web is a concrete attempt to implement the principles, but whether it succeeds in providing "stable and reusable feedback" (Section 4) is not demonstrated through comparisons with alternative verification designs.

Missing Experiments That Would Have Strengthened the Paper

Head-to-head architecture ablation: A comparison of GLM-5V-Turbo against a version with a standard vision encoder (e.g., SigLIP-only) and standard MTP (or no MTP) trained on the same data and evaluated on the same agentic benchmarks. This would isolate the contribution of the CogViT + MMTP design choices.
RL stage ablation: The joint RL stage trains on 30+ task categories simultaneously — what happens if you train on perception tasks only, or agentic tasks only, or reasoning tasks only? Does multi-task RL genuinely outperform single-task or few-task RL with equivalent total compute? Without this, the "weaker interference" and "stability" claims remain qualitative.
Text-only capability preservation across NLP benchmarks: A table showing GLM-5-Turbo vs. GLM-5V-Turbo on standard benchmarks (MMLU, GSM8K, HumanEval, HellaSwag, etc.) to support the claim that multimodal training does not degrade general language capability.
Difficulty-stratified analysis: The example paper provides extensive analysis by difficulty bin, showing when search helps and when it hurts. GLM-5V-Turbo reports no such analysis — we do not know whether the model's gains are concentrated on easy tasks, medium tasks, or hard tasks within each benchmark.
Statistical significance and variance: The paper reports point estimates without error bars, confidence intervals, or multiple evaluation runs. For benchmarks with small test sets (ImageMining: 217 cases; specific benchmark subsets may be smaller), score differences of a few percentage points may not be statistically reliable.
Compute cost analysis: The paper does not report training compute (GPU-hours, FLOPs), making it impossible to assess whether the reported performance gains are cost-effective compared to alternative approaches (e.g., simply scaling up a standard VLM architecture with the same compute budget).

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Headline Performance — and Is Extraordinarily Expensive

The assumption or constraint. The paper's training framework — particularly the joint RL stage over 30+ task categories (Section 2.3) — implicitly assumes that high-quality reward signals can be computed reliably and efficiently for a broad distribution of tasks. This is not a stated assumption but rather a structural one: the training strategy only works if verifiers exist for each task category, and those verifiers are sufficiently accurate and computationally cheap that running them at scale is feasible. The paper acknowledges this obliquely in Lens 3 (Section 4): "the real challenge is often not extending tasks to longer horizons, but making end-to-end tasks stable enough to serve as meaningful targets for evaluation and optimization." This transparency about the difficulty of verification design is valuable, but it also reveals a fundamental cost the paper does not account for.

The consequence. The headline improvements in Section 2.3 — +4.8% on RefCOCO, +7.7% on CharXiv, +4.9% on OSWorld — all depend on having verifiers for those tasks. Building these verifiers requires substantial engineering investment, and running them at RL scale requires compute that is not included in any reported efficiency metric. The paper describes an "independent reward system that centrally orchestrates multiple verifiers" (Section 2.4), including "rule-based verifiers executed locally and synchronously" and "model-based judges invoked asynchronously through APIs." Model-based judges — which are separate LLMs evaluating the quality of outputs — are particularly expensive: for every training sample, the system must run an additional (potentially large) model to compute the reward signal. This cost scales with the number of RL training steps and the number of tasks using model-based judges, and it is not amortized in any reported metric.

The practical consequence is that the reported improvements cannot be achieved simply by "running joint RL over 30+ tasks" — they require, as a prerequisite, the construction and deployment of a reliable verification infrastructure for each of those 30+ tasks. For a team attempting to replicate this approach on a different domain, the hidden cost is not the RL training itself but the verifier development and operation. The paper provides no guidance on how much verification infrastructure is needed, what the tradeoffs are between rule-based and model-based verifiers, or what the compute overhead of verification is relative to the main training loop.

What evidence exists in the paper. The paper does not measure this cost. Section 2.4 describes the reward system architecture but provides no numbers: no GPU-hours for model-based judges, no latency measurements for asynchronous API calls, no comparison of reward computation cost vs. main model training cost. The "performance improvements" in Section 2.3 are reported as raw accuracy gains without any efficiency denominator — we do not know whether achieving a +0.2% improvement on CC-Backend required more FLOPs in verification than in training. The 7 GB memory savings from GPU communication path optimization (Section 2.4) is the only concrete resource number in the entire training infrastructure description, and it addresses memory, not verification cost.

Mitigation status. The paper does not address this. Section 2.4 focuses on making the training pipeline efficient (overlapping stages, managing memory, balancing loads) but does not address the cost of producing the reward signals that drive the training. The "configurable aggregation strategies" for combining multiple verifiers suggest flexibility but no attempt to minimize verification cost. Lens 3 (Section 4) discusses verification design as a methodology problem but not as a cost problem. A practitioner reading this paper would have no way to estimate the total compute required to replicate the approach — they would see the training infrastructure description and the benchmark results but would be missing the verification compute budget entirely.

Text-Only Capability Preservation Is Tested Only on Coding Benchmarks, Not on General Language Capabilities

The assumption or constraint. A central claim of the paper is that adding multimodal capabilities does not degrade the model's text-only performance — and may even improve it. The paper states that GLM-5V-Turbo "preserves the coding capability of its language-only base model GLM-5-Turbo and even surpasses it on CC-Backend (22.8), CC-Frontend (68.4), and CC-RepoExploration (72.2)" (Section 5). This is presented as evidence that "native multimodal agentic capability can be built without sacrificing text-based reasoning" (Section 1). The implicit assumption is that text-only coding benchmarks are representative of text-only capability in general — that preserving or improving coding performance implies preserving language reasoning, factual knowledge, and other text-only skills that matter for agentic deployment.

The consequence. This assumption matters because catastrophic interference — where new capabilities degrade existing ones — is a well-documented risk in multimodal model development. A model that excels at UI-to-code generation but has degraded its ability to, say, follow complex instructions, reason about abstract concepts, or recall factual knowledge from pretraining would be a poor foundation for an agent. Agents need to read documentation, plan multi-step workflows, debug errors, and communicate with users — all text-heavy tasks that depend on general language capabilities, not just coding. If those capabilities degraded during multimodal training, the model might perform well on Design2Code (which it was heavily optimized for) but fail on tasks that require, for example, understanding a nuanced error message or following a complex multi-constraint instruction.

The paper's evidence is also surprisingly directional: GLM-5V-Turbo surpasses its text-only base on two of three coding benchmarks. This suggests positive transfer — that multimodal training somehow improved text-only coding. While this is a favorable result, it raises questions about mechanism: is this genuine capability improvement, or is it an artifact of the specific benchmark? If multimodal coding data (e.g., UI-to-code pairs) improved the model's general frontend coding skill, that is a positive finding worth understanding. But if the improvement is due to overfitting to the Claude Code evaluation framework (which is also used in some of the multimodal agent evaluations), the result may not generalize to other coding tasks or frameworks.

What evidence exists in the paper. The evidence is solely the CC-Bench-V2 results in Section 5 and Figure 5 — three numbers covering backend coding, frontend coding, and repository exploration within the Claude Code framework. The paper does not report scores on any standard NLP benchmarks: no MMLU (general knowledge and reasoning), no GSM8K (mathematical reasoning), no HumanEval (code generation outside Claude Code), no HellaSwag (commonsense reasoning). The paper does not provide baseline GLM-5-Turbo scores on these benchmarks either, making it impossible for a reader to assess whether the base model was already strong on coding but weak on other dimensions — and thus whether "preserving coding" is a selective report of the one capability that survived multimodal training intact.

The RL stage improvements in Section 2.3 provide one additional data point: the +0.2% on CC-Backend is notably small, while gains on perceptual and agentic tasks are much larger (+4.9% on OSWorld, +7.7% on CharXiv). This pattern is consistent with the RL stage primarily improving multimodal capabilities while leaving text-only coding largely unchanged — but it does not speak to text capabilities beyond coding.

Mitigation status. The paper partially addresses this by showing preserved coding performance, but the scope is too narrow to support the implication that all text-only capabilities are preserved. The paper does not acknowledge this as a limitation — it presents the coding results as sufficient evidence for the "preserving text-based reasoning" claim. A comprehensive text-only evaluation suite would be the standard mitigation, but the paper does not provide one. The authors could have reported a standard set of NLP benchmarks (MMLU, GSM8K, ARC, HellaSwag, etc.) to establish that general language capabilities are intact, but they did not. This leaves open the possibility that the model's text-only capabilities outside of coding have degraded in ways not captured by the CC-Bench-V2 evaluation.

Hard Problems and Out-of-Distribution Tasks Remain Essentially Unsolved — Vision-Centric Deep Search Scores Are Modest

The assumption or constraint. The paper presents GLM-5V-Turbo as a step toward "native foundation models for multimodal agents" — capable of perceiving, reasoning, planning, and executing across heterogeneous environments. The assumption is that the model's capabilities, while strong on the evaluated benchmarks, will transfer to the kinds of open-ended, complex tasks that real-world agents encounter. The paper explicitly introduces ImageMining (Section 3.3) as a benchmark that tests this integration — requiring models to "actively mine visual inputs through agentic behaviors" including multi-step tool calls, localized cropping, and cross-referencing of visual evidence.

The consequence. The ImageMining score of 30.7 (Section 5, Figure 4) is the paper's own chosen metric for vision-centric deep search capability, and it reveals a substantial capability gap. On 217 curated test cases spanning seven domains and five reasoning categories, the model succeeds on fewer than one-third. This is not a low score because ImageMining is artificially hard — it is low because the tasks are genuinely complex, requiring the model to chain together visual perception, tool use, and multi-step reasoning in ways that push beyond current capabilities. The paper emphasizes that "task performance correlates strongly with the precision of on-image tool usage" (Section 3.3), meaning that failures stem from the core capability the paper claims to have improved — the integration of perception with tool-based action.

Other challenging benchmarks tell a similar story: MMSearch-Plus at 30.0 (an ~8× improvement over the previous generation but still a 30% success rate), BrowseComp-VL at 51.9 (slightly better than half), ZClawBench at 57.6. These are not catastrophic failures — they represent state-of-the-art or near-state-of-the-art performance — but they reveal that on the hardest, most integrated tasks, the model's success rate is modest. For a practitioner considering deployment, these numbers mean that on complex visual reasoning tasks requiring multiple tool calls, the model will fail roughly two-thirds to three-quarters of the time — a failure rate that may be unacceptable for autonomous deployment without human oversight.

The paper does not provide a difficulty-stratified breakdown of these results (unlike the example paper, which extensively analyzes performance by difficulty bin). We do not know whether the successes on ImageMining are concentrated in easier categories (e.g., Universal Recognition) while harder categories (e.g., Spatio-Temporal Reasoning, Event Reasoning) show near-zero performance, or whether performance is uniformly modest across categories. This stratification would be practically valuable: a practitioner could assess whether the model is reliable enough for their specific domain (e.g., product search, where Universal Recognition may dominate) or too unreliable (e.g., investigative research, where Event Reasoning is critical).

What evidence exists in the paper. The ImageMining score (30.7) is the primary evidence, reported in Section 5 and Figure 4. Section 3.3 describes ImageMining's composition — 217 cases, seven domains, five reasoning categories — but does not provide per-category or per-domain breakdowns. The paper does not analyze failure modes on ImageMining or characterize what kinds of tasks the model succeeds on vs. fails on. The MMSearch-Plus score (30.0) and BrowseComp-VL score (51.9) provide convergent evidence that integrated vision-search-reasoning tasks remain challenging. OSWorld at 62.3 and AndroidWorld at 75.7 are stronger, suggesting that GUI tasks (where the action space is more constrained and the perception demands are somewhat more structured) are more tractable than open-ended visual search. But this pattern is not discussed or analyzed in the paper.

Mitigation status. The paper does not attempt to address this limitation — it does not analyze failure modes, provide difficulty breakdowns, or propose strategies for improving performance on the hardest tasks. Lens 2 (Section 4) discusses hierarchical optimization as a way to improve training efficiency, but this addresses how to train the model, not what the model's current capability ceiling is. Section 6 acknowledges that "the hardest open problems increasingly lie not in isolated capability improvement, but in agentic strategy emergence" — which is a forward-looking recognition that current performance is insufficient, but not an analysis of the current capability boundary. A practitioner who wants to know whether GLM-5V-Turbo can handle their specific use case (which may be more like hard ImageMining tasks than like AndroidWorld) receives little guidance from the presented results.

The Paper Does Not Isolate the Contribution of Individual Design Choices — It Is a Full-System Report, Not a Controlled Ablation Study

The assumption or constraint. GLM-5V-Turbo is the result of coordinated advances across model architecture (CogViT, MMTP), training methodology (two-stage pretraining, joint RL over 30+ tasks), infrastructure (decoupled asynchronous pipeline, topology-aware partitioning), toolchain (multimodal tools, framework integration), and evaluation (ImageMining, Vision2Web). The paper presents all of these as an integrated system and reports the aggregate performance of the final model. The assumption is that the combination of these components is what produces the observed capabilities — that CogViT matters, that MMTP matters, that the broad RL task distribution matters, that the infrastructure redesigns matter.

The consequence. A practitioner attempting to learn from this paper — or to replicate its approach on a different model family, domain, or scale — cannot determine which components are essential and which are incidental. If CogViT's two-stage training with dual teachers is critical for the Design2Code result, but the MMTP placeholder design is only a minor efficiency improvement, a team with limited resources should prioritize CogViT and skip MMTP. Conversely, if the broad RL task distribution is the dominant factor and the specific vision encoder matters less, a team with an existing strong vision encoder should focus on expanding their RL task coverage. The paper provides no evidence to guide these decisions.

The specific missing ablations include:

CogViT vs. standard encoder: What would performance be with a SigLIP-only or CLIP-only encoder, keeping all other components identical? Figure 1 compares CogViT to other encoders on "general and fine-grained multimodal tasks," but this compares encoder outputs, not downstream agentic performance with the full GLM-5V-Turbo system.
MMTP placeholder vs. alternatives: Section 2.2 shows training loss curves on a 0.5B model (Option 3 achieves lower loss than Option 1), but does not show downstream task performance. Does the lower training loss translate to better agentic task success, or is it a minor optimization that doesn't affect final capability?
30-task RL vs. narrower RL: The paper claims multi-task RL shows "weaker interference" and "more stable optimization" than single-task RL (Section 2.3). No comparison is provided: what if the model were trained on only the 5 best-performing task categories, or only agentic tasks without the perception tasks? The +0.2% on CC-Backend suggests some tasks benefit minimally — could resources be reallocated from coding RL to GUI RL for larger gains?
Two-stage CogViT vs. single-stage: The dual-teacher distillation followed by contrastive alignment is a design choice. What if the contrastive stage were trained directly on the same data without the distillation stage? The paper argues the distillation stage builds dense spatial features that contrastive training alone would miss, but this is not empirically verified.
Toolchain expansion contribution: The ~8× improvement on MMSearch-Plus from GLM-4.6V to GLM-5V-Turbo conflates model improvements (CogViT, MMTP, broader RL) with toolchain expansion (new tools, better framework integration). How much of the improvement comes from the model being smarter vs. the model having access to better tools? A practitioner considering whether to invest in better tools vs. better model training gets no guidance.

What evidence exists in the paper. The only clean ablation is the MMTP design choice comparison (Option 1 vs. Option 3 training loss, Figure 2) on a 0.5B model — which is a small-scale proxy that may not transfer to the full model and does not measure downstream task impact. All other design choices are presented as part of the integrated system without component-level isolation. The RL improvements in Section 2.3 are reported relative to the SFT baseline, but this compares the entire joint RL stage against no RL — not against alternative RL configurations that would isolate the contribution of task breadth, hierarchical structure, or on-policy distillation.

Mitigation status. The paper does not address this limitation. It is presented as a full-system report ("this report summarizes the main improvements behind GLM-5V-Turbo across model design, multimodal training, reinforcement learning, toolchain expansion, and integration with agent frameworks," Section 1) rather than as a controlled scientific study. This is a reasonable choice for a technical report from an industry team — the goal is to describe what was built, not to provide a rigorous component-by-component analysis — but it limits the paper's value as a source of generalizable design principles. The "Design Lenses" (Section 4) attempt to extract general insights ("perception remains foundational," "hierarchical optimization is more efficient," "task specification and verification are critical"), but these insights are based on the authors' development experience, not on controlled experiments that would allow a reader to assess their validity independently. A practitioner who adopts these lenses based on this paper's authority may invest in directions (e.g., building a two-stage vision encoder with dual teachers) that were not actually responsible for the observed gains.

The Hardware, Scale, and Engineering Requirements Are Not Characterized, Making Replication and Cost-Benefit Assessment Impossible

The assumption or constraint. The paper describes a training and deployment system of substantial complexity: a two-stage vision encoder trained on 8 billion image-text pairs, a pipeline-parallel distributed training setup with custom memory management and topology-aware partitioning, joint RL over 30+ task categories with an independent multi-verifier reward system, integration with multiple external agent frameworks, and a multimodal toolchain. All of this is presented without any quantification of the resources required — no GPU count, no GPU-hours, no training wall-clock time, no inference latency, no throughput numbers, no model parameter count.

The consequence. A practitioner evaluating whether to adopt this approach has no basis for cost-benefit analysis. They cannot answer basic deployment questions:

How many GPUs are needed to train CogViT? The two-stage recipe requires distillation from two frozen teacher models (SigLIP2 and DINOv3) plus the student ViT — potentially 3-4× the GPU memory of single-model training. The contrastive stage uses a 64K batch size with SigLIP loss — this requires distributed training across many GPUs, but how many?
What is the inference cost of multimodal context? The paper identifies long-horizon multimodal context management as a core bottleneck (Section 6): "images and especially videos consume context budget much more aggressively, making them expensive to retain over long trajectories." This directly affects deployment cost — if each GUI agent step requires processing a new screenshot (potentially thousands of visual tokens), and a task requires 50 steps, the total inference cost per task could be substantial. No latency or throughput numbers are provided for any task.
How much does the RL stage cost? Joint RL over 30+ task categories with model-based judges invoked through APIs suggests significant computational overhead — but no numbers. The infrastructure optimizations in Section 2.4 (asynchronous reward evaluation, reference model CPU residency, early-abort modes) are described as efficiency improvements, but efficiency relative to what baseline? A reader cannot determine whether the infrastructure work reduced training cost by 10% or by 10×.
What is the model size? The paper does not report the number of parameters for GLM-5V-Turbo, the CogViT encoder, or the MMTP heads. Without this, comparisons to other models (Claude Opus 4.6, Kimi K-2.5) are ambiguous — is GLM-5V-Turbo outperforming these models at similar scale, or is it a much larger model whose performance advantage is primarily a scale advantage?

The only concrete resource number in the entire paper is the 7 GB reduction in GPU communication buffer overhead from moving Python objects to the CPU communication path (Section 2.4). This is a useful engineering detail but provides no basis for estimating total resource requirements.

What evidence exists in the paper. The paper provides qualitative descriptions of the training infrastructure (Section 2.4) and includes a figure showing memory management strategies, but no quantitative characterization. The paper positions itself as a "report" summarizing the development of GLM-5V-Turbo (Section 1), which implies a different standard of evidence than a research paper — but even for a technical report, the absence of scale information is notable. The benchmark results (Section 5) are quantitative, but the cost of achieving those results is entirely opaque.

Mitigation status. The paper does not address this. It does not provide a "compute budget" appendix, a model card with parameter counts, or any resource characterization. This is a structural limitation of the paper — it reports what was achieved but not at what cost. For an academic research paper, this might be considered a significant omission; for an industry technical report, it is more common but still limits the paper's usefulness for practitioners making resource allocation decisions. The open-source release of tools and skills (Section 3.5, with GitHub repositories) partially mitigates this for deployment — practitioners can test inference costs themselves — but the training cost remains entirely uncharacterized, which matters for teams considering whether to replicate the training approach rather than just use the released model.

The Model-Harness Co-Evolution Insight Is Profound but Undermines the Paper's Own Evaluation Claims

The assumption or constraint. Section 6 argues that "the effective capability boundary is no longer determined by the model alone, but jointly shaped by the model and the harness around it... the same model may behave very differently under different decomposition strategies, tool-use policies, memory designs, or verification workflows." This is presented as a challenge for the field — and the paper's own conclusion — but it also creates a fundamental tension with the paper's evaluation methodology.

The consequence. If model capability is jointly determined by the model and the harness, then benchmark scores that result from a specific harness configuration do not cleanly measure "model capability" — they measure the capability of a specific model-harness combination. The paper reports strong results on framework-based evaluations: 87.0/80.7 on PinchBench, 57.7/75.0 on ClawEval, 57.6 on ZClawBench, all evaluated within the Claw agent framework (Section 5, Figure 5). But these results depend on:

How well the Claw framework's action space matches GLM-5V-Turbo's capabilities.
What memory mechanisms the framework uses (Section 6 notes these are "fundamentally text-centric").
How tool calls are routed and executed.
What verification and error-recovery loops are in place.

A different harness configuration — even with the same model — could produce substantially different scores. This means the paper's own evaluation results are not pure measurements of model capability but of a specific model-harness integration. The paper's claim that "these results provide further evidence that the model's multimodal capability is not limited to isolated benchmark gains, but carries over to realistic end-to-end agent execution" (Section 5) simultaneously acknowledges and understates the problem: the results carry over within the specific Claw harness configuration, not necessarily to other harness configurations or to deployment environments with different tooling, memory management, or error handling.

This also makes comparison to other models (Claude Opus 4.6, Kimi K-2.5) problematic unless those models were evaluated with identical harness configurations. The paper does not specify whether comparison models were evaluated under the same framework conditions — if Claude Opus 4.6 was evaluated with a different harness (e.g., Anthropic's native tool-use interface rather than the Claw framework), the comparison conflates model differences with harness differences.

What evidence exists in the paper. The paper itself provides the evidence for this limitation, in Section 6: "the same model may behave very differently under different decomposition strategies, tool-use policies, memory designs, or verification workflows; conversely, what appears to be a model limitation may sometimes reflect a poor harness choice instead." This is an accurate diagnosis of the problem. The dual scores on PinchBench (87.0/80.7) and ClawEval (57.7/75.0) — two numbers for each benchmark with no explanation of what distinguishes them — may in fact be evidence of exactly this sensitivity: different harness configurations producing different results for the same model on the same task. Without an explanation of the dual scores, this remains speculative, but it is consistent with the paper's own diagnosis.

Mitigation status. The paper acknowledges this limitation in Section 6 but does not attempt to mitigate it in the evaluation. The mitigation would require evaluating the model under multiple harness configurations, reporting variance across configurations, and comparing against other models under identical configurations — a substantial additional experimental burden that the paper does not undertake. The open-source release of official skills and the unified master skill (Section 3.5) is a partial mitigation for deployment: practitioners can test the model with their own harness configurations. But for the paper's evaluation claims, the limitation remains: the reported numbers are specific to the (unspecified) harness configurations used, and the paper's own framework predicts they may not generalize to other configurations. This is not a failure of the paper — it is a consequence of the paper's own insight being correct, and the evaluation methodology not yet catching up to that insight.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents a conceptual reframing rather than a paradigm shift or incremental refinement. It does not introduce a single breakthrough technique that solves a previously intractable problem — instead, it demonstrates that building a capable multimodal agent requires coordinated advances across architecture, training, infrastructure, toolchain, and evaluation simultaneously, and that the integration of these components is the primary intellectual contribution. The specific findings (CogViT's two-stage training, the <|image|> placeholder in MMTP, joint RL over 30+ task categories, topology-aware visual input partitioning) are individually modest, but collectively they constitute an argument about how agentic model development should be organized.

The paper changes the landscape in three ways:

First, it shifts the burden of proof for what counts as "building a capable multimodal agent." Prior to this work, the dominant development paradigm treated models as self-contained artifacts — design an architecture, train on data, benchmark outputs, declare victory. Tools and frameworks were afterthoughts bolted onto a finished model. This paper argues, through its structure and results, that this paradigm is insufficient: the model cannot be separated from the harness, the training objective from the infrastructure, or the evaluation from the verification design. A paper that reports strong VQA or captioning scores but cannot demonstrate that the model functions as a controller in a realistic agent framework (as GLM-5V-Turbo does with Claude Code, AutoClaw, and OpenClaw) is now making a weaker claim about agentic capability. This is not a methodological contribution per se, but a redefinition of what evidence is needed to claim progress toward multimodal agents.

Second, it provides a diagnostic for why prior multimodal agent efforts have underperformed. The paper's four "Design Lenses" (Section 4) function as a retrospective diagnosis of common failure modes:

Lens 1 (perception as bottleneck): Explains why models with strong reasoning benchmarks fail on GUI tasks — they do not see accurately enough for their reasoning to matter. This reconciles the contradiction between high VLM benchmark scores and poor agentic performance.
Lens 2 (hierarchical optimization): Explains why end-to-end training on agent trajectories often fails or plateaus — lower-level perceptual and action primitives are not reliable enough for higher-level optimization to be stable. This suggests that single-task agent training approaches (e.g., training only on WebVoyager trajectories) are structurally disadvantaged compared to hierarchical approaches.
Lens 3 (task specification and verification): Explains why many agent benchmarks produce noisy or irreproducible results — the tasks are underspecified and the verification is unreliable, so the evaluation signal is too weak for optimization. This reframes benchmark design as a first-class research problem rather than an evaluation afterthought.
Lens 4 (model-harness co-evolution): Explains why models that excel in isolated evaluations fail in deployment — the deployment harness imposes different constraints than the evaluation harness, and model capabilities that depended on the evaluation harness do not transfer.

These lenses do not resolve prior contradictions with new data; rather, they provide a conceptual framework that makes sense of contradictory findings. A researcher who previously observed that "better perception encoders didn't improve my agent's task success" can now hypothesize: the bottleneck was not perception quality per se, but the absence of hierarchical training (the model had good perception but couldn't chain it into reliable actions) or the absence of perception-aware RL (the model could see but wasn't rewarded for seeing accurately).

Third, it redirects research investment toward integration-heavy development rather than component-level optimization. If the paper's central claim is correct — that agentic capability emerges from the integrated model-harness-tool-evaluation system, not from any single component — then the highest-return research investments are in:

Infrastructure for multi-task multimodal RL (Section 2.4) rather than better RL algorithms. The paper's improvements came from making RL feasible at scale (decoupled pipeline, memory management, load balancing) rather than from a novel RL formulation.
Verifier design and task specification (Section 4, Lens 3) rather than better model architectures. The Vision2Web benchmark's workflow-based verification is a methodology contribution that may transfer to other agent domains more readily than CogViT's specific encoder architecture.
Toolchain and framework co-design (Sections 3.1–3.2) rather than isolated model capability improvements. The ~8× improvement on MMSearch-Plus from GLM-4.6V to GLM-5V-Turbo is attributed partly to toolchain expansion, suggesting that capability gains from better tools may be larger than gains from better models for certain task families.
Comprehensive evaluation suites that span perception, reasoning, and execution (Section 5) rather than narrow benchmarks. The paper's evaluation strategy — covering multimodal coding, tool use, GUI agents, text-only coding, and framework-based execution — implicitly argues that any single benchmark category gives an incomplete picture.

Conversely, this paper makes certain research directions less attractive: developing ever-more-sophisticated vision encoders without corresponding investment in how those encoders are trained jointly with the LLM (since CogViT's two-stage recipe suggests the training integration matters as much as the architecture), designing complex planning algorithms without first ensuring perceptual reliability (since Lens 1 argues perception is the binding constraint), and training on narrow agent benchmarks without verifying that capabilities transfer to real frameworks (since Section 6 argues the harness co-determines capability).

The most significant long-term implication is that the paper's "model-harness co-evolution" diagnosis (Section 6) may be self-undermining for static benchmarks as evaluation tools. If the model and the harness jointly determine capability, and the harness evolves as the model improves, then a static benchmark — which fixes the harness — becomes increasingly misaligned with deployment capability as models advance. The paper does not resolve this tension, but it makes it visible: the field needs evaluation methodologies that can track capability despite evolving harnesses, or it will continue to see gaps between benchmark performance and deployment performance. This is not a problem the paper solves, but a problem the paper makes it impossible to ignore.

Follow-Up Research This Work Enables

1. Ablation of the 30-task RL distribution to identify which task categories drive cross-domain transfer. The paper reports simultaneous improvements across perception, reasoning, and agentic execution from joint RL (Section 2.3) and claims that "multi-task RL tends to show weaker interference across domains" compared to SFT. However, it does not establish which task categories are responsible for which transfers. A targeted follow-up would systematically remove task categories from the RL mixture — e.g., train one variant without perception tasks (no RefCOCO, PointBench, OCRBench), one without reasoning tasks (no MMMU, MathVista, LogicVista), one without agentic tasks (no OSWorld, MMSearch) — and measure the impact on each category. The key measurement would be: does removing perception tasks degrade GUI-agent performance (supporting the Lens 1 claim that perception is foundational), or do GUI-agent tasks improve independently? Does removing reasoning tasks degrade tool-use planning (supporting the "transfer of thinking patterns" claim), or is planning capability learned primarily from agentic tasks themselves? A strong result would identify specific cross-domain dependencies (e.g., "grounding training is necessary for >50% of the GUI-agent improvement") and would provide actionable guidance for resource allocation: which task categories can be dropped with minimal impact, and which are essential for the observed gains.

2. Head-to-head comparison of hierarchical optimization vs. end-to-end training with equivalent total data and compute. Lens 2 (Section 4) argues that "agent capability is developed more effectively when optimization is distributed across multiple levels of the capability hierarchy" — but this is presented as a qualitative insight, not a controlled finding. A rigorous test would construct two training pipelines with identical data sources and compute budgets: one that distributes training across the hierarchy (element perception → grounding → single-step actions → trajectory-level actions) and one that trains only on end-to-end trajectories (the same trajectory data, but without the lower-level SFT and RL stages). The models would be compared on the same GUI agent benchmarks (OSWorld, AndroidWorld, WebVoyager). The specific hypothesis is that hierarchical training achieves higher task success for the same total training FLOPs, and the effect should be largest on long-horizon tasks where lower-level errors compound. If hierarchical training shows no advantage over end-to-end training, Lens 2 collapses — agent capability is not "more efficiently" built through hierarchy, and the paper's development strategy was incidental rather than essential. This would be an important negative result that would redirect research away from hierarchical curriculum design and toward better end-to-end RL methods.

3. Difficulty-stratified analysis of ImageMining and agentic benchmarks to characterize the capability ceiling. The paper reports aggregate scores (ImageMining 30.7, MMSearch-Plus 30.0, BrowseComp-VL 51.9) but provides no analysis of which tasks the model succeeds on vs. fails on. A difficulty-stratified breakdown — analogous to the five-quintile analysis from the example paper — would categorize ImageMining's 217 test cases by some measure of difficulty (number of required tool calls, length of reasoning chain, presence of the WEB_VISUAL constraint, domain category) and report performance per bin. The key questions: is the 30.7 average composed of >70% success on easy tasks (Universal Recognition) and near-zero on hard tasks (Event Reasoning, Spatio-Temporal Reasoning), or is performance uniformly modest across categories? Are failures concentrated in tasks requiring >5 tool calls, suggesting a planning horizon limitation? Do tasks with the WEB_VISUAL constraint (forcing visual transitions) show significantly lower performance than tasks where visual search could be bypassed by parametric knowledge? This analysis would transform ImageMining from a single-number benchmark into a diagnostic tool that identifies specific capability gaps, and would guide whether future work should focus on perception (if recognition tasks fail), planning (if multi-step tasks fail), or tool-use precision (if cropping/magnification tasks fail).

4. Controlled experiment on the contribution of toolchain expansion to agentic performance, isolating model improvement from tool improvement. The paper reports an ~8× improvement on MMSearch-Plus from GLM-4.6V to GLM-5V-Turbo (Section 3.1) and attributes this partly to toolchain expansion — but this conflates model upgrades (CogViT, MMTP, broader RL) with tool upgrades (new multimodal search, browser, and image processing tools). A clean follow-up would evaluate both GLM-4.6V and GLM-5V-Turbo on MMSearch-Plus with (a) the old toolchain, (b) the new toolchain, and (c) a matched toolchain where both models have access to the same tools. This would decompose the 8× improvement into a "model capability" component (how much better is GLM-5V-Turbo at using the same tools?) and a "tool capability" component (how much does access to better tools improve performance even with the same model?). A surprising result — e.g., that >50% of the improvement comes from toolchain expansion rather than model improvement — would redirect investment toward tool design rather than model training for agentic tasks. Conversely, if the model improvement dominates, toolchain expansion is secondary and the paper's emphasis on tools (Section 3.1) may be overstated. This experiment matters for resource allocation: improving tools is typically cheaper than training better models, so knowing the relative contribution guides cost-effective development.

5. Measurement of catastrophic forgetting in text-only capabilities beyond coding benchmarks. The paper claims that GLM-5V-Turbo "preserves the coding capability of its language-only base model" and shows three CC-Bench-V2 scores (Section 5), but does not evaluate on standard NLP benchmarks (MMLU, GSM8K, HumanEval, HellaSwag, ARC). A comprehensive follow-up would benchmark GLM-5V-Turbo against GLM-5-Turbo across a standard suite of 10+ text-only tasks spanning knowledge, reasoning, math, code generation (outside Claude Code), and commonsense. The specific hypothesis to test: does multimodal training cause narrow degradation in capabilities not represented in the multimodal training data, or is the degradation (if any) broad and uniform? A finding of selective degradation (e.g., coding preserved but factual knowledge declined) would suggest that the multimodal training data inadvertently overwrote certain parametric knowledge, and would motivate techniques to isolate text-only capabilities during multimodal training (e.g., replay buffers, elastic weight consolidation). A finding of no degradation across any benchmark would strengthen the paper's claim substantially, but would require explanation: how did the model add multimodal capability with zero interference? Understanding this mechanism — whether through the MMTP placeholder design, the broad RL distribution, or some other factor — would be a significant contribution to multimodal training methodology.

6. Cross-framework robustness study to test the model-harness co-evolution diagnosis. Section 6 argues that "the same model may behave very differently under different decomposition strategies, tool-use policies, memory designs, or verification workflows" — a claim with direct implications for how the paper's own results should be interpreted. A rigorous test would evaluate GLM-5V-Turbo on the same set of agent tasks (e.g., the PinchBench and ClawEval tasks) under 3-5 systematically varied harness configurations: different memory management strategies (full history vs. summarized history vs. no history), different tool routing policies (model decides vs. fixed pipeline), different error recovery mechanisms (retry on failure vs. single attempt vs. human-in-the-loop). The key measurement is the variance in task success rate across harness configurations. If the paper's diagnosis is correct, variance should be substantial — the same model should show significantly different performance depending on the harness. A finding of low variance would challenge the diagnosis and suggest that the model's capabilities are more robust to harness variation than the paper claims. A finding of high variance would have two implications: (a) it would validate the paper's core insight that model capability is not separable from harness design, and (b) it would make the paper's own evaluation results harder to interpret, since they represent a single (unspecified) harness configuration. This tension — that the paper's insight undermines its own evaluation methodology — is the deepest unresolved issue in the work, and a cross-framework robustness study would quantify how serious the problem is.

Practical Applications and Downstream Use Cases

1. Automated website reproduction and frontend development from visual specifications. The combination of GLM-5V-Turbo's Design2Code performance (94.8, Section 5) and its integration with Claude Code for web replication (Section 3.1, with the official glmv-web-replication skill) enables a concrete deployment scenario: a designer provides a mockup, screenshot, or reference URL, and the model autonomously explores the target (using multimodal GUI interaction to navigate pages, collect assets, and understand interaction flows), then generates production-ready HTML/CSS that reproduces the design with high visual fidelity. The paper's demonstration (Appendix A, Figures 7–13) shows this working for e-commerce sites, mobile app interfaces, and research paper websites. The specific benefit is that the model handles both the perceptual task (understanding layout, spatial relationships, visual style from screenshots) and the coding task (generating semantically correct, executable code) in a unified pipeline, eliminating the traditional handoff between designer inspection and developer implementation. The 94.8 Design2Code score and the Vision2Web workflow-based verification (Section 4) provide evidence that the pipeline can achieve functional correctness and visual consistency at a level approaching manual implementation for standard web interfaces.

2. Vision-enabled deep research with automated evidence gathering from heterogeneous sources. The multimodal deep research workflow (Section 3.4, Figure 3) and the ImageMining benchmark (Section 3.3) demonstrate a deployment scenario where the model autonomously conducts research that requires processing visual evidence — not just reading text. A user provides an open-ended objective (e.g., "compare OpenClaw and Hermes agent systems"), and the model iteratively searches the web, reads visually rich pages (extracting information from charts, tables, screenshots, and figures, not just text paragraphs), crops and magnifies relevant visual details, cross-references findings, and synthesizes an interleaved text-image report or slide deck. The specific benefit is that the model accesses evidence that text-only research pipelines discard — the visual content of webpages (slides, figures, layouts) that often contains the most valuable information for comparative analysis, technical documentation, and literature reviews. The 30.7 ImageMining score and 51.9 BrowseComp-VL score indicate that this capability is still limited (the model fails on ~50-70% of complex visual search tasks), but the demonstration cases in Appendix A (Figures 14–15) show that for well-scoped research questions, the pipeline can produce professional-quality outputs that integrate visual and textual evidence.

3. GUI automation for mobile and desktop task completion. GLM-5V-Turbo's strong GUI agent performance (75.7 on AndroidWorld, 62.3 on OSWorld, Section 5) and integration with AutoClaw for browser-based automation (Section 3.2) enables deployment as a vision-language controller for task automation on real devices and operating systems. A concrete scenario: a user describes a multi-step task in natural language (e.g., "find flights from Beijing to Shanghai on May 1-5, book the cheapest one that departs before 10 AM, and send the confirmation to my email"), and the model perceives the screen state at each step, decides which UI elements to interact with, executes actions through the framework, observes the results, and adapts its plan if the interface changes or an action fails. The specific benefit is that the model handles visual interface variation — different app designs, unexpected popups, layout changes — that break scripted automation approaches, because it natively perceives the interface rather than relying on brittle element selectors or accessibility APIs. The 75.7 AndroidWorld score and 62.3 OSWorld score suggest that for moderately complex tasks (5-15 steps), the model succeeds more often than it fails, making it viable for supervised automation where a human reviews the result, if not for fully autonomous deployment.

4. Document-grounded content creation with visual-textual integrity. The paper's demonstrations of automatic slide deck generation from research papers (Appendix A, Figure 13), technical blog creation from academic papers (Figure 3b), and document-based writing with image preservation (Figure 15) enable a practical deployment scenario: users provide complex source materials (PDFs, papers, reports), and the model reorganizes them into presentation formats (slides, blogs, interleaved reports) while preserving the relationship between textual claims and supporting visual evidence. This is distinct from standard summarization because the model must decide which figures, tables, and charts to include, where to place them relative to text, and how to crop or annotate them for the target format — tasks that require joint visual-textual reasoning. The specific benefit is that the output maintains "the synergy between textual conclusions and supporting visual evidence" (Section 3.4), which is lost when text is extracted and summarized independently of figures. The official skills for PDF-to-Web and PDF-to-PPT (Section 3.5, Table 2) provide pre-configured workflows for this deployment scenario, and the demonstration cases show outputs of professional quality for research paper and textbook inputs.

When to Prefer This Method

The paper does not articulate a clear tradeoff decision rule positioning GLM-5V-Turbo's approach against named alternatives with specified conditions. It presents itself as a full-system development report rather than as a method competing against other methods, and it does not state conditions under which a practitioner should adopt the CogViT + MMTP + joint RL approach over, for example, a standard VLM architecture with single-task RL, or a text-only model with API-based vision tools, or a different multimodal model like Claude Opus 4.6 or Kimi K-2.5. The comparisons to other models (Section 5) are performance benchmarks, not tradeoff analyses — they report that GLM-5V-Turbo achieves certain scores, not when its architecture or training methodology is preferable to alternatives. The Design Lenses (Section 4) describe insights from the development process but do not articulate conditions under which those insights apply vs. don't apply.

Because the paper does not provide this analysis, a formulaic "Prefer A when... Prefer B when..." matrix would be fabricated rather than extracted from the paper's evidence and arguments. The closest the paper comes to comparative guidance is implicit: the hierarchical optimization strategy (Lens 2) is presented as more efficient than end-to-end training, suggesting that teams building agentic models should prefer hierarchical over flat training when resources are limited. The model-harness co-evolution insight (Section 6) implies that teams deploying models as agents should invest in harness co-design rather than treating the model as a standalone component. But neither of these is articulated with the specificity — conditions, thresholds, benchmarks — that would constitute a clear decision rule grounded in the paper's results.