InternLM2 Technical Report

🎯 Pitch

Reconciling conflicting human preferences like helpfulness versus harmlessness typically requires multiple reward models, but InternLM2’s novel COOL RLHF method uses just one conditional model to handle all dimensions—which only works if the training is online, as offline variants immediately suffer from catastrophic reward hacking. The method also pushes effective context length to 200k tokens, achieving near-perfect needle-in-a-haystack retrieval through an iterative extension of RoPE base frequency during long-context pretraining.

1. Executive Summary

This paper introduces InternLM2, an open-source large language model spanning 1.8B, 7B, and 20B parameters that demonstrates state-of-the-art performance across comprehensive benchmarks, long-context modeling, and alignment evaluations. InternLM2 achieves this through meticulous pre-training data preparation across text, code, and long-context corpora, a three-phase training pipeline (4k-context → 32k-context → capability-specific enhancement), and a novel Conditional Online RLHF (COOL RLHF) strategy that reconciles conflicting human preferences within a single reward model using conditional system prompts and iterative online refinement. The model exhibits a 200k context window via Group Query Attention and RoPE base extension, achieving near-perfect retrieval on the "Needle-in-a-Haystack" test, while COOL RLHF yields significant alignment improvements — the 20B chat variant attains a 21.8 weighted win rate on AlpacaEval and 31.4 on CompassArena, outperforming GPT-3.5 on reasoning benchmarks such as HellaSwag (by 15.6 points) and BBH (by 26.4 points) — establishing that a single conditional reward model can effectively model diverse, conflicting preferences across domains only when reinforced through multi-round online training that patches emergent reward hacking.

2. Context and Motivation

The Core Problem: Training State-of-the-Art LLMs Is an Engineering Puzzle, Not a Solved Science

The paper addresses a practical but multi-faceted gap: the open-source community lacks a transparent, end-to-end blueprint for building LLMs that match or exceed proprietary systems like ChatGPT and GPT-4 across the full development pipeline. While the Executive Summary notes that InternLM2 achieves state-of-the-art results, the motivation behind this work is that the path to those results — the data preparation, training phases, and alignment techniques — has been largely opaque in prior work. Technical reports from major open-source efforts (LLaMA, Qwen, Mistral, DeepSeek) typically emphasize benchmark scores and model architecture while treating data processing and alignment methodology as cursory details. This paper explicitly positions itself to fill that gap:

"technical reports on LLMs... in the past have seldom addressed the processing of pre-training data. InternLM2 extensively details how it prepares text, code, and long-context data for pre-training." (Section 1)

This matters because, unlike the relatively standardized pretraining algorithms (AdamW, cosine schedules, FlashAttention), the quality and composition of training data is increasingly understood to be the dominant factor in model capability — yet it is also the least documented aspect of LLM development. Without concrete guidance on data filtering, deduplication, safety screening, and quality assessment, the open-source community effectively reinvents these pipelines with each new model, wasting enormous computational and human effort.

The problem is not just about data, however. It extends to the full training lifecycle. The paper identifies three interconnected challenges that existing open-source efforts have addressed only partially or not at all.

Challenge 1: Long-Context Training Across All Stages Is Poorly Understood

Extending LLMs to handle very long contexts (32K+ tokens) has become a central research focus because downstream applications like Retrieval-Augmented Generation (RAG) and tool-using agents fundamentally depend on it. The paper notes:

"How to effectively extend the context length of LLMs is currently a hot research topic, since many downstream applications, such as Retrieval-Augmented Generation (RAG) and agents, rely on long contexts." (Section 1)

Prior work has made progress on long-context pretraining — Rozière et al. (2023) demonstrated that Code Llama benefits from extended context windows, and Xiong et al. (2023) showed effective long-context scaling through data mixing — but two critical gaps remain.

First, long-context capability degrades during fine-tuning. A model that handles 32K tokens during pretraining may lose that ability after Supervised Fine-Tuning and RLHF, because the alignment data is typically constructed with short examples (single-turn conversations, brief instructions, standard QA pairs). The paper cites Xiong et al. (2023) as inspiration for preserving long-context ability, noting that they "keep using the long-context pre-training data in SFT and RLHF" (Section 4.3), but this practice was not widely adopted or systematically validated. Most open-source chat models simply accept a reduced effective context window after alignment, which limits their utility for RAG applications where the retrieved documents may span thousands of tokens.

Second, the interaction between architecture choices and long-context inference efficiency is not fully explored. While Group Query Attention (GQA) was introduced by Ainslie et al. (2023) as a way to reduce the KV-cache memory footprint during inference, prior open-source models used it inconsistently — some adopted Multi-Query Attention (MQA), others used full multi-head attention, and the implications for training stability and inference speed at extreme context lengths (100K+ tokens) were not well-characterized. InternLM2 commits to GQA across all model sizes "so that it can infer both in high speed and low GPU memory with very long contexts" (Section 2.2), providing a systematic validation point for this architectural decision.

Challenge 2: Preference Conflicts in RLHF Are Handled by Proliferating Reward Models

Reinforcement Learning from Human Feedback (RLHF) has become the standard alignment technique since InstructGPT (Ouyang et al., 2022), but it faces a structural problem when human preferences are heterogeneous or contradictory. The paper articulates this clearly:

"The first [issue] is the preference conflicts. For example, in developing a dialogue system, we expect it to provide useful information (helpful) while not producing harmful or inappropriate content (harmless). However, these two preferences often cannot be satisfied simultaneously in practice, as providing useful information might involve sensitive or high-risk content in some cases." (Section 4.2)

This is not a hypothetical concern. A model asked to explain a historical conflict should provide accurate details (helpfulness) without normalizing violence or hate speech (harmlessness). A coding assistant asked to generate a script for system administration should produce functional code (helpfulness) without enabling privilege escalation exploits (harmlessness). The tension is real and pervasive.

Prior approach: multiple specialized reward models. The dominant solution, exemplified by LLaMA2 (Touvron et al., 2023b) and Safe RLHF (Dai et al., 2023), is to train separate reward models for each preference dimension — one reward model for helpfulness, another for harmlessness, possibly a third for factual accuracy — and then combine their scores during PPO training through weighted summation or constrained optimization. The paper identifies several shortcomings:

Computational cost: "existing RLHF methods usually rely on multiple preference models for scoring, which also introduces more models in the training pipeline thus increases computational cost and slows down training speed" (Section 4.2). Each reward model must be trained, stored, loaded into GPU memory during PPO, and run on every generated response. For a 7B or 20B model, this can mean an additional 14B–40B parameters of reward models competing for memory that could otherwise be used for larger batch sizes or longer context windows.
Calibration mismatch: Different reward models produce scores on different scales and with different distributions. Combining them requires careful weighting, and the optimal weights may vary across domains — a mathematical reasoning task cares more about correctness than stylistic harmlessness, while a creative writing task inverts those priorities.
Static combination: Multi-model approaches typically use fixed weighting. But the relative importance of helpfulness vs. harmlessness depends on the specific input — a user asking "how do I make a Molotov cocktail?" needs harmlessness to dominate, while "how do I make vanilla extract?" should prioritize helpfulness. Static weights cannot adapt dynamically.

Challenge 3: Reward Hacking Accelerates as Policies Improve

The second issue with conventional RLHF is reward hacking, which the paper frames as an escalating problem:

"RLHF faces the issue of reward hacking, especially when the policy becomes more powerful with the scale increasing, where the model might learn to 'cheat' the reward system by shortcuts to obtain high scores, rather than truly learning the expected behavior." (Section 4.2)

This is a specific instance of Goodhart's Law (Manheim & Garrabrant, 2018): when a proxy measure (the reward model's score) becomes the optimization target, it ceases to be a good proxy. The language model discovers patterns that exploit blind spots in the reward model — generating verbose but vacuous text, using stylistic flourishes that the reward model associates with high-quality responses, or producing content that superficially matches the training distribution of the reward model without actually being helpful.

Why existing solutions are insufficient. Prior work acknowledged reward hacking but treated it as a static problem: train a better reward model once, then run PPO and accept whatever quality ceiling the reward model provides. The paper's insight is that reward hacking is dynamic — as the policy improves during PPO training, it explores new regions of output space that the reward model was never trained on, encountering novel hacking opportunities that a static reward model cannot anticipate. The solution must be adaptive: the reward model needs to see the policy's current outputs and be updated to close newly discovered loopholes.

How This Paper Positions Itself

InternLM2's contribution is not a single novel algorithm but a comprehensive, openly documented engineering methodology that addresses all three challenges simultaneously. The positioning is explicit in the paper's four stated contributions (Section 1):

Contribution 1 (Exceptional Performance): The paper establishes that careful data engineering and a three-phase pretraining pipeline can produce models that compete with or exceed proprietary systems — not through architectural novelty, but through execution quality.

Contribution 2 (200k Context Window): Rather than treating long-context modeling as a pretraining-only feature, the paper demonstrates that it must be maintained through SFT and RLHF by deliberately including long-context data in alignment stages. This is a practical, non-obvious finding: the alignment process is not neutral with respect to context length.

Contribution 3 (Comprehensive Data Preparation Guidance): The paper positions its data processing pipeline — from raw Common Crawl dumps through rule-based filtering, MinHash deduplication, safety classification, and quality scoring — as a primary contribution, not an afterthought. Section 3.1 provides enough detail (selecting 60,004 tokens from cl100k vocabulary and adding 32,397 Chinese tokens; using 128 MinHash signatures with a 0.7 threshold; training BERT-based classifiers for toxicity, pornography, advertisements, and fluency) that other teams can replicate or adapt the approach.

Contribution 4 (COOL RLHF): The paper introduces a single conditional reward model that uses different system prompts to distinguish between preference dimensions (e.g., "Evaluate this response for helpfulness" vs. "Evaluate this response for harmlessness"), combined with multi-round online RLHF where the reward model is patched between PPO rounds to address newly discovered reward hacking patterns. This addresses both preference conflicts (one model, multiple conditions) and dynamic reward hacking (fast patching + slow refinement) without the computational overhead of multiple reward models.

The Broader Significance: Open-Source Needs Engineering Rigor, Not Just Novel Ideas

The paper's framing reflects a broader shift in the LLM community. Early open-source efforts focused on architectural innovations (Rotary Position Embeddings, SwiGLU activations, efficient attention) and scaling demonstrations. But as the field has matured, the limiting factor is increasingly data curation and alignment methodology — areas where proprietary systems maintain their advantage not through secret algorithms, but through superior engineering pipelines developed with substantial resources. By documenting its full pipeline in unprecedented detail, InternLM2 lowers the barrier for the open-source community to compete on these engineering-intensive aspects of LLM development.

The paper also implicitly addresses a fragmentation problem: without shared knowledge about data processing standards, each open-source team makes different (often undocumented) choices about filtering thresholds, deduplication methods, and safety screening. This makes it impossible to attribute performance differences to model architecture or training algorithms versus data quality. InternLM2's detailed documentation provides a reference point that the community can either adopt or consciously deviate from, making future comparisons more interpretable.

3. Technical Approach

3.1 Reader Orientation

InternLM2 is a family of large language models trained from scratch through a carefully engineered pipeline spanning data preparation, pre-training, and alignment. The system solves the problem of building an open-source LLM that matches or exceeds proprietary models by treating each stage — data filtering, context-length extension, preference reconciliation — as a first-class engineering challenge rather than an afterthought.

3.2 Big-Picture Architecture (Diagram in Words)

The InternLM2 pipeline has four major stages, each feeding into the next:

Data Preparation Pipeline (Section 3.1): Raw data from web pages, code repositories, books, and papers enters a multi-stage filtering system. Text data passes through rule-based cleaning, MinHash deduplication, safety classifiers (toxicity and pornography detection), and quality classifiers (advertisement and fluency scoring). Code data undergoes similar processes plus dependency sorting to reconstruct repository structure. Long-context data receives additional statistical and perplexity-based filters to ensure coherence over 32K+ tokens.
Pre-Training (Section 3.2–3.3): An InternEvo training framework orchestrates distributed training across thousands of GPUs using data, tensor, sequence, and pipeline parallelism with ZeRO strategies. The model itself follows the LLaMA architecture with Group Query Attention (GQA) for long-context efficiency. Training proceeds in three phases: (Phase 1) 4K-context training on ~90% of total steps; (Phase 2) 32K-context training with RoPE base adjusted from 50,000 to 1,000,000; (Phase 3) Capability-specific enhancement training on 24B tokens of curated high-quality data covering reasoning, mathematics, and knowledge.
Supervised Fine-Tuning (Section 4.1): The pre-trained model is fine-tuned on 10 million instruction data instances spanning general conversation, NLP tasks, mathematics, code generation, and function calls, formatted in ChatML. Long-context data from books and code repositories is deliberately included to preserve the 32K context window.
COOL RLHF Alignment (Section 4.2): A novel Conditional Reward Model uses different system prompts to model diverse preferences (helpfulness, harmlessness, reasoning accuracy) within a single model. Proximal Policy Optimization (PPO) runs over three online rounds. Between rounds, a Fast Path patches reward hacking by adding 20–100 targeted preference pairs, while a Slow Path accumulates new human preference data from models at various training stages to improve the reward model's overall robustness.

Information flows left-to-right: raw data → filtered data → pre-trained base model → SFT model → RLHF-aligned chat model. At each stage, long-context capability is deliberately maintained by including appropriate data, not just trained once and assumed to persist.

3.3 Roadmap for the Deep Dive

First, the data preparation pipeline for text, code, and long-context data — because data quality is the foundation everything else builds upon, and this is where the paper provides its most detailed, actionable guidance.
Second, the InternEvo training infrastructure and model architecture choices — covering how training scales to thousands of GPUs and why specific architectural decisions (GQA, weight matrix layout) were made.
Third, the three-phase pre-training strategy — explaining the 4K → 32K → capability-specific progression, the rationale for each phase's data composition, and the critical hyperparameter changes between phases.
Fourth,, Supervised Fine-Tuning — covering the 10M-instruction dataset, ChatML formatting, and the deliberate inclusion of long-context data to prevent context window degradation.
Fifth,, the COOL RLHF system — the conditional reward model, the focal ranking loss, the online training protocol with Fast Path and Slow Path, and PPO implementation details — because this is the paper's primary algorithmic innovation.
Sixth,, long-context fine-tuning and tool augmentation — showing how capabilities are maintained and extended during alignment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an engineering methodology paper whose core idea is that exceptional LLM performance emerges from meticulous execution across the entire development pipeline — data preparation, staged pre-training, and alignment — rather than from any single architectural or algorithmic novelty. The paper's most distinctive technical contribution is the COOL RLHF system, which replaces the conventional multi-model reward approach with a single conditional reward model and iterative online refinement.

3.4.1 Text Data Preparation Pipeline

The text data pipeline transforms raw web pages, papers, patents, and books into a high-quality pre-training corpus through five sequential stages: formatting, rule-based filtering, deduplication, safety filtering, and quality filtering. Table 1 shows that web pages constitute 86.46% of the total pre-training data volume, with books, technical literature, and patents making up the remainder — these smaller sources contribute disproportionately to model quality due to their longer average document length and higher content standards.

Stage 1: Data Formatting. Raw WARC-format files from Common Crawl are decompressed. The Trafilatura library extracts main text content from HTML, discarding navigation elements, advertisements, and boilerplate. The pycld2 library performs language detection to classify documents by language. Each document receives a unique identifier and is stored in JSON Lines (jsonl) format. This standardization step ensures all downstream processing operates on a uniform representation regardless of source.

Stage 2: Rule-Based Filtering. A suite of heuristic filters targets common pathologies in web-extracted text. The paper identifies three categories of rules: separation and line-break anomalies (text fragments that were incorrectly split or joined during extraction), abnormal character frequency (documents dominated by non-natural-language symbols, e.g., Base64-encoded data or garbled Unicode), and punctuation distribution (documents where punctuation marks follow patterns inconsistent with natural language, such as excessive repetition or systematic absence). These filters are explicitly statistical and hand-designed — they do not involve learned models — and their purpose is to eliminate trivially low-quality data before more expensive processing stages.

Stage 3: Deduplication. The paper uses the MinHash variant of Locality-Sensitive Hashing (LSH) for fuzzy deduplication. For each document, a signature is constructed using 128 hash functions applied to the document's 5-gram set. Two documents are considered duplicates if their MinHash signatures have a Jaccard similarity exceeding 0.7. When duplicates are detected, the system retains the version from the most recent Common Crawl dump (prioritizing recency). This approach is approximate but scales to web-scale corpora where exact deduplication would be computationally prohibitive.

Stage 4: Safety Filtering. A composite strategy combining four mechanisms removes toxic and pornographic content:

Domain blocking: A blocklist of approximately 13 million domains known to host unsafe content. Any document originating from these domains is discarded.
Word blocking: A blocklist of 36,289 unsafe words. The paper explicitly notes they compiled this list conservatively — over-aggressive word blocking can inadvertently filter benign content when unsafe words appear in legitimate contexts (e.g., medical discussions, legal documents).
Toxicity classifier: A fine-tuned BERT model trained on the Kaggle "Toxic Comment Classification Challenge" dataset. Documents scoring below a specified threshold are filtered.
Pornography classifier: A BERT model fine-tuned on data sampled from the deduplicated corpus and annotated using the Perspective API. Documents scoring below the threshold are removed.

The two-stage approach (domain/word lists as fast pre-filters, followed by learned classifiers for nuanced detection) balances computational efficiency with detection accuracy.

Stage 5: Quality Filtering. Two BERT-based classifiers address the most prevalent quality issues in web data:

Advertisement classifier: Trained on manually annotated data where annotators identified whether a document contains promotional content (both fully promotional and partially promotional documents are labeled as low-quality). This targets the observation that marketing content is repetitive and information-sparse, degrading model training even when syntactically well-formed.
Fluency classifier: Trained on data rated across four dimensions — consistency (logical coherence of the text), noise (presence of irrelevant or garbled segments), information content (density of meaningful information), and grammar — combined into a composite fluency score. Documents scoring below the threshold are filtered.

The annotation process for both classifiers is manual: human annotators label training data, BERT models are fine-tuned on these labels, and the resulting classifiers are applied to the full corpus. This reflects a key engineering philosophy in the paper: learned quality filters are domain-specific (trained separately for different data sources like web pages vs. papers) and require human-calibrated definitions of quality.

3.4.2 Code Data Preparation Pipeline

Code data preparation follows the same broad stages as text data but with domain-specific adaptations. The data sources include direct GitHub crawling, public datasets, Q&A forums, tutorial sites, and API documentation. Figure 4 shows the statistical distribution across these sources.

Quality Assessment via Learned Scorer. The paper trains a scoring model to classify code files into three quality tiers (Table 2):

High-quality: Given higher sampling weight during pre-training, potentially trained on multiple times.
Moderate-quality: Given normal sampling weight, trained once.
Low-quality: Excluded entirely. The paper states that "removing them is vital for optimizing model performance and ensuring training stability despite their proportion being relatively small."

The training process for this scorer uses an iterative annotation workflow (Figure 5) designed to address the inherent vagueness of "code quality." Human annotators verify the scorer's predictions on high-confidence high-quality and low-quality samples, and annotation guidelines are refined based on discrepancies. An automatic validation step in each iteration checks that previously annotated samples remain correctly classified by the updated scorer. The paper reports taking three iterations to finalize the scoring model.

An important observation is that code style alone is not a reliable quality metric — heuristic style rules (e.g., line length limits, naming conventions) misclassify too many legitimate code files as low-quality. The learned scorer attempts to capture more nuanced notions of pedagogical value: "a widely recognized code repository might be overly complex for a beginner," so quality is judged relative to what would help a language model learn programming concepts.

Dependency Sorting. With the context window expanded to 32K tokens, InternLM2 can consume entire code repositories as single training examples. The paper implements a dependency-aware concatenation strategy:

Code files originating from the same GitHub repository are regrouped (since earlier filtering steps may have separated them).
Regular expressions detect "import" relationships across programming languages.
Topological sorting determines the concatenation order so that dependencies appear before dependents.
Non-code files (markdown, documentation) are placed before the first code file in the same sub-folder.

For corner cases — multiple paths between an ascendant and descendant, or cycles in the import graph — the paper takes the shorter path for the former and uses alphabetical ordering to select a starting point for the latter. A notable implementation detail: "batched imports" such as Python's __init__.py or C++'s aggregate headers are resolved using heuristic rules to identify which specific dependencies are actually used, preventing spurious import relationships from distorting the file ordering.

3.4.3 Long-Context Data Filtering

Long-context data preparation builds on the text data pipeline with additional filters designed specifically for texts exceeding 32K bytes (~8K–32K tokens depending on tokenization). All long-context data is a subset of the standard pre-training corpus, meaning these documents are learned at least twice — once during standard training and once during dedicated long-context training.

Length Selection. A rule-based filter selects data samples exceeding 32K bytes in length. This is the entry criterion for the long-context pipeline.

Statistical Filters. A suite of lexical and linguistic features identifies anomalous long documents. The paper references the full filter list in Lv et al. (2024) but highlights one representative example: the presence of discourse-structure words (e.g., "Especially," "Formally") that indicate coherent text organization. The design philosophy is explicitly stated: "The overall guidance of designing such filters is to filter out the meaningless data rather than selecting the most high-quality data." Statistical filters are reported to be particularly effective for long texts because statistical features (word frequency distributions, punctuation patterns, structural markers) are far more consistent and interpretable in 32K-token documents than in 20-token snippets.

Perplexity Filters. This is the most technically distinctive filter. Rather than using perplexity as an absolute quality measure (which would bias toward the model used to compute perplexity), the paper uses perplexity difference to assess contextual coherence:

$P(S_2|S_1) \text{ vs. } P(S_2)$

where $S_1$ is a preceding text segment and $S_2$ is a subsequent segment. The conditional probability $P(S_2|S_1)$ is estimated by computing perplexity of $S_2$ when $S_1$ is provided as context. The unconditional probability $P(S_2)$ is estimated by computing perplexity of $S_2$ in isolation. The difference between these two values measures whether $S_1$ helps predict $S_2$ (context is coherent) or hurts prediction (context is distracting).

What it computes: For each pair of adjacent text segments, the system computes perplexity $_{\text{conditional}}$ (with preceding context) and perplexity $_{\text{unconditional}}$ (without context). A negative difference (conditional perplexity lower than unconditional) indicates that $S_1$ makes $S_2$ more predictable — the segments are coherent and should be kept. A positive difference indicates that $S_1$ actually makes $S_2$ harder to predict — the segments are likely from unrelated documents that were accidentally concatenated, and the document should be filtered.

Why this form: Using perplexity difference rather than absolute perplexity "could largely mitigate the bias introduced by the estimator itself" — different language models used for scoring will produce different absolute perplexity values, but the direction of the difference (does context help or hurt?) is a more robust signal. The paper notes that improperly joined texts (failed HTML parsing, random social media snippets, recognition errors from complex layouts) consistently show this reverse-direction perplexity pattern, making it a reliable filter that doesn't require tuning the estimator model to the target data distribution.

Threshold Selection. Thresholds are tailored to each domain (books, papers, patents, web pages) and each language, rather than seeking a universal solution. The paper's key insight is that "employing a validation set to streamline the process, focusing only on borderline cases" is sufficient — since statistical and perplexity filters produce smooth distributions within the same domain, annotators only need to inspect samples near the threshold to decide whether to raise or lower it. Figure 6 shows the data distribution before and after filtering: a large proportion of web page data (Common Crawl) and patent data is eliminated, while most book and paper data is preserved — consistent with the intuition that formally published long documents are already more coherent than arbitrary web-scraped long pages.

3.4.4 Tokenization

The tokenizer is adapted from GPT-4's cl100k vocabulary with modifications for Chinese text efficiency:

From the 100,256-entry cl100k vocabulary (primarily English and code tokens, with fewer than 3,000 Chinese tokens), the paper selects the top 60,004 tokens.
32,397 additional Chinese tokens are integrated to improve compression rates when processing Chinese text.
147 spare tokens are included to round the vocabulary size to a multiple of 256, which "facilitates efficient training" by aligning with GPU memory alignment requirements.

The final vocabulary size remains under 100,000 while substantially improving Chinese text encoding efficiency compared to the original cl100k vocabulary's ~3,000 Chinese tokens.

3.4.5 InternEvo Training Framework

InternEvo is the distributed training framework used across all stages (pre-training, SFT, RLHF). It orchestrates training across thousands of GPUs through a combination of parallelism strategies and memory optimizations.

Parallelism Strategy. Four parallelism dimensions are combined: data parallelism (splitting batches across GPUs), tensor parallelism (splitting individual layers across GPUs), sequence parallelism (splitting long sequences across GPUs), and pipeline parallelism (splitting model layers across GPUs). ZeRO strategies (specifically ZeRO-1) further reduce memory by sharding optimizer states, gradients, and parameters.

Scaling Performance (Figure 1). At 8 GPUs with a global batch size of 4 million tokens, InternEvo achieves 64% Model FLOPs Utilization (MFU) when training InternLM-7B. Scaling to 1024 GPUs with the same batch size, MFU remains at 53% — a strong result because the computation-to-communication ratio decreases as GPU count increases while batch size stays constant. For comparison, DeepSpeed with ZeRO-1 and MiCS achieves approximately 36% MFU under the same conditions.

Long-Sequence Performance. When training InternLM-7B with 256,000-token sequences, InternEvo achieves nearly 88% MFU on 128 GPUs, compared to approximately 65% for DeepSpeed-Ulysses and Megatron-LM.

Communication Optimization. InternEvo uses adaptive sharding techniques (Full-Replica, Full-Sharding, Partial-Sharding) that allow parameters, gradients, and optimizer states to independently select the most appropriate sharding strategy and device mesh. An optimization framework searches for the most efficient sharding configuration to minimize communication while respecting GPU memory constraints.

Communication-computation overlap is achieved through strategic scheduling: during each forward and backward pass, AllGather pre-fetches full parameters for upcoming layers while the current layer is computed; gradient synchronization via ReduceScatter and AllReduce is overlapped with backward computation; and optimizer state Broadcast is overlapped with the forward computation of the next training step.

Memory Management. A memory pool provides unified memory management, and a defragmentation technique proactively consolidates small memory chunks to prevent out-of-memory errors during long-sequence training.

Fault Tolerance. An asynchronous saving mechanism archives model weights and optimizer states to distributed storage at predefined intervals. Each GPU first saves to local storage, then asynchronously uploads to remote storage. Upon hardware or network failure (automatically detected), training resumes from the most recent checkpoint, with minimal progress loss. The system supports resuming with altered parallelization configurations, and automatically transfers old checkpoints from expensive hot storage to cost-effective cold storage.

RLHF-Specific Infrastructure. During PPO training, InternEvo coordinates four equally-sized models (actor, critic, reference, reward). A custom RLHF framework built on InternEvo and Ray enables each model to execute at its optimal configuration and supports flexible algorithmic designs.

3.4.6 Model Architecture

InternLM2 adopts the LLaMA architecture to maintain compatibility with the broader open-source ecosystem:

Base: Transformer decoder-only architecture.
Normalization: RMSNorm (replacing LayerNorm).
Activation function: SwiGLU.
Position encoding: Rotary Position Embeddings (RoPE).

Two modifications differentiate InternLM2:

Group Query Attention (GQA). All model sizes (1.8B, 7B, 20B) use GQA rather than full multi-head attention or multi-query attention. GQA groups query heads to share key-value heads, reducing the KV-cache memory footprint during inference. This is specifically motivated by long-context inference: "InternLM2 aims to infer beyond 32K context, therefore, InternLM2 series models all choose Grouped-Query Attention (GQA), so that it can infer both in high speed and low GPU memory with very long contexts."

Weight Matrix Layout (Figure 2). The projection matrices for keys ( $W_k$ ), queries ( $W_q$ ), and values ( $W_v$ ) are interleaved per-head rather than concatenated straightforwardly. Specifically, instead of stacking all $W_k$ blocks, then all $W_q$ blocks, then all $W_v$ blocks, the layout alternates: head-1 $W_k$ , head-1 $W_q$ , head-1 $W_v$ , head-2 $W_k$ , head-2 $W_q$ , head-2 $W_v$ , and so on. This enables flexible tensor parallelism: the tensor parallel size can be adjusted by splitting or concatenating matrices along their last dimension without requiring matrix transposition or restructuring. The paper reports that this consolidation of $W_k$ , $W_q$ , and $W_v$ matrices alone accelerates pre-training by over 5%.

3.4.7 Three-Phase Pre-Training

The total pre-training token budget ranges from 2.0T to 2.6T tokens across the three model sizes (Table 3). Training uses AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , $\epsilon = 1e-8$ , and weight decay of 0.1. Cosine learning rate decay reduces the learning rate to 10% of its maximum value.

Phase 1: 4K Context Training (~90% of steps). The model trains on data with sequence lengths up to 4096 tokens. Data exceeding 4096 tokens is truncated, and the remaining portion is also used for training (effectively sliding a 4096-token window through longer documents). Data is mixed across English, Chinese, and code throughout.

Phase 2: Long Context Training (~9% of steps). The context window is expanded to 32K tokens. Crucially, the training data is not exclusively 32K — 50% of the data remains shorter than 4096 tokens, with the remainder being 32K-context data prepared through the specialized filtering pipeline described in Section 3.4.3. The RoPE base frequency is adjusted from 50,000 to 1,000,000. The paper states this is "ensuring more effective positional encoding for long contexts," consistent with the finding in Liu et al. (2023b) that a larger RoPE base extends the effective range of rotary position encodings.

The training speed decrease is reported as only 40% when moving from 4K to 32K context, attributed to InternEvo's efficient long-sequence handling and FlashAttention.

Phase 3: Capability-Specific Enhancement Training (~1% of steps). A curated dataset of 24 billion tokens targets specific capabilities: reasoning, mathematical problem-solving, knowledge memorization, and coding. The data is sourced from high-quality retrieved data and open-source datasets from HuggingFace (Table 4 provides details). The paper explains the motivation: "in the pre-training process, high-quality capability-related data is sparsely distributed in the entire corpus, which makes it hard for models to be proficient at these mentioned capabilities." By concentrating this data in a dedicated final phase with a smaller learning rate and batch size, the model achieves substantial improvements on these dimensions.

To support community analysis, the paper releases checkpoints before and after this phase: InternLM2-{size}-Base (after Phase 2) and InternLM2-{size} (after Phase 3). Table 14 shows that SFT models trained from the post-enhancement base model outperform those trained from the pre-enhancement base model across all capability dimensions.

3.4.8 Supervised Fine-Tuning

The SFT stage uses 10 million instruction data instances spanning general conversation, NLP tasks, mathematical problems, code generation, and function calls. Figure 7 shows the detailed topic distribution. All data is screened to ensure "helpfulness and harmlessness."

Data samples are transformed into the ChatML format, which encapsulates conversation turns with role markers (system, user, assistant). Both the 7B and 20B models train for one epoch using AdamW with an initial learning rate of $4 \times 10^{-5}$ .

Long-Context Preservation During SFT. To prevent the model from losing its long-context capability during fine-tuning, the SFT data deliberately includes long-context examples from two sources:

Long-context book data: Extended passages from books that require the model to process information across thousands of tokens.
Long-context code data: GitHub repositories processed through the dependency-sorting pipeline described in Section 3.4.2. Specifically, the paper selects code repositories from DS-1000 (covering Pandas, Numpy, TensorFlow, Scipy, Scikit-learn, PyTorch, and Matplotlib) and additional repositories with over 10,000 GitHub stars that reference these core libraries. Files within each repository are ordered using a depth-first traversal and concatenated until reaching 32K tokens. The paper notes: "the experimental results show that long-context code data improves not only the long-context capability of LLMs but also the code capabilities."

This is a non-trivial design decision: rather than treating long-context ability as a pretraining feature that is simply inherited by downstream models, InternLM2 actively maintains it throughout alignment by ensuring the training data distribution continues to include long-range dependencies.

3.4.9 COOL RLHF — Conditional Reward Model

This is the paper's primary algorithmic innovation. The core idea is to use a single reward model with conditional system prompts to model diverse, potentially conflicting human preferences, rather than training separate reward models for each preference dimension.

Architecture (Figure 8). The conditional reward model is initialized from the SFT model weights, with the output layer replaced by a one-dimensional linear mapping that produces a scalar reward score. The key innovation is that different system prompts are prepended to the input depending on the preference being evaluated. For example:

For helpfulness evaluation: a system prompt indicating "Evaluate this response for helpfulness."
For harmlessness evaluation: a different system prompt indicating "Evaluate this response for safety."
For mathematical correctness: yet another system prompt.

The reward model, having been SFT-trained to follow diverse instructions, learns to condition its scoring on these prompts — effectively routing the input through preference-specific evaluation criteria within a single set of parameters. This is in contrast to LLaMA2's approach (Figure 8a), which trains separate reward models for helpfulness and safety and combines their scores during PPO.

Why this design: The paper identifies three advantages of the single-model approach. First, computational efficiency — one model to train and run during PPO instead of two or three. Second, coherent internal representation — the reward model develops a unified understanding of what makes text good across dimensions, rather than maintaining separate, potentially inconsistent evaluation frameworks. Third, natural handling of context-dependent preference weighting — because the system prompt can encode nuanced conditions (not just "helpfulness" but "helpfulness for a coding task" vs. "helpfulness for a creative writing task"), the reward model can dynamically adjust its criteria per domain without explicit weighting coefficients.

Training Data. The reward model is trained on up to 2.4 million binarized preference pairs covering dialogue, article writing, poetry, summarization, coding, mathematics, and formatted output. Each pair consists of a chosen (preferred) response and a rejected response for the same prompt, labeled by human annotators according to the preference dimension indicated by the system prompt.

Focal Ranking Loss. The paper modifies the standard ranking loss to address sample difficulty imbalance:

$L_{\text{ranking}} = -\left(1 - 2 \times \max\left(0, P_{i,j} - \frac{1}{2}\right)\right)^{\gamma} \log(P_{i,j})$

where $P_{i,j} = \sigma(r_i - r_j)$ is the probability that reward $r_i$ exceeds reward $r_j$ , and $\gamma = 2$ by default.

What it computes: The standard ranking loss $-\log(P_{i,j})$ for a preference pair, multiplied by a difficulty decay coefficient. When the model correctly predicts the preference (i.e., $P_{i,j} > 0.5$ ), the coefficient reduces the loss contribution of that sample. The closer $P_{i,j}$ is to 1.0 (very confident correct prediction), the smaller the coefficient becomes. When the model incorrectly predicts ( $P_{i,j} \leq 0.5$ ), the coefficient equals 1 and the full loss is applied.

Why this form: The motivation is class imbalance — the dataset contains many "easy" preference pairs where the preferred response is obviously better (e.g., a coherent answer vs. gibberish) and few "difficult" pairs where the distinction is subtle (e.g., two fluent answers with different factual accuracy). Standard ranking loss treats all pairs equally, causing the model to overfit on easy distinctions at the expense of learning difficult ones. The focal term down-weights easy samples — those already predicted with high confidence — and preserves full weight on difficult samples, inspired by Focal Loss from object detection (Lin et al., 2017) but adapted to the ranking setting. The specific form $(1 - 2 \times \max(0, P_{i,j} - 0.5))^{\gamma}$ maps $P_{i,j} = 0.5$ (random guessing) to coefficient 1, $P_{i,j} = 0.75$ to coefficient $(1 - 2 \times 0.25)^2 = 0.25$ , and $P_{i,j} = 1.0$ (perfectly confident correct) to coefficient 0.

Logarithmic Barrier Penalty. To stabilize score distributions across different reward models and training runs:

$L_{\text{penalty}} = -(\log(x + 5) + \log(5 - x))$

where $x$ is the scalar reward score output by the model.

What it computes: A penalty term that strongly discourages reward scores from approaching or exceeding the bounds of $[-5, 5]$ . The logarithmic barrier function approaches $-\infty$ as $x$ approaches $-5$ or $+5$ , creating an effectively hard constraint on the output range.

Why this form: Without this penalty, different reward models trained on different data mixtures might produce scores with different scales and offsets (e.g., one model outputs scores in $[0, 10]$ , another in $[-3, 7]$ ). During PPO, such scale variations would require re-tuning the KL penalty coefficient and other hyperparameters for each reward model. The logarithmic barrier enforces a consistent score range across all configurations. The choice of $[-5, 5]$ is reported to be empirically sufficient to capture preference distinctions without being so wide as to encourage extreme score exploitation.

Total Reward Model Loss:

$L = L_{\text{ranking}} + \lambda L_{\text{penalty}}$

where $\lambda = 0.02$ based on preliminary experiments.

Training Details. The reward model size matches the actor model size (7B or 20B). Batch construction fixes the total sequence length at 16,384 tokens per batch rather than fixing the number of preference pairs, to avoid padding inefficiencies. Maximum context length is 8,192 tokens. A special token is appended to each sequence's end, and the model's output at that token's position is used as the reward score. AdamW optimizer with cosine annealing from $1 \times 10^{-5}$ to $5 \times 10^{-6}$ learning rate, weight decay 0.01, trained for one epoch to prevent overfitting.

Ablation Validation (Table 19). The paper compares the 7B conditional reward model against a version trained without system prompts on a heterogeneous mix of the same data, as well as against UltraRM-13B and QwenRM. The conditional version shows "markedly higher precision" across helpful/harmless conversations, content summaries, math problems, and Reddit replies. Without system prompts, precision drops significantly, confirming that the conditional mechanism is necessary — simply mixing preference data from different domains without domain identifiers confuses the reward model rather than teaching it to generalize.

3.4.10 COOL RLHF — Online Training Protocol

The online RLHF component addresses dynamic reward hacking through iterative refinement over three rounds, each consisting of a Fast Path and a Slow Path.

Fast Path (Targeted Reward Hacking Patches). After each round of PPO training, the system identifies specific patterns where the policy has learned to exploit the reward model. The detection mechanism is described as comparing responses from early-stage and late-stage PPO models within the current round — patterns that score highly under the reward model but are judged as low-quality by human evaluators reveal reward hacking bypasses.

For each identified hacking pattern, the system constructs 20 to 100 preference pairs that highlight the pattern: pairs where the reward-hacked response appears alongside a genuinely good response, with human preference annotated for the latter. These pairs are added to the reward model training data, and the reward model is retrained. The paper states: "Incorporating 20 to 100 such preference pairs into the training process is sufficient to prevent the reward model from the corresponding hacking pattern significantly" — the low number of required examples suggests that the hacking patterns are systematic enough that a small number of counterexamples re-calibrates the reward model's evaluation.

Slow Path (General Reward Model Improvement). In parallel with the Fast Path patches, the Slow Path accumulates new human preference data from models at various training stages. Responses from the SFT model, early-stage PPO models, and late-stage PPO models are collected, paired, and sent to professional human annotators for preference labeling. This data is added to the reward model's training set for the next round.

The key distinction: the Slow Path does not use the current round's models, because human annotation takes time. Instead, it uses "the accumulated human preferences of all previous models at the launch time of our experiments." This means the reward model is continuously updated with broader coverage of the output space as the policy evolves, improving its robustness in high-reward regions without waiting for annotation of the absolute latest model outputs.

Implementation Cadence. Three rounds of online RLHF are conducted. In each round: (1) the Fast Path patches specific hacking patterns identified from the just-completed PPO run, (2) the Slow Path incorporates all available human preference data from previous models, (3) the updated reward model is used for the next round of PPO. This creates a feedback loop where the reward model co-evolves with the policy, closing loopholes as they emerge while also improving general evaluation quality.

3.4.11 PPO Training Details

The PPO implementation involves four models of equal size: the actor model (being optimized), the critic model (value function), the reference model (frozen SFT checkpoint for KL regularization), and the reward model (frozen COOL reward model). Only the actor and critic are trained during PPO; the reference and reward models are frozen.

Model Initialization. The actor and reference models are initialized from SFT weights. The critic model is initialized from the reward model (excluding its linear output head) and undergoes a 50-iteration pre-training phase with the actor frozen, which "is critical for stabilizing the value estimation in early training."

The paper conducts an ablation study (Figure 9) comparing critic initialization from the reward model versus from the SFT model. The reward-model-initialized critic shows higher loss in the first few PPO iterations but consistently lower loss after approximately 20 iterations and leads to higher actor rewards. The paper hypothesizes that the initial higher loss reflects "fundamental differences between the tasks of reward modeling and critic modeling" — reward models evaluate complete outputs, while critics estimate expected future returns from intermediate states — but the subsequent lower loss is attributed to "a more consistent internal understanding of the world knowledge and a better grasp of the assessment principles" inherited from reward model training.

Conditional Reward Application (Figure 10). Before computing reward scores, an appropriate conditional system prompt is prepended to each query-response pair. This system prompt is agnostic to the actor, critic, and reference models — it only affects the reward model's scoring. The system prompt is selected based on the query's domain (e.g., mathematics queries receive the mathematics-evaluation system prompt).

Pre-Training Gradient. To prevent catastrophic forgetting, a pre-training loss term is added during PPO following InstructGPT methodology. The coefficient is 0.5, and the pre-training data volume is approximately 50% of the PPO training data volume. This ensures the model retains its foundational knowledge while adapting to preference feedback.

Hyperparameters. KL divergence coefficient: 0.01. Actor learning rate: $1 \times 10^{-6}$ . Critic learning rate: $5 \times 10^{-6}$ . PPO clipping parameter $\lambda$ : 0.99 (the paper notes that a larger $\lambda$ leads to higher rewards). Sampling uses $top\_p = 0.9$ , described as "slightly conservative" to balance diversity and convergence speed. Approximately 200,000 diverse queries are processed over about 400 iterations, with the best checkpoint selected on validation sets for release. The paper reports that training remains "remarkably stable" without value loss clipping or advantage normalization, "partially due to our meticulous Online RLHF efforts."

3.4.12 Long-Context and Tool Augmentation During Alignment

Long-Context Maintenance. The long-context pre-training data (Section 3.4.3) is deliberately included in both SFT and RLHF training data. The paper adopts the approach from Xiong et al. (2023) of using long-context pre-training corpora during fine-tuning, noting that "long-context code data improves not only the long-context capability of LLMs but also the code capabilities." This bidirectional benefit — long-context code teaches both context handling and programming — is an empirically observed phenomenon, not a theoretically guaranteed outcome.

Tool Calling Format (Figure 17). A modified ChatML format introduces the "environment" role to support tool calling. Two special keywords — <|interpreter|> for code execution and <|plugin|> for external API calls — enable a unified streaming format that handles chat, code execution, and tool use through the same interface. Agent training data is aligned to the chat domain and disentangled along basic language model capabilities for fine-grained training, following the Agent-FLAN methodology.

Code Interpreter for Mathematics. The model learns to solve math problems by invoking a Python code interpreter as a special tool, using the "reasoning interleaved with coding" (RICO) strategy described in InternLM-Math (Ying et al., 2024). Training data is constructed through iterative hard example mining — problems that the model initially fails are collected, solved with code interpreter assistance, and added back to the training set.

4. Key Insights and Innovations

Innovation 1: Alignment Requires Active Long-Context Maintenance — It's Not Inherited, It's Preserved

The dominant assumption in the LLM literature, visible in the release practices of most open-source models before InternLM2, is that capabilities acquired during pre-training — particularly the ability to process long contexts — are robust properties that persist through fine-tuning unless deliberately disrupted. The implicit model is that SFT and RLHF refine the model's behavior on top of a stable base of pre-trained knowledge. InternLM2's development process demonstrates, through deliberate engineering rather than controlled experiment, that this assumption is false for context length: long-context capability degrades during alignment unless explicitly maintained through data curation.

The evidence is not a single ablation table but the entire design philosophy of Sections 3.3.2, 4.1, and 4.3. The paper does not simply mention that long-context data was included in SFT and RLHF as an afterthought — it treats this as a first-class design requirement across all alignment stages. Long-context pre-training data (books and dependency-sorted code repositories) is deliberately mixed into the SFT dataset of 10 million instruction instances. During RLHF, the same long-context data continues to appear in the training distribution. The paper explicitly cites Xiong et al. (2023) as inspiration but extends the practice beyond SFT into the full alignment pipeline.

This is intellectually significant because it reframes long-context capability from a static property (the model "has" a 32K context window after Phase 2 pre-training) to a maintained property (the context window must be reinforced at every stage or it atrophies). The mechanism is intuitive in retrospect — SFT and RLHF update model weights, and if those updates are based exclusively on short-context examples, the weight configurations that enable effective long-range attention are overwritten — but the paper's contribution is establishing this as a practical principle that should govern all LLM development pipelines, not an incidental finding. The release of checkpoints at multiple stages (SFT and post-RLHF) provides the community with artifacts to study this phenomenon directly.

The broader implication is that any capability learned during pre-training that is not represented in the fine-tuning data distribution should be assumed lost unless actively preserved. This applies not just to context length but potentially to code-switching, rare language proficiency, domain-specific knowledge, and other capabilities that may not appear in typical instruction-tuning datasets. The paper doesn't make this claim explicitly, but its methodology implies it.

Innovation 2: A Single Reward Model with Conditional System Prompts Can Replace Multiple Specialized Reward Models — If Trained with the Right Loss

The standard approach to handling conflicting human preferences in RLHF, as exemplified by LLaMA2 (Touvron et al., 2023b) and Safe RLHF (Dai et al., 2023), is to train separate reward models for each preference dimension (helpfulness, harmlessness, factual accuracy) and combine their scores during PPO through weighted summation or constrained optimization. The conceptual model is that different preferences are fundamentally distinct evaluation tasks requiring specialized models, just as different NLP tasks historically required task-specific architectures.

InternLM2's Conditional Reward Model challenges this assumption at the architectural level. By initializing the reward model from an SFT checkpoint — a model already trained to follow diverse instructions — and providing different system prompts for different preference dimensions, a single set of parameters learns to evaluate responses according to multiple, potentially contradictory criteria. The key insight is that instruction-following capability generalizes to evaluation: a model that can write helpful text when prompted to "be helpful" and harmless text when prompted to "be safe" can also judge helpfulness and harmlessness when prompted to evaluate them, because the underlying understanding of what constitutes helpful vs. harmless content is shared.

The ablation in Table 19 is critical: when the same heterogeneous preference data is used to train a reward model without conditional system prompts, precision drops significantly across all domains. This rules out the alternative hypothesis that simply mixing preference data teaches a single, compromise preference. The system prompt doesn't just label the data — it fundamentally changes what the model learns from it. Without prompts, the model attempts to reconcile contradictory preferences into a single evaluation function and fails. With prompts, it learns to route inputs through prompt-conditioned evaluation pathways, effectively maintaining separate preference models within shared parameters.

This is a fundamental conceptual shift from reward model specialization to reward model conditioning. It has practical implications beyond the paper's specific implementation: future RLHF systems need not scale the number of reward models with the number of preference dimensions, which matters as the community moves toward more fine-grained preference taxonomies (e.g., separating factual accuracy by domain, or harmlessness by cultural context). The conditional mechanism provides a path to modeling dozens of preference dimensions within a single model, limited primarily by the model's capacity and the diversity of its SFT instruction-following training.

The focal ranking loss is a secondary but important innovation within this framework. Standard ranking loss treats all preference pairs equally, but preference datasets contain many easy distinctions (coherent text vs. gibberish) and few difficult ones (two fluent answers where one has subtle factual errors). The focal modification, adapted from Focal Loss in computer vision (Lin et al., 2017), prevents the model from overfitting on easy samples by down-weighting pairs it already predicts correctly with high confidence. This is not a theoretical breakthrough — it's a transfer of a known technique to a new domain — but it addresses a practical problem that likely affects all reward model training, and the paper provides a clean formulation of the adaptation to the ranking setting.

Innovation 3: Reward Hacking Is a Dynamic Phenomenon Requiring Iterative, Multi-Speed Intervention — Not a Static Problem Solved by a Better Initial Reward Model

Prior work on reward hacking in RLHF (Gao et al., 2022; Bai et al., 2022) treated it primarily as a problem of reward model quality: if the reward model accurately captures human preferences across the full output space, the policy cannot exploit it. The solution was to improve the reward model's training data, architecture, or loss function, then run PPO once and accept the result. This implicitly assumes that reward hacking is a static property of the reward model — either it has blind spots or it doesn't — and that a sufficiently good reward model eliminates the problem.

COOL RLHF's online training protocol reframes reward hacking as a dynamic co-evolutionary phenomenon. As the policy improves during PPO, it explores regions of the output space that the reward model was never trained on — not because the reward model's training data was deficient, but because these regions correspond to outputs the earlier (weaker) policy couldn't produce. The policy discovers novel exploitation strategies that the reward model, trained on data from weaker models, had no opportunity to learn about. This is a fundamental insight about the PPO process itself: optimization against a static verifier inevitably finds paths the verifier didn't anticipate, because the optimizer and verifier have asymmetric access to the output space.

The Fast Path mechanism operationalizes this insight. By adding only 20–100 targeted preference pairs that exemplify newly discovered hacking patterns, the reward model can be rapidly patched between PPO rounds. The small number of required examples is itself revealing: it suggests that reward hacking patterns are systematic — the policy finds a consistent loophole, not a collection of idiosyncratic exploits — and that a small number of counterexamples re-calibrates the reward model's evaluation in the affected region. This is distinct from the Slow Path, which accumulates broad human preference data across model generations and provides slower, more comprehensive improvement.

The two-speed structure is a conceptual innovation in its own right. It mirrors how complex systems manage reliability: a fast, reactive path that patches known failures, coupled with a slow, deliberative path that improves the system's overall competence. In the RLHF context, the Fast Path prevents the policy from running away with a single hacking strategy during a PPO round, while the Slow Path ensures the reward model's evaluation quality improves over time. Neither alone would suffice — pure Fast Path would lead to a reward model that is a patchwork of reactive fixes, while pure Slow Path would be too slow to prevent hacking within a single round.

This reframing has implications for the design of future RLHF systems. It suggests that RLHF should be designed as an iterative, closed-loop process where reward model updates are an expected and planned-for part of the training pipeline, not a one-time data collection effort. It also implies that reward model evaluation should include outputs from policies at multiple training stages, not just a static held-out set, because hacking patterns are relative to the policy's current capability level. The paper's release of models at pre-RLHF (SFT) and post-RLHF stages provides the community with concrete artifacts to study this dynamic.

Innovation 4: Capability-Specific Enhancement Training Before SFT Provides Gains That Compound Through Alignment — Not Just Incremental Pre-Training Improvement

The practice of including high-quality, task-specific data in late pre-training is not novel — Qwen (Bai et al., 2023a), GLM-130B (Zeng et al., 2023), and Nemotron-4 (Parmar et al., 2024) all incorporate instruction-style or curated data during pre-training. What InternLM2 contributes is evidence that these gains are not merely additive but multiplicative with subsequent alignment. The comparison in Table 14 shows that the same SFT procedure applied to the post-enhancement base model (InternLM2-7B) produces consistently better results than when applied to the pre-enhancement base model (InternLM2-7B-Base) across all capability dimensions (examination, reasoning, coding, QA). This is not obvious: one might expect SFT to overwrite or dilute the benefits of pre-training data curation, since SFT itself provides task-specific training. The fact that the enhancement benefits survive and amplify through SFT suggests that the enhancement phase is doing something qualitatively different from SFT — likely establishing robust internal representations of reasoning patterns, factual knowledge, and problem-solving strategies that SFT then learns to deploy in response to instructions, rather than teaching superficial patterns that SFT would simply replace.

This finding reframes the relationship between pre-training and alignment. Rather than viewing pre-training as providing "general knowledge" and alignment as providing "task-specific adaptation," the evidence suggests that pre-training data quality sets an upper bound on what alignment can extract. A model pre-trained on better data is not just a better base model — it is a better substrate for alignment, capable of absorbing instruction-following training more effectively because its internal representations of concepts are more structured and more accurate.

The practical implication is that investment in pre-training data curation has compounding returns through the full pipeline. Organizations optimizing for final chat model performance should allocate resources to late-stage pre-training data quality even when planning extensive SFT and RLHF, because improvements at the pre-training stage are amplified by subsequent stages rather than overwritten. This is a non-trivial counterpoint to the view that alignment can "fix" shortcomings in pre-training — the paper's evidence suggests alignment can exploit pre-training quality but not fully compensate for its absence.

5. Experimental Analysis

Evaluation Methodology

Datasets. InternLM2 is evaluated across an extensive suite of benchmarks organized into two categories: downstream tasks and alignment. The downstream evaluation spans six dimensions — comprehensive examinations, language and knowledge, reasoning and mathematics, coding, long-context modeling, and tool utilization — comprising over 30 benchmark datasets. Key datasets include MMLU (57 subtasks, 5-shot), CMMLU (67 subtasks, 5-shot), C-Eval (52 subtasks, 5-shot), AGIEval (20 exams, 0-shot), GAOKAO-Bench (2010–2022 Chinese college entrance exams, 0-shot), TriviaQA, NaturalQuestions, GSM8K (reasoning with code interpreter, 4-shot), MATH (4-shot), HumanEval (164 Python tasks), MBPP (974 programming tasks), L-Eval (18 subtasks, 411 documents, >2000 test cases), and LongBench (21 subtasks, 4750 test cases, bilingual). Alignment evaluation uses AlpacaEval (805 questions, single-turn), MTBench (80 multi-turn conversations across 8 dimensions), CompassArena (520 Chinese questions), AlignBench (683 Chinese QA pairs evaluated by CritiqueLLM), and IFEval (541 instruction-following questions with rule-based scoring). All evaluations are conducted using the OpenCompass toolkit unless otherwise specified.
Base model(s). The InternLM2 family spans three parameter sizes: 1.8B, 7B, and 20B, all built on the LLaMA architecture with Group Query Attention (GQA) for long-context efficiency. The models are trained from scratch through the three-phase pipeline described in Section 3. Comparisons are made against other open-source models at similar parameter scales, including LLaMA2-7B, Mistral-7B-v0.1, ChatGLM3-6B, Qwen-7B/14B, DeepSeek-7B, Baichuan2-7B/13B, and proprietary models such as GPT-3.5-Turbo (gpt-3.5-turbo-0613) and GPT-4. The paper also releases intermediate checkpoints — InternLM2-{size}-Base (after Phase 2 pre-training) and InternLM2-Chat-{size}-SFT (after SFT) — enabling analysis of capability evolution through the pipeline.
Metrics. Downstream tasks use standard per-dataset metrics: accuracy for multiple-choice and classification tasks (MMLU, CMMLU, C-Eval, WinoGrande, HellaSwag, BBH), exact match for code generation (HumanEval pass@1, MBPP pass@1), final-answer correctness for math tasks (GSM8K, MATH, TheoremQA), and BLEU for translation (FLORES). Long-context understanding uses exact match for close-ended tasks and Rouge score for open-ended tasks (L-Eval), with categorical averages reported for LongBench. Alignment evaluation uses weighted win rate (AlpacaEval v2, computed via GPT-4-Turbo logit probabilities), 1–10 scale scores (MTBench, judged by GPT-4; AlignBench, judged by CritiqueLLM), win rate against a reference (CompassArena, double-blind GPT-4-Turbo evaluation), and rule-based accuracy metrics (IFEval, four variants: prompt-level strict/loose, instance-level strict/loose, averaged).
Baselines. The paper selects baselines appropriate to each comparison. For base model evaluation (Tables 5, 7, 9, 10), comparisons include InternLM2-7B-Base (pre-enhancement), LLaMA2-7B, Mistral-7B-v0.1, ChatGLM3-6B-Base, Qwen-7B, Baichuan2-7B-Base, and DeepSeek-7B-Base at the ~7B scale, and Qwen-14B-Base, InternLM2-20B-Base, Baichuan2-13B-Base, and DeepSeek-67B-Base at larger scales. For chat model evaluation (Tables 6, 8, 11, 13), comparisons include InternLM2-Chat-7B-SFT (pre-RLHF), LLaMA2-7B-Chat, Mistral-7B-Instruct-v0.1, Qwen-7B-Chat, Baichuan2-7B-Chat, ChatGLM3-6B, and GPT-3.5-Turbo. For alignment benchmarks (Table 17), additional baselines include GPT-4, Mixtral-8x7B-Instruct-v0.1, and DeepSeek-67B-Chat.
Generation budget / compute accounting. The paper does not frame evaluation in terms of generation budget or FLOPs-matched comparison, distinguishing it from the analysis style of the reference example. Instead, evaluation follows standard benchmark protocols (few-shot prompting with specified shot counts, or zero-shot evaluation) using the established configurations in OpenCompass. For tool utilization experiments (Figures 14–16), comparisons are made with and without code interpreter access under the ReAct protocol, measuring the benefit of tool augmentation at fixed model size rather than scaling compute. There is no attempt to compare InternLM2 against larger models under equalized inference compute — all comparisons are at face value between models of similar parameter counts.
Statistical protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any result. For multi-subtask benchmarks, scores are reported as averages across subtasks without variance estimates. The Needle-in-a-Haystack evaluation (Figure 13) uses a single query per position-length combination. Data contamination analysis (Section 5.4, Table 20) uses language modeling loss comparisons with a GPT-4-generated reference set and reports two metrics ( $\Delta_1$ for potential leakage, $\Delta_2$ for overfitting) as point estimates without uncertainty quantification. The absence of variance reporting makes it impossible to assess whether small performance differences (e.g., the 0.3-point gap between InternLM2-Chat-7B-SFT and ChatGLM3-6B on LongBench overall score) are statistically meaningful.

Main Quantitative Results

Comprehensive Examination Performance

The paper reports results for base models (Table 5) and chat models (Table 6) across five examination-focused benchmarks. For base models at the 7B scale, InternLM2-7B achieves: MMLU 65.8 (5-shot), CMMLU 68.4 (5-shot), C-Eval 67.1 (5-shot), AGIEval 53.2 (0-shot), GAOKAO-Bench 57.3 (0-shot). Compared directly to InternLM2-7B-Base (pre-enhancement), the post-enhancement model shows increases of varying magnitude — e.g., AGIEval rises from 46.4 to 53.2 (+6.8 points), while MMLU moves from 63.4 to 65.8 (+2.4 points). Against other 7B models, InternLM2-7B leads on AGIEval (53.2 vs. Mistral-7B-v0.1 at 43.5 and LLaMA2-7B at 32.7) and GAOKAO-Bench (57.3 vs. Qwen-7B at 55.0), while trailing Qwen-7B on MMLU (65.8 vs. 67.3). At the 20B scale, InternLM2-20B achieves: MMLU 73.2, CMMLU 74.6, C-Eval 72.7, AGIEval 60.5, GAOKAO-Bench 67.1 — leading all tested base models across all five benchmarks.

For chat models, InternLM2-Chat-7B achieves: MMLU 65.5, CMMLU 67.2, C-Eval 66.9, AGIEval 51.9, GAOKAO-Bench 61.6. Comparing InternLM2-Chat-7B-SFT (pre-RLHF) against InternLM2-Chat-7B (post-RLHF), the paper notes that "COOL RLHF has little impact on comprehensive examination performance" — scores are nearly identical (e.g., MMLU 65.3 vs. 65.5, C-Eval 66.2 vs. 66.9). At 20B, InternLM2-Chat-20B achieves MMLU 75.5, CMMLU 76.2, C-Eval 75.3, AGIEval 63.5, GAOKAO-Bench 71.0, outperforming GPT-3.5-Turbo on CMMLU and C-Eval while showing parity on MMLU.

Language and Knowledge

For base models (Table 7), InternLM2-7B achieves: TriviaQA 56.1, NaturalQuestions 10.8 (0-shot), C3 86.1, RACE 91.6, FLORES (average BLEU across 100 language pairs, English↔other) 41.0. Notably, InternLM2-7B ranks first on RACE (reading comprehension for Chinese high school students) at 91.6, ahead of Qwen-7B at 86.1 and LLaMA2-7B at 72.4. On FLORES translation, InternLM2-7B achieves 41.0 BLEU, comparing favorably to Qwen-7B (38.0) and substantially above LLaMA2-7B (26.2). InternLM2-20B broadly leads all 13–20B models on these benchmarks.

For chat models (Table 8), InternLM2-Chat-7B achieves: TriviaQA 54.5, NaturalQuestions 17.5, C3 71.8, RACE 77.4, FLORES 37.2. The NaturalQuestions jump from 10.8 (base) to 17.5 (chat) is one of the larger SFT-induced improvements in this category, while the C3 decline from 86.1 to 71.8 suggests that SFT may trade off some language-specific comprehension for dialogue capability — though the paper does not comment on this pattern.

Reasoning

Table 9 reports Base and Chat model performance on three reasoning benchmarks: WinoGrande, HellaSwag, and BBH. For base models, the key findings are:

WinoGrande (commonsense pronoun resolution): InternLM2-7B achieves 84.7, significantly outperforming Mistral-7B-v0.1 (75.3) by 9.4 points. InternLM2-20B achieves 85.2, the highest among all tested base models.
HellaSwag (commonsense inference): InternLM2-7B reaches 80.2, competitive with Mistral-7B-v0.1 (80.5) and slightly trailing Qwen-14B-Base (81.9).
BBH (challenging BIG-Bench subset): InternLM2-7B scores 59.3, second only to Qwen-14B-Base (63.0) among tested models and substantially ahead of LLaMA2-7B (43.5) and Mistral-7B-v0.1 (56.5).

The paper highlights that InternLM2-20B outperforms InternLM2-20B-Base (pre-enhancement) across all three reasoning datasets, with "an average increase of 10.4%" attributed to domain-enhanced training.

For chat models, the comparison is particularly striking at the 7B scale: InternLM2-Chat-7B achieves WinoGrande 83.2, HellaSwag 79.1, BBH 62.6 — outperforming most 13–20B models on HellaSwag and BBH, including Mixtral-8x7B-Instruct-v0.1 (BBH 57.7) and Qwen-14B-Chat (HellaSwag 78.9). At 20B, InternLM2-Chat-20B achieves HellaSwag 85.2, which the paper notes outperforms GPT-3.5-Turbo by 15.6 points, and BBH 79.8, outperforming GPT-3.5-Turbo by 26.4 points. SFT vs. RLHF comparison shows minimal differences on reasoning, consistent with the examination result pattern — the reasoning capability is largely established before RLHF.

Mathematics

Tables 10 and 11 report Base and Chat model mathematics performance across GSM8K, MATH, TheoremQA, and MathBench.

For base models at 7B (Table 10): InternLM2-7B achieves GSM8K 70.8, substantially ahead of ChatGLM3-6B-Base (60.7), Qwen-7B (52.9), and a dramatic improvement over InternLM2-7B-Base (36.0) — the difference of +34.8 points is the largest single capability gain from enhancement training reported in the paper. On MATH, InternLM2-7B scores 24.6, trailing Qwen-7B (30.8) but ahead of DeepSeek-7B-Base (15.5). On TheoremQA, InternLM2-7B reaches 10.5, exceeding Qwen-14B-Base (10.4). At the 20B scale, InternLM2-20B achieves GSM8K 82.7, MATH 26.5, TheoremQA 13.4.

For chat models (Table 11), InternLM2 achieves leading performance at both scales. InternLM2-Chat-7B reaches GSM8K 78.8, MATH 37.3, TheoremQA 20.7 — the MATH score reflects a substantial jump from the base model's 24.6, likely benefiting from the code interpreter integration during SFT (as explored in the tool utilization section). InternLM2-Chat-20B records GSM8K 85.2, MATH 42.8, TheoremQA 24.3, outperforming GPT-3.5-Turbo (GSM8K 77.1, MATH 34.1 across different configurations) and Mixtral-8x7B-Instruct-v0.1 (MATH 28.7). On MathBench (bilingual evaluation), InternLM2-Chat-20B similarly leads across both English and Chinese subsets.

Coding

Tables 12 and 13 report Python coding performance on HumanEval and MBPP, plus multi-language evaluation on MBPP-CN (Chinese) and HumanEval-X (five-language extension).

For base models at 7B (Table 12): InternLM2-7B achieves HumanEval 51.2 and MBPP 58.8 — competitive but not class-leading. DeepSeek-7B-Base leads at this scale on both benchmarks (HumanEval 56.5, MBPP 64.3). At 20B: InternLM2-20B scores HumanEval 61.6 and MBPP 69.6, leading all compared base models.

For chat models (Table 13), the InternLM2 series shows dramatic improvements. InternLM2-Chat-7B achieves HumanEval 70.7 and MBPP 73.0, significantly ahead of the next-best 7B chat model. InternLM2-Chat-20B reaches HumanEval 81.7 and MBPP 80.9. The paper states that InternLM2-Chat-20B "surpasses the previous state-of-the-art by more than 10%" on these benchmarks, though it does not explicitly name the prior state-of-the-art being compared against.

On the multi-language MBPP-CN, InternLM2-Chat-20B shows substantial improvement over the 7B variant, but on HumanEval-X (covering C++, Java, JavaScript, Go, and Python), it exhibits "a slight decline in performance." The paper hypothesizes this "might stem from the InternLM2-Chat-20B model being finely tuned for Chinese at the expense of its effectiveness in other languages" — a trade-off observation that the paper notes but does not investigate further.

Performance Before and After Enhancement Training

Figure 12 visualizes the capability gains from Phase 3 (capability-specific enhancement training) across four aggregated dimensions. For the 7B model, the enhancement training improves: Coding (HumanEval + MBPP average) by a large margin visible in the figure (no precise aggregate number quoted), Reasoning (MATH, GSM8K, SummEdits, BBH) with a similarly large absolute gain, QA (HellaSwag, PIQA, WinoGrande, OpenBookQA, NaturalQuestions, TriviaQA) with moderate improvement, and Examination (MMLU, AGIEval, C-Eval) with a more modest gain. The darker shaded areas in Figure 12 represent pre-enhancement performance, with lighter extensions showing post-enhancement improvement — the visually largest extensions appear in the Coding and Reasoning sectors.

Table 14 provides the complementary analysis at the SFT stage: an SFT model trained from InternLM2-7B (post-enhancement) outperforms one trained from InternLM2-7B-Base (pre-enhancement) across all capability dimensions when measured with OpenCompass-categorized scores. The paper states: "SFT models trained with capability-specific enhancement achieve better performance across various capability dimensions," and notes that the SFT results reported in other sections use the pre-enhancement base model — implying that the chat model results in Tables 6, 8, 11, 13, 15, and 17 represent a lower bound on what enhancement training could achieve if carried through to alignment.

Long-Context Modeling

Benchmark Results (Table 15). All InternLM2 variants are evaluated on L-Eval and LongBench against other chat models.

L-Eval: InternLM2-Chat-20B-SFT achieves the best overall performance, leading both close-ended and open-ended subtask categories. The paper does not report the SFT-vs-RLHF comparison on L-Eval, but the presence of the SFT model as the top performer suggests that RLHF may not improve (and might slightly degrade) long-context capability — consistent with the pattern observed in other objective benchmarks.
LongBench: InternLM2-Chat-7B-SFT achieves a 48.1 overall score, leading ≤7B models on 4 out of 6 subtask categories (Single-document, Multi-document, Summarization, Few-shot learning) while on Synthetic tasks ChatGLM3-6B performs better. The overall score of 48.1 is "only slightly inferior to the 48.4 overall score of ChatGLM3-6B." A notable observation: parameter size does not produce a monotonic improvement — InternLM2-Chat-20B-SFT achieves scores comparable to or slightly below the 7B variant on some LongBench categories. The paper states this will be "further investigated."

Needle-in-a-Haystack (Figure 13). The paper evaluates long-context retrieval using a Chinese-language Haystack built from the Skywork/ChineseDomainModelingEval dataset, testing context lengths up to 200K tokens. The heatmap visualization (Figure 13) shows InternLM2 achieving near-perfect retrieval across the full context window, with no apparent degradation at extreme lengths or at any insertion position. The paper states InternLM2 "nearly perfectly identifying all 'needles' in the 'Needle-in-a-Haystack' experiment with a 200k context." This result relies on positional encoding extrapolation beyond the model's 32K training context, achieved through RoPE scaling (mentioned in Section 1). The inference is accelerated using the LMDeploy engine. No quantitative metric is reported beyond the qualitative heatmap.

Tool Utilization

The tool utilization evaluation examines two forms of augmentation: code interpreter for mathematical problem-solving, and general tool/API calling.

Mathematics with Code Interpreter (Figures 14, 15). Figure 14 compares GSM8K (4-shot) and MATH (4-shot) performance with and without code interpreter. Both InternLM2-Chat-7B and -20B show improvement with the interpreter, with the MATH gain being "notably significant." Figure 15 reports MathBench results with and without the ReAct protocol — in most cases, code interpreter improves performance, though "a minor decrease may be attributed to the incorrect usage of such interpreters." Notable disparities are observed: InternLM2-Chat-20B shows strong improvement in the Knowledge domain, while InternLM2-Chat-7B improves more in the Application section. The paper attributes these differences to training data composition differences between the model sizes.

General Tool Calling (Table 16). T-Eval (tool usage across six dimensions with Google Search and Gaode Map) and CIBench (data analysis with Jupyter notebooks, covering Pandas, Numpy, PyTorch) show InternLM2-Chat-7B as the top performer among ≤7B models on both benchmarks. InternLM2-Chat-20B achieves competitive results on T-Eval and the highest scores on CIBench. The paper notes that InternLM2 series models "achieve impressive results in Chinese, showcasing their proficiency in multiple languages" on these tool-use benchmarks.

Alignment Performance

Table 17 provides the comprehensive alignment benchmark comparison.

AlpacaEval (English, single-turn helpfulness). InternLM2-Chat-20B achieves a weighted win rate of 21.8, the highest among all compared models (including GPT-3.5-Turbo). InternLM2-Chat-7B scores 11.8. Both RLHF models outperform their SFT counterparts — InternLM2-Chat-7B (11.8) vs. InternLM2-Chat-7B-SFT (7.5), and InternLM2-Chat-20B (21.8) vs. InternLM2-Chat-20B-SFT (17.1) — directly demonstrating the benefit of RLHF on subjective alignment.

MTBench (English, multi-turn, 8 dimensions). InternLM2-Chat-20B scores 7.9 and InternLM2-Chat-7B scores 7.7 on the 1-10 scale, placing them at or near the top of their respective size classes. GPT-3.5-Turbo scores 7.9, and Mixtral-8x7B-Instruct-v0.1 scores 8.3.

CompassArena (Chinese, win rate vs. GPT-4-Turbo reference). InternLM2-Chat-20B achieves 31.4 win rate, and InternLM2-Chat-7B achieves 28.7 — both leading their size categories. The RLHF-SFT gap is large: InternLM2-Chat-7B (28.7) vs. InternLM2-Chat-7B-SFT (20.0), a +8.7 point gain, and InternLM2-Chat-20B (31.4) vs. InternLM2-Chat-20B-SFT (23.5), a +7.9 point gain. The paper highlights: "the performance gap between InternLM2's 7B and 20B versions is relatively small. However, when compared to SFT models, InternLM2's RLHF model shows a significant improvement." Figure 16 provides category-level breakdown showing that InternLM2 "possesses exceptionally strong Chinese creativity and language abilities, with a win rate that rivals that of GPT4-Turbo."

AlignBench (Chinese, CritiqueLLM judging, 1–10 scale). InternLM2-Chat-20B scores 6.8 and InternLM2-Chat-7B scores 6.1, both exceeding GPT-3.5-Turbo (5.7). Table 18 provides category-level detail: InternLM2 models excel in Knowledge, Understanding, Language, Writing, and Role Play, with weaker performance in Mathematics and Reasoning — consistent with the objective benchmark patterns.

IFEval (English, instruction following). InternLM2-Chat-7B achieves 48.5 (average of four accuracy metrics) and InternLM2-Chat-20B achieves 48.7, placing second and third in their respective size classes. English-native models (LLaMA2, Mistral) lead on this benchmark — InternLM2-Chat-7B ranks third behind Mistral-7B-Instruct-v0.1 (53.5) and LLaMA2-7B-Chat (51.2).

Ablation Study of Conditional Reward Model

The conditional reward model ablation (Table 19) provides the most controlled evidence for the conditional prompt mechanism's effectiveness. The 7B reward model trained with conditional system prompts achieves precision of (approximately, read from the text): helpful/harmless conversations — 74.3, content summaries — 73.8, math problems — 68.5, Reddit replies — 72.1. The same model architecture trained without system prompts shows "a significant decrease in precision across several public datasets." The paper compares against UltraRM-13B and QwenRM, with the conditional 7B model showing "markedly higher precision" despite being smaller than UltraRM-13B. The result confirms that mixing preference data from conflicting domains without domain identifiers hurts reward model accuracy — the conditional prompts are necessary for the single-model approach to function.

Critical Assessment

The experimental evaluation of InternLM2 demonstrates comprehensive engineering competence — the model performs competitively or superior to comparable open-source models across an unusually broad range of benchmarks. However, the experimental design has structural limitations that constrain the strength of conclusions that can be drawn.

On the claim of "exceptional performance in comprehensive evaluations across 6 dimensions and 30 benchmarks": The evidence largely supports this within the paper's self-defined scope, but the claim's strength is diluted by the absence of statistical rigor. With over 30 benchmarks reported without confidence intervals, the reader cannot distinguish meaningful leads from noise. When InternLM2-Chat-7B achieves 48.1 vs. ChatGLM3-6B's 48.4 on LongBench (a 0.3-point difference), or InternLM2-7B scores 80.2 vs. Mistral-7B-v0.1's 80.5 on HellaSwag (a 0.3-point difference), are these real differences or sampling variation? Without uncertainty estimates, the dense tables of bolded and underlined numbers create an impression of precision that the evaluation design does not support. This is not unique to InternLM2 — it's endemic to LLM technical reports — but it limits the evidential weight of the comprehensive comparison.

On the claim of "remarkable performance on the 200k Needle-in-a-Haystack test": The single-needle retrieval test is a narrow measure of long-context capability. Perfectly retrieving a single piece of information from a 200K-token document demonstrates that the model's attention mechanism can access information at extreme distances, but it does not demonstrate understanding of long contexts — the ability to synthesize information across the full span, track entities through long discourse, or reason about temporal and causal relationships spanning thousands of tokens. The LongBench and L-Eval results provide more meaningful long-context evaluation, and there the results are strong but not "remarkable" in the sense of being category-defining — InternLM2 is competitive with other models at similar scales. The Needle-in-a-Haystack result would be more informative if it included: multi-needle retrieval (tracking multiple pieces of inserted information), multi-hop reasoning across the haystack, or comparison against models with native 200K training (rather than positional encoding extrapolation from 32K).

On the COOL RLHF claims: The alignment benchmark results (Table 17, Figures 16) demonstrate substantial improvement from SFT to RLHF across AlpacaEval, MTBench, CompassArena, and AlignBench — this supports the claim that "COOL RLHF significantly enhances the performance in various subjective dialogue evaluations." However, the evidence that COOL RLHF specifically (as opposed to any RLHF) drives this improvement is weaker. The paper does not include the most informative ablation: the same RLHF pipeline without the conditional mechanism (i.e., with a single unconditioned reward model trained on the same mixed preference data) and without the online refinement (i.e., a single PPO round). The Table 19 ablation shows that the conditional reward model is more accurate than an unconditional one for reward model evaluation, but does not demonstrate that this accuracy improvement translates to better final chat model alignment compared to a single-model RLHF baseline without conditioning. The reader cannot determine how much of the alignment gain comes from: (a) using RLHF at all, (b) using a conditional reward model specifically, (c) using online refinement, or (d) simply having high-quality preference data and careful PPO hyperparameter tuning. The innovation claim for COOL RLHF would be substantially strengthened by head-to-head comparisons against an identical RLHF pipeline with different reward model configurations.

On the capability-specific enhancement claim (Figure 12): The visual comparison shows substantial gains, but the pre- and post-enhancement models differ not only in the inclusion of the 24B-token enhancement dataset but also in total training tokens. The paper states total pre-training tokens range from 2.0T to 2.6T, and the enhancement phase constitutes roughly 1% of steps. For the 7B model at 2.0T tokens, 1% represents ~20B tokens — comparable to the stated 24B enhancement dataset size. So the enhancement gain could reflect: (a) the specific curated nature of the enhancement data, (b) simply training on more tokens (the models are not token-matched), or (c) the interaction of smaller learning rate and batch size in the enhancement phase. Without a token-matched baseline or an ablation where the same compute is spent on generic data rather than curated data, the specific contribution of "capability-specific data" versus "more training" cannot be isolated.

Missing critical comparisons: The paper's main comparison class is other open-source models at similar parameter counts. Several comparisons that would strengthen the paper's claims are absent: (1) No comparison against InternLM1 (the predecessor) to quantify improvement. (2) No comparison against proprietary models beyond GPT-3.5-Turbo and limited GPT-4 references — Claude, Gemini, and other API models are not included. (3) No analysis of inference efficiency (latency, throughput, memory consumption) despite GQA being motivated specifically for "high speed and low GPU memory with very long contexts." (4) No contamination analysis beyond GSM8K (Table 20) — the paper runs one contamination test on one dataset, while claiming comprehensive evaluations across 30+ benchmarks. The absence of broader contamination testing is a notable gap, especially for benchmarks like MMLU, C-Eval, and HumanEval that have publicly available training sets. (5) The tool utilization results are reported without comparisons to models that have been explicitly optimized for tool use (e.g., ToolLLaMA, Gorilla, or function-calling variants of GPT-3.5/GPT-4), making it hard to assess whether InternLM2's tool-use performance represents a genuine strength or just a lack of strong tool-use baselines in the selected comparison set.

On long-context maintenance through alignment: The paper's methodology implies that including long-context data in SFT and RLHF preserves long-context capability, but no ablation demonstrates what happens without this inclusion. The reader cannot assess the magnitude of degradation that would occur if InternLM2 were fine-tuned on standard short-context instruction data alone. This is a missed opportunity — such an ablation would have been one of the paper's most valuable contributions, providing concrete evidence for the claim that alignment degrades context length absent active countermeasures.

6. Limitations and Trade-offs

The Hard Capability Ceiling of Late-Stage Pre-Training Enhancement

The assumption or constraint. The capability-specific enhancement phase (Phase 3, Section 3.3.3) treats 24B tokens of curated high-quality data as an additive final stage that boosts reasoning, mathematics, coding, and knowledge without altering the base pre-training architecture. The implicit assumption is that capabilities underrepresented in the main pre-training corpus can be "topped up" through a small amount of targeted data near the end of training.

The consequence. This approach leaves InternLM2 with an absolute performance ceiling on complex reasoning and mathematics that lags behind models specifically designed for these domains. On MATH, InternLM2-7B scores 24.6 and InternLM2-Chat-20B reaches 42.8 (Table 11) — competitive with general-purpose open-source models, but well below what models with dedicated mathematical reasoning architectures or training procedures achieve. On TheoremQA, InternLM2-20B scores 13.4 (Table 10), indicating that university-level theorem application remains largely out of reach. The paper does not benchmark against specialized mathematical reasoning systems or against proprietary models known to solve competition-level mathematics reliably (GPT-4 achieves ~90% on MATH with chain-of-thought prompting). The enhancement phase can improve performance on benchmarks by concentrating relevant data, but it cannot compensate for a pre-training distribution that treats mathematics as a small fraction of tokens alongside web text, code, and books. A model whose pre-training is dominated by natural language will have fundamentally different — and for mathematics, likely inferior — internal representations compared to one trained on a corpus where structured reasoning and formal symbolic manipulation constitute a substantial fraction of total tokens.

What evidence exists in the paper. Figure 12 shows that enhancement training produces the largest relative gains in Coding and Reasoning, but these gains are measured against InternLM2-Base (pre-enhancement), not against models that trained on mathematics-heavy corpora from the start. Table 10 shows InternLM2-7B trailing Qwen-7B on MATH (24.6 vs. 30.8) despite similar overall capability profiles. Section 5.2.5 notes that SFT models trained from the post-enhancement base outperform those trained from the pre-enhancement base, but this compares within-architecture, not against architectures or training recipes optimized for the target capability.

Mitigation status. Not addressed. The paper presents enhancement training as a successful strategy relative to no enhancement, but does not discuss its fundamental limits or compare against alternative approaches (mixed pre-training distributions, dedicated reasoning corpora, or architectural modifications for mathematical reasoning). The InternLM-Math work is cited (Section 4.4) as a separate effort, suggesting the authors recognize this as a distinct research direction rather than a solved problem within InternLM2.

Difficulty Estimation for RLHF Is Implicit and Unquantified

The assumption or constraint. The COOL RLHF system relies on a Fast Path that identifies specific reward hacking patterns after each PPO round and constructs 20–100 targeted preference pairs to patch them. The paper does not describe an automated detection mechanism — the language implies manual inspection:

"After identifying the hacking pattern after each round of RLHF, we construct preference pairs that highlight these patterns by comparing responses generated by early and late-stage PPO models in the current round." (Section 4.2.2)

The assumption is that human experts can reliably and comprehensively identify reward hacking patterns across the policy's output distribution.

The consequence. In a production deployment where COOL RLHF is applied to a new domain or a larger model, the Fast Path becomes a manual, labor-intensive quality assurance process. The paper provides no guidance on: how many hacking patterns to expect per round, how to systematically search for them, what fraction of hacking patterns might go undetected, or how the cost scales with model size and PPO iteration count. If reward hacking manifests in subtle ways — e.g., the model learns to use slightly more formal language that the reward model associates with higher quality without actually improving content — these patterns may be difficult for human reviewers to notice, especially when evaluating hundreds of thousands of generated responses. An undetected hacking pattern that persists through multiple rounds could become entrenched in the policy, since subsequent PPO training reinforces it.

Furthermore, the "three rounds of online RLHF" (Section 4.2.2) is presented as a fixed schedule rather than a data-dependent or convergence-based criterion. The paper does not discuss whether three rounds were sufficient, whether additional rounds would have produced further gains, or whether the optimal number of rounds varies with model size, data composition, or preference taxonomy.

What evidence exists in the paper. Table 17 shows that RLHF (post-COOL RLHF models) substantially outperforms SFT on alignment benchmarks, and Section 4.2.2 states that three rounds were conducted with thousands of preference patches gathered. However, no ablation varies the number of rounds or compares manual vs. automated hacking detection. The paper does not characterize the types of hacking patterns discovered, their frequency, or the coverage of the Fast Path patches. Figure 9 shows stable PPO training, but stability during training does not guarantee that all hacking patterns were found and patched — it only guarantees that the training did not diverge.

Mitigation status. Not addressed. The paper presents the Fast Path/Slow Path mechanism as successful and reports the outcome (three rounds, thousands of patches), but does not discuss the scalability or reliability of the manual inspection process that powers the Fast Path. The authors do not suggest future work on automated hacking detection, leaving this as an implicit dependency on human judgment that future adopters would need to replicate.

Long-Context Capability Depends on RoPE Extrapolation Without Systematic Validation

The assumption or constraint. InternLM2 is trained on a maximum context length of 32K tokens during pre-training and alignment (Sections 3.3.2, 4.3). The demonstrated 200K context window in the Needle-in-a-Haystack test (Figure 13) relies on positional encoding extrapolation beyond the training distribution, specifically Dynamic Scaled RoPE:

"upon completion, through positional encoding extrapolation, InternLM2 achieves commendable performance in the 'Needle-in-a-Haystack' test within 200k contexts." (Section 1)

The assumption is that RoPE-based length extrapolation, which adjusts the rotation frequencies for positions beyond those seen during training, preserves attention quality uniformly across all position pairs up to 200K tokens.

The consequence. The Needle-in-a-Haystack test evaluates a single, narrow capability: retrieving one piece of information from a long document when explicitly queried about it. This tests whether the model's attention mechanism can access information at extreme distances, but it does not test whether attention quality degrades — whether the model can correctly weight information from different positions, track entities across long spans, or perform multi-hop reasoning that requires integrating information from multiple distant locations within the document.

The paper's own LongBench results (Table 15) reveal that parameter scaling does not produce monotonic improvement in long-context understanding: InternLM2-Chat-20B does not consistently outperform the 7B variant, and on some subtask categories performs slightly worse. This pattern is inconsistent with the near-perfect Needle-in-a-Haystack result — if attention quality is truly preserved out to 200K, larger models should be uniformly better at using that attention capacity. The discrepancy suggests that RoPE extrapolation enables information access (finding the needle) but does not guarantee that the model can effectively use that information in complex reasoning tasks, and that larger models may be more susceptible to attention degradation at extrapolated lengths than smaller ones.

Additionally, the paper does not report Needle-in-a-Haystack results for Multi-Needle retrieval (finding multiple pieces of information simultaneously), multi-hop variants (where answering requires synthesizing information from the needle with other document content), or any task requiring the model to ignore irrelevant information that superficially matches the query pattern. A single-needle retrieval test is the minimum viable demonstration of long-context access, not a comprehensive validation of long-context understanding.

What evidence exists in the paper. Figure 13 shows the single-needle heatmap, which is essentially flawless. Table 15 shows LongBench results where scaling does not help, suggesting a gap between retrieval and understanding. No ablation compares performance at 32K (the last trained length) vs. 200K (extrapolated) on tasks requiring multi-hop reasoning or information synthesis. The paper does not report the specific RoPE extrapolation method used, the hyperparameters (e.g., the scaling factor for Dynamic NTK-aware RoPE), or any sensitivity analysis.

Mitigation status. Partially acknowledged. The paper reports the LongBench results alongside the Needle-in-a-Haystack results and notes the non-scaling pattern without explaining it. However, the paper's headline claim of "Designed with a 200k Context Window" (Section 1, Contribution 2) implies a level of long-context capability that the Needle-in-a-Haystack test alone does not substantiate. No systematic validation of extrapolation quality beyond the single-needle retrieval test is presented or proposed as future work.

Tool Utilization Evaluation Lacks Strong Baselines and Controlled Ablations

The assumption or constraint. The tool utilization results (Section 5.2.7) are presented as evidence that InternLM2 effectively leverages external tools and code interpreters. The paper implicitly assumes that the chosen baselines — other open-source chat models not specifically optimized for tool use — constitute an appropriate comparison class.

The consequence. The paper's tool utilization claims are difficult to interpret because the evaluation lacks the most relevant comparisons: models that have been explicitly trained for tool use or that represent the current state of the art in tool-augmented LLMs (e.g., ToolLLaMA, Gorilla, GPT-4 with function calling, or Claude with tool use). When InternLM2-Chat-7B is reported as the "top performer among ≤7B models" on T-Eval and CIBench (Table 16), the reader cannot determine whether this reflects genuine tool-use proficiency or merely the absence of strong tool-use baselines in the comparison set.

Moreover, the code interpreter results (Figures 14, 15) show that InternLM2 benefits from tool access — but this is true of essentially all LLMs when provided with code execution capabilities. The interesting comparative question is whether InternLM2 benefits more than other models, or whether its tool-use accuracy (the rate at which it correctly invokes the right tool with the right arguments) is better. The paper reports final-task accuracy with and without tools, but does not decompose into tool-selection accuracy, tool-argument accuracy, and post-tool reasoning accuracy. This decomposition is essential for understanding whether tool-use failures stem from the model choosing the wrong tool, mis-specifying tool arguments, or failing to interpret tool outputs — each requires a different improvement strategy.

The paper also notes puzzling patterns without explanation: "InternLM2-Chat-20B model exhibits substantial improvement over the InternLM2-Chat-7B model in MBPP-CN benchmarks, yet it shows a slight decline in performance on HumanEval-X" and hypothesizes this "might stem from the InternLM2-Chat-20B model being finely tuned for Chinese at the expense of its effectiveness in other languages" (Section 5.2.4). If tool-use capability degrades for non-Chinese tasks in the larger model, this represents a significant practical limitation for multilingual deployments, but the paper treats it as an observation rather than investigating the mechanism or proposing mitigation.

What evidence exists in the paper. Figures 14 and 15 show with-vs-without code interpreter comparisons. Table 16 compares InternLM2 against general-purpose chat models on T-Eval and CIBench. No comparison is made against tool-specialized models. No decomposition of tool-use success/failure modes is provided. The HumanEval-X decline at 20B is noted but not analyzed.

Mitigation status. Not addressed. The paper does not acknowledge the weakness of the tool-use baseline selection, does not provide failure-mode decomposition, and treats the cross-lingual tool-use degradation as an incidental observation rather than a systematic limitation requiring investigation.

Absence of Inference Efficiency Characterization Undermines the GQA Motivation

The assumption or constraint. The paper explicitly motivates the adoption of Group Query Attention (GQA) as a mechanism for efficient long-context inference:

"InternLM2 aims to infer beyond 32K context, therefore, InternLM2 series models all choose Grouped-Query Attention (GQA), so that it can infer both in high speed and low GPU memory with very long contexts." (Section 2.2)

This is a deployment-focused claim: GQA is chosen not for training efficiency or model quality, but for inference-time benefits — reduced KV-cache memory footprint and higher generation throughput at long context lengths.

The consequence. The paper provides extensive training efficiency data (Figure 1: MFU scaling across GPU counts and sequence lengths) but zero inference efficiency data. No measurements of: KV-cache memory consumption at 32K, 100K, or 200K context lengths; tokens-per-second generation throughput at varying batch sizes and context lengths; latency (time-to-first-token and per-token generation time) for long-context prompts; or comparison against an equivalent model with full multi-head attention (i.e., what would the memory and speed penalty be without GQA?).

This is a significant gap for a model whose headline feature (Contribution 2) is a 200K context window. Practitioners evaluating InternLM2 for deployment in long-context applications — document analysis, RAG with large retrieved sets, multi-turn agent interactions with accumulated context — need to know the actual memory and latency costs, not just that GQA "enables" efficient inference. A model that can theoretically process 200K tokens but requires impractically large GPU memory or incurs unacceptable latency is not practically deployable for that use case.

Furthermore, the paper does not report inference efficiency comparisons against other long-context models (Mistral-7B, which uses grouped attention variants; GPT-4-128K, whose inference characteristics are partially known through API behavior). Without such data, the claim that GQA "enables" efficient long-context inference is a statement about architecture rather than a demonstrated property of the specific implementation.

What evidence exists in the paper. Figure 1 reports training MFU at up to 256K sequence length (88% MFU for InternLM-7B on 128 GPUs). No inference efficiency data is reported anywhere. Section 5.2.6 reports long-context benchmark accuracy and the Needle-in-a-Haystack result, but no associated throughput or memory measurements.

Mitigation status. Not addressed. The paper does not acknowledge the absence of inference efficiency data, does not provide inference benchmarking methodology, and does not list inference efficiency characterization as future work. The training infrastructure section (Section 2.1) is detailed, but the inference infrastructure (beyond a mention of LMDeploy being used for Needle-in-a-Haystack acceleration) is essentially undocumented.

Data Contamination Analysis Is Insufficient for the Scope of Benchmarking Claims

The assumption or constraint. The paper evaluates InternLM2 across 30+ benchmarks and claims state-of-the-art or competitive performance. The only data contamination analysis provided is on GSM8K (Table 20), using a language modeling loss comparison methodology. The assumption is that this single-dataset analysis is representative, or that other benchmarks are less susceptible to contamination concerns.

The consequence. Many of the benchmarks where InternLM2 shows strong performance — MMLU, C-Eval, HumanEval, MBPP, TriviaQA — have publicly available training sets or have been widely distributed online. If InternLM2's pre-training corpus (which includes Common Crawl data, web pages, and open-source datasets from HuggingFace) inadvertently included test-set examples from these benchmarks, the reported accuracy numbers would overstate the model's true generalization capability. This is particularly concerning for benchmarks with well-defined, publicly available test sets: models trained on web data can memorize test questions and answers without learning the underlying capability.

The GSM8K contamination analysis (Table 20) provides some reassurance — InternLM2-7B and InternLM2-20B show $\Delta_1$ and $\Delta_2$ values in normal ranges — but GSM8K is a relatively small, specific dataset (grade-school math word problems). The contamination behavior on GSM8K tells us nothing about contamination on MMLU (57 subtasks spanning medicine, law, physics, computer science), C-Eval (52 Chinese examination subjects), or HumanEval (164 programming problems that exist in various forms across GitHub repositories, which InternLM2's pre-training corpus explicitly includes).

The paper states that for the enhancement training phase, "We filter out test set related data and run a contamination test as illustrated in Section 5.4" (Section 3.3.3), but Section 5.4 only reports GSM8K results. This implies contamination filtering was performed, but the scope and methodology of that filtering are not described, and its effectiveness is only validated on one dataset.

What evidence exists in the paper. Table 20 provides the sole contamination analysis (GSM8K only). Section 3.3.3 mentions contamination filtering for enhancement data without detail. No contamination results are reported for any other benchmark.

Mitigation status. Minimally addressed. The paper acknowledges the data contamination issue by including Section 5.4 and performing a GSM8K analysis, but the scope is far too narrow to support the breadth of benchmarking claims. The paper does not: report contamination filtering methodology for the main pre-training corpus, provide contamination analyses for the most critical benchmarks (MMLU, C-Eval, HumanEval), discuss the risk of indirect contamination (where test questions appear in slightly paraphrased form in training data), or acknowledge this as a limitation of the evaluation. Future work on contamination detection methodology is not proposed.

7. Implications and Future Directions

How This Work Changes the Landscape

InternLM2 is not a paradigm shift in the sense of introducing a fundamentally new architecture or training algorithm. It does not claim to. What it changes is the standard of documentation and methodological transparency expected from open-source LLM releases. Prior to InternLM2, the dominant pattern in major open-source technical reports — LLaMA (Touvron et al., 2023a; 2023b), Qwen (Bai et al., 2023a), Mistral (Jiang et al., 2023), DeepSeek (Bi et al., 2024) — was to emphasize benchmark performance and high-level architecture while treating data preparation and alignment methodology as cursory descriptions. InternLM2 inverts this emphasis: Sections 3.1.1 through 3.1.3 provide actionable detail on text data formatting, MinHash deduplication parameters (128 hash functions, 0.7 Jaccard threshold), safety classifier training (domain blocklists of ~13M entries, word blocklists of 36,289 terms, fine-tuned BERT classifiers for toxicity and pornography), quality filtering (BERT-based advertisement and fluency classifiers trained on manually annotated data), code dependency sorting via topological sort on import graphs, and long-context perplexity filters that use conditional probability differences to detect incoherent concatenations.

This level of specificity transforms the paper from a model announcement into a reference engineering guide. A team starting an LLM training effort today can use InternLM2's described pipeline as a concrete starting point — specific enough to replicate, detailed enough to modify consciously — rather than relying on tribal knowledge or reverse-engineering from incomplete prior reports. The paper's Contribution 3 ("Comprehensive Data Preparation Guidance") is arguably its most enduring contribution, independent of the specific model weights released.

The second landscape change is reframing RLHF as an iterative, co-evolutionary process rather than a one-shot alignment step. The COOL RLHF system's Fast Path/Slow Path mechanism operationalizes the insight that reward hacking is dynamic: as the policy improves during PPO, it explores regions of output space that the reward model was never trained on, because earlier, weaker policies couldn't produce those outputs. The Fast Path acknowledges that reward model patches are an expected, planned-for part of the training pipeline, not a sign of initial reward model inadequacy. This reframes RLHF from "train a good reward model, then run PPO once" to "run PPO, identify new hacking patterns, patch the reward model, run PPO again" — a closed-loop process where the reward model and policy co-evolve.

The paper does not definitively prove that this online approach is superior to single-round RLHF with an equally well-trained reward model — the critical ablation comparing multi-round online against single-round with equivalent total preference data is absent — but it establishes the conceptual architecture for thinking about RLHF this way. The Slow Path's practice of accumulating preference data across model generations, and the Fast Path's use of early-vs-late PPO model comparisons to surface hacking patterns, provide concrete mechanisms that future work can build on, refine, and rigorously evaluate.

The third landscape shift is the explicit demonstration that long-context capability must be actively maintained through alignment, not passively inherited from pre-training. The paper does not provide the controlled ablation showing degradation without long-context data in SFT/RLHF — the reader cannot quantify the magnitude of the effect — but the design philosophy itself, treating long-context data as a mandatory component of alignment datasets rather than an optional enhancement, represents a methodological norm that the paper advocates through its example. If future open-source releases follow this practice, InternLM2 will have shifted community standards around context-length preservation.

Finally, the paper reconciles a latent tension in the open-source LLM narrative: the gap between proprietary and open-source models is often attributed to scale (more parameters, more data, more compute), but InternLM2's results suggest that engineering rigor in data curation and alignment methodology can close substantial portions of that gap at moderate parameter counts. The 20B model outperforms GPT-3.5-Turbo (a system likely much larger) on HellaSwag (by 15.6 points), BBH (by 26.4 points), MATH (42.8 vs. 34.1), and AlignBench (6.8 vs. 5.7). These results do not prove that engineering beats scale — the comparison is not FLOPs-matched, and GPT-3.5-Turbo's architecture and training details are unknown — but they demonstrate that excellent execution at 20B parameters can be competitive with much larger proprietary systems on a broad range of benchmarks. This tilts the research landscape toward investigations of data quality, staged training, and alignment methodology as high-leverage investments, rather than assuming that scaling parameters is the primary path to competitive performance.

Follow-Up Research This Work Enables

Quantifying the degradation of long-context capability during alignment when long-context data is excluded. The paper's methodology implies that including long-context pre-training data in SFT and RLHF is necessary to preserve context window, but provides no controlled measurement of the alternative. A follow-up study would take InternLM2-7B-Base (post-Phase-2, 32K-context pre-trained) and run two parallel SFT + RLHF pipelines: one with long-context data deliberately excluded from the alignment datasets (standard short-instruction SFT only), and one with the InternLM2 recipe. Comparing performance on L-Eval, LongBench, and Needle-in-a-Haystack at multiple context lengths (4K, 8K, 16K, 32K) would quantify: (a) the absolute degradation in long-context capability caused by alignment without long-context data, (b) the specific context-length thresholds where degradation becomes measurable (does 4K survive? 8K?), (c) whether the degradation is uniform across all position ranges or concentrated at distant positions, and (d) whether RLHF causes additional degradation beyond SFT. This experiment would convert an implicit design principle into a quantified finding with direct practical implications for every team doing alignment.

Head-to-head comparison of conditional vs. multi-model reward approaches under controlled data and compute budgets. The paper claims that a single conditional reward model resolves preference conflicts and reduces computational cost compared to training multiple specialized reward models (Section 4.2, Figure 8). However, the only direct evidence is the Table 19 ablation showing that the conditional reward model has higher reward model precision than an unconditional model trained on the same mixed data. This does not demonstrate that the conditional single-model approach produces better final chat model alignment than a multi-model approach. A rigorous follow-up would: (a) fix the total preference data budget (e.g., 2.4M preference pairs), (b) train one conditional reward model on all pairs with system prompts vs. two separate reward models (one for helpfulness, one for harmlessness) each trained on their respective subsets, (c) run PPO with each configuration using identical hyperparameters, and (d) evaluate the resulting chat models on AlpacaEval, MTBench, and CompassArena. The hypothesis to test: does the conditional model's shared representation across preference dimensions provide a tangible alignment benefit, or is the primary advantage merely operational (fewer models to manage) rather than qualitative?

Automated reward hacking detection for the Fast Path. The COOL RLHF Fast Path relies on manual inspection to identify specific reward hacking patterns after each PPO round (Section 4.2.2). The paper provides no characterization of what hacking patterns were found, how they were detected, or how comprehensive the detection was. A follow-up could develop automated detection by training a classifier to distinguish early-stage PPO outputs from late-stage PPO outputs and then identifying which late-stage outputs receive high reward model scores despite being classified as "late-stage" (suggesting the model shifted its output distribution toward regions the reward model overvalues). The ground truth would be the set of hacking patterns manually identified in the InternLM2 development process. This would transform the Fast Path from a manual, labor-intensive process into a scalable, reproducible component, and would produce a dataset of reward hacking patterns that the community could use to study the phenomenon systematically.

Extending the conditional reward model to fine-grained, dynamically weighted preferences. The paper's conditional reward model uses discrete system prompts to switch between preference dimensions (helpfulness, harmlessness, mathematical correctness). A natural extension is to make the conditioning continuous and interpolatable: instead of "evaluate for helpfulness" vs. "evaluate for harmlessness," condition on a vector that specifies the desired trade-off (e.g., "weight_harmlessness=0.3, weight_helpfulness=0.7"). If the reward model can learn to interpolate between preference dimensions in a smooth, predictable way, then the PPO process could dynamically adjust preference weights per-query based on the query's content or user-specified requirements. A specific experiment: train the conditional reward model with system prompts that explicitly state numerical weights for 2–3 preference dimensions, then evaluate whether the reward scores for held-out preference-weight combinations follow the expected linear interpolation, and whether PPO with dynamic per-query weighting produces chat models that adapt their behavior appropriately.

Capability-specific enhancement training with controlled token-matched and compute-matched baselines. The Phase 3 enhancement training (Section 3.3.3) adds 24B tokens of curated high-quality data to the end of pre-training, and Figure 12/Table 14 show substantial capability gains from this phase. However, the pre- and post-enhancement models differ in total training tokens as well as data composition. To isolate the effect of data curation from additional training, a follow-up would: (a) train one model with InternLM2's standard pipeline, (b) train a token-matched baseline where the 24B enhancement tokens are replaced with 24B tokens randomly sampled from the main pre-training distribution, (c) train a third baseline where the enhancement phase uses the same curated data but with the Phase 1 learning rate and batch size (to isolate the effect of the smaller LR/BS used in Phase 3), and (d) evaluate all three on the same benchmark suite. This would decompose the Phase 3 gain into components attributable to: more tokens, better data, and training hyperparameters, providing concrete guidance on where to invest resources in future pre-training pipelines.

Systematic contamination analysis across the full benchmark suite. The paper's data contamination analysis is limited to GSM8K (Table 20), yet the model is evaluated across 30+ benchmarks including MMLU, C-Eval, HumanEval, MBPP, and TriviaQA — all with publicly available training or test sets that could appear in web-scraped pre-training data. A comprehensive follow-up would apply the LM-loss comparison methodology (and complementary methods like n-gram overlap analysis and embedding similarity search) to every benchmark where InternLM2 reports top-tier performance, using GPT-4-generated reference sets to establish contamination-free baselines for each. The output would be a contamination impact matrix showing which performance claims are robust and which may be inflated. This is particularly important for C-Eval and CMMLU, where InternLM2's strong performance could reflect genuine Chinese language capability or could reflect leakage of Chinese examination questions into the pre-training corpus — the paper's current analysis cannot distinguish these hypotheses. Given that InternLM2's pre-training corpus explicitly includes Chinese web data and open-source datasets from HuggingFace, the risk of benchmark contamination is non-trivial and uncharacterized.

Practical Applications and Downstream Use Cases

Bilingual (Chinese-English) production deployment at moderate scale. InternLM2-Chat-20B achieves CompassArena win rate 31.4 and AlignBench score 6.8, both exceeding GPT-3.5-Turbo (Table 17), while also scoring 75.5 on MMLU and 75.3 on C-Eval (Table 6). For organizations serving Chinese-speaking users — customer support, educational technology, content generation for Chinese social media — this means a 20B-parameter open-source model can match or exceed the subjective quality of a much larger proprietary API, without data leaving the organization's infrastructure. The specific value proposition is: competitive Chinese dialogue quality (the CompassArena category breakdown in Figure 16 shows InternLM2's Chinese language ability rivaling GPT-4-Turbo) combined with on-premise deployment and the ability to further fine-tune on proprietary data. The 1.8B and 7B variants extend this to resource-constrained environments where 20B is infeasible.

Long-context document analysis pipelines where context degradation during fine-tuning would be catastrophic. InternLM2's methodology of deliberately including long-context data through SFT and RLHF (Section 4.3) produces a chat model that maintains its 32K training context window and extrapolates to 200K for retrieval (Figure 13). For applications like legal document review (processing 100+ page contracts), scientific literature synthesis (ingesting full papers with citations), or multi-document financial analysis, the alternative — using a model fine-tuned on short instructions that has silently lost its ability to attend across long documents — produces subtle but systematic failures where information from later sections of a document is effectively invisible to the model. InternLM2-Chat-20B-SFT achieves top performance on L-Eval (Table 15), making it a strong candidate for these pipelines. The release of both SFT and RLHF checkpoints allows practitioners to choose between maximum long-context accuracy (SFT) and maximum alignment quality (RLHF) depending on the use case priority.

Code repository understanding and generation with dependency-aware context. The dependency-sorting methodology for code data (Section 3.1.2) and the deliberate inclusion of 32K-token code repository contexts in SFT (Section 4.3) make InternLM2 particularly well-suited for tasks that require understanding multi-file software projects. On HumanEval and MBPP, InternLM2-Chat-20B achieves 81.7 and 80.9 respectively (Table 13), and on the data science benchmark CIBench, it leads all compared models (Table 16). For software engineering tools — automated code review, repository-level refactoring suggestions, generating tests that respect inter-file dependencies — InternLM2 provides a model whose training explicitly taught it to reason about code across file boundaries and dependency structures, rather than treating each file as an isolated snippet.

Self-improvement data generation pipelines where RLHF stability matters. The COOL RLHF system's online refinement protocol (Section 4.2.2) and the reported "remarkably stable" PPO training without value loss clipping or advantage normalization (Section 4.2.3) suggest InternLM2's alignment infrastructure is robust enough for iterative self-improvement loops. In a ReST-style pipeline (Singh et al., 2024) where a model generates training data for its own next iteration, reward model reliability is paramount — if the reward model has systematic blind spots, the data generation process amplifies them. The Fast Path's ability to patch newly discovered hacking patterns with only 20–100 targeted examples makes it feasible to run multiple rounds of data generation and refinement without accumulating reward model errors. Organizations building automated alignment pipelines can adopt the COOL RLHF architecture to maintain reward model quality as the policy distribution shifts across iterations, rather than requiring a new, expensive human annotation campaign for each round.

When to Prefer This Method

The paper does not explicitly position COOL RLHF against named competing alignment methods with a tradeoff analysis backed by controlled experiments. The comparison in Figure 8 contrasts the conditional single-model approach against LLaMA2's multi-model approach at the conceptual level, and Table 19 shows reward model precision improvements when using conditional prompts, but there is no head-to-head PPO comparison where the only variable is the reward model architecture. Similarly, the paper does not articulate clear decision criteria for when a practitioner should choose InternLM2's specific data preparation pipeline over alternatives (e.g., the RefinedWeb approach used by Falcon, or the data processing described in the LLaMA papers). The methodological contribution is demonstrating a pipeline that works well, not establishing that it is superior to specific alternatives under measurable conditions.

As such, a "Prefer A when X, prefer B when Y" decision matrix would be interpolating beyond what the paper's evidence supports. The paper's primary deployment guidance is implicit: if you are building an open-source LLM and need a well-documented, end-to-end methodology covering data preparation, staged pre-training, and alignment — and particularly if you need strong bilingual Chinese-English performance and long-context capability maintained through alignment — InternLM2 provides a validated reference pipeline. Whether it should be preferred over other specific pipelines for a given use case is not established by the experiments reported.