Finding Experts in Transformer Models

ArXiv: 2005.07647

🎯 Pitch

A mere 0.012% of neurons in GPT‑2 can force it to write entire paragraphs saturated with a target concept like “football” — no retraining, no extra parameters. This neuron‑level controllability also predicts how well any Transformer will generalize across NLP benchmarks, with a striking r² = 0.833.

---

1. Executive Summary

This paper studies the presence of expert units — individual neurons that classify a specific concept with high average precision — in pre-trained Transformer models, analyzing their relationship to model generalization, concept co-learning, and controllable text generation using BERT, RoBERTa, GPT2, and other models on a 1,641-concept dataset built from OneSec. The paper introduces concept expertise (the percentage of concepts for which the model contains at least one neuron achieving a precision above a learned threshold γ⋆, where γ⋆ = 0.997 for sense concepts and γ⋆ = 0.985 for homograph concepts), and demonstrates that this metric strongly correlates with downstream task performance (r² = 0.833), surpassing the average inter-task correlation across GLUE and SQuAD benchmarks while requiring no fine-tuning. The paper further shows that forcing only the top expert neurons for a given concept to remain active during generation — as few as 0.012–0.048% of all units — induces that concept in the model's output without retraining or additional parameters, establishing that pre-trained language models already contain specialized, causally effective concept representations that can be directly exploited at inference time.

2. Context and Motivation

The Core Problem: We Don't Understand What Transformer Neurons Actually Learn

The fundamental question this paper tackles is deceptively simple: when a Transformer model learns from massive text corpora, do individual neurons develop specialized, interpretable roles — and if so, how can we find and exploit them? This matters because, despite the widespread deployment of pre-trained Transformer models (TMs) across NLP, the field lacked a systematic, general-purpose method for identifying what specific concepts individual neurons have learned to detect, and whether those neurons actually cause concept-relevant behavior rather than merely correlating with it.

The paper frames this gap through a neuroscience-inspired lens (Section 1), drawing on evidence that the human brain organizes into "hierarchically organized specialized modules" (Hill and Schneider, 2006) and that individual neurons become increasingly selective as learning progresses (Najafi et al., 2019). The hypothesis is straightforward: if biological neural networks develop specialized units during learning, perhaps artificial ones do too — and perhaps the presence of such specialization is what makes pre-trained models generalize well.

This gap is significant for several practical and theoretical reasons the paper raises throughout Section 1 and Section 8:

• Model evaluation without fine-tuning: The standard practice for comparing pre-trained models is to fine-tune each one on a suite of downstream tasks (GLUE, SQuAD, etc.) and average the scores — a process that is computationally expensive, requires hyperparameter tuning per task, and introduces "downstream task bias" (Niven and Kao, 2019), where a model may look good on one benchmark but fail on others. If generalization power could be estimated directly from the model's internal representations, evaluation would become dramatically cheaper and less biased.
• Explainability of model behavior: When a model fails on a specific input — say, a dialogue system misinterprets a user's intent — there is no standard way to understand which concepts the model has learned that might explain the failure. A method that discovers which concepts individual neurons detect would provide a vocabulary for explaining internal representations.
• Controllable generation without retraining: Existing methods for conditioning language model outputs on specific concepts (sentiment, topic, style) typically require training dedicated architectures with disentangled latent variables (Hu et al., 2017; Romanov et al., 2019; Chen et al., 2019) or training massive conditional models from scratch like CTRL (Keskar et al., 2019). These approaches are effective but require access to the training procedure, substantial computational resources, and often architectural modifications. If pre-trained models already contain neurons that causally control concept presence, one could condition generation by simply manipulating those neurons — no retraining needed.
• Understanding what makes pre-training work: Despite the empirical success of pre-training, there is no consensus on why models like BERT and GPT2 generalize so well. The expert unit hypothesis offers a possible mechanistic explanation: pre-training produces a diverse collection of specialized feature detectors, and generalization emerges from the model's ability to compose these specialists for novel tasks.

Conflicting Signals in Prior Interpretability Work

The paper is motivated by a landscape of interpretability research that has produced useful but fragmented insights, with several methodological limitations that prevent a unified picture of neuron-level specialization in Transformers.

Saliency-based methods provide word-level, not concept-level, understanding. Several works analyze Transformer self-attention layers by visualizing which words attend to which other words (Clark et al., 2019; Ghaeini et al., 2018). While these visualizations can reveal syntactic patterns — such as attention heads that specialize in attending to objects of verbs or coreference links — they produce word-word relationships that are "hard to extract model-wide conclusions" from (Section 2). More critically, recent work has shown that saliency methods can be "invariant to the model or the data" (Adebayo et al., 2018) and "can be easily manipulated" (Dombrowski et al., 2019), raising fundamental questions about whether attention patterns actually reflect the computations the model performs.

Probing classifiers require training external models and are task-specific. Another major approach trains auxiliary classifiers on top of intermediate representations to assess whether those representations encode specific linguistic properties — sentence-level tasks like sentiment or entailment (Conneau et al., 2018; Adi et al., 2017), or word-level properties like part-of-speech and dependency relations (Tenney et al., 2019; Liu et al., 2019). The paper explicitly acknowledges this line of work as inspirational ("These approaches inspired our work"), but identifies a key limitation: these methods train new classifiers to solve downstream tasks, rather than treating the model's existing units as classifiers themselves. The distinction is subtle but important — a probing classifier can find that some linear combination of neurons encodes a concept, but it doesn't tell you whether any individual neuron is an expert for that concept. The probing approach conflates distributed and local representations; this paper specifically seeks local, per-neuron expertise.

Disentangled learning methods require training from scratch or with modified objectives. Approaches that achieve controllable generation by disentangling concepts like syntax/semantics (Chen et al., 2019), meaning/form (Romanov et al., 2019), or sentiment/tense (Hu et al., 2017) all bake the desired structure into the training process — either through multi-task objectives, adversarial losses, or architectural constraints. The resulting models can generate text conditioned on these concepts, but the method cannot be applied to an existing, off-the-shelf pre-trained model. As the paper notes in Section 2, "these methods suffer from the requirements of TM training. Our approach does not require training or knowledge of the training procedure. It requires only a pre-trained model and a dataset of concepts."

Image processing has shown neuron-level interpretability works, but NLP lags behind. A crucial motivation for this paper comes from the computer vision literature, specifically Network Dissection (Bau et al., 2017) and GAN Dissection (Bau et al., 2019). These works demonstrated that individual CNN filters in image classifiers and individual units in GAN generators correspond to specific objects (trees, doors, windows, etc.) — even when those objects were never explicitly labeled during training. The method worked by treating CNN feature maps as segmentation masks and computing intersection-over-union (IoU) with densely labeled images. The paper explicitly draws on this inspiration but identifies three fundamental differences that make the vision approach inapplicable to NLP (Section 1):

1. Concepts can't be defined by single sentences. In vision, an image either contains a "tree" or it doesn't, and a single labeled image suffices to represent the concept. In NLP, a single sentence is too noisy to define whether a concept is present — the word "bank" might refer to a financial institution or a river bank in the same sentence structure. The paper proposes representing concepts with sets of positive and negative sentences (Section 3), collecting 100–1,000 sentences per concept.

2. Neurons, not filters, are the units of analysis. In CNNs, filters produce spatial feature maps that can be aligned with pixel-level labels. In Transformers, the most basic computational units are individual neurons in the feed-forward layers (the linear projections A, A_proj, B, B_proj in each Transformer block, as illustrated in Figure 2). These neurons produce scalar activations per token position, not spatial maps. The paper handles this by max-pooling each neuron's responses across the temporal (token) dimension to produce a single scalar per sentence, making the neuron a sentence-level classifier.

3. Average precision (AP) replaces IoU as the evaluation metric. Because sentences are of variable length and concept presence is a sentence-level property (not a spatial one), IoU across a segmentation mask doesn't apply. Instead, the paper treats each neuron's scalar responses across a set of sentences as prediction scores for a binary classification task (concept present vs. absent) and computes the area under the precision-recall curve (average precision, AP) as the measure of how well that neuron disambiguates the concept.

Where Existing Approaches Fall Short

The paper identifies several specific gaps in the interpretability landscape that it aims to fill:

No method existed for finding concept-specific neurons in pre-trained Transformers without training auxiliary models. The probing literature trains classifiers; the saliency literature visualizes attention; the disentanglement literature modifies training. None of these approaches can take an arbitrary pre-trained Transformer and answer the question: "Which individual neurons in this model have learned to detect concept X?" The paper's core methodological contribution (Section 4.1) is a method that does exactly this: run a forward pass over a concept's dataset, max-pool each neuron's responses across time, and compute AP against the binary labels. No training, no auxiliary models, no architectural modifications.

No systematic study linked the presence of expert neurons to model generalization. Prior work had observed specialized units in specific models on specific concepts, but no one had compiled a large enough concept dataset and analyzed enough models to establish a statistical relationship between the density of expert units and downstream task performance. The paper's finding that concept expertise X_γ⋆ correlates with GLUE+SQuAD average performance at r² = 0.833 (Table 2) — and that this correlation is higher than the average inter-task correlation (r² = 0.826) — is the first quantitative evidence that neuron-level specialization is a meaningful proxy for model quality.

No method exploited pre-existing expert neurons for controllable generation. Prior work on conditional language modeling either trained conditional architectures from scratch (CTRL; Keskar et al., 2019) or fine-tuned models with auxiliary objectives (Hu et al., 2017). The idea that you could take an off-the-shelf GPT2 model, identify the 50–200 neurons (out of 414,720) that best classify a concept like bird%1:05:00, and force those neurons to fire during decoding to induce that concept in the output — without any retraining — was entirely novel. The paper explicitly claims this as a first: "To the best of our knowledge, our work is the first to condition an off-the-shelf pre-trained LM without fine-tuning, re-training or using additional parameters" (Section 1).

Concept co-learning had no quantitative measure in NLP. While the image processing literature had observed that individual filters can represent multiple related objects (Bau et al., 2017, 2019; Fong and Vedaldi, 2018), and that these overlaps reveal semantically meaningful relationships, there was no analogous method for NLP. The paper's concept overlap metric Ω(q, v) (Section 5, Equation 4) — the Jaccard index between the top 1% of neurons for two concepts — provides a simple, interpretable measure of how much two concepts share expert units. The finding that chair%1:06:00 (a seat for one person) overlaps heavily with table%1:06:01, bed%1:06:00, and cup%1:06:00 (Table 3) — all furniture items — demonstrates that this overlap captures genuine semantic similarity, not just word-level co-occurrence.

Homograph disambiguation was an unresolved challenge. The paper distinguishes between "sense" concepts (positive sentences contain a keyword with a specific meaning; negative sentences don't contain the keyword at all) and "homograph" concepts (positive sentences contain a keyword with meaning A; negative sentences contain the same keyword with meaning B). Finding neurons that can tell "bank" (financial) from "bank" (river) is fundamentally harder than finding neurons that detect "bank" (financial) vs. unrelated sentences — and the paper's results confirm this: across all models, homograph expertise X_γ=0.985 peaks at only 5.72% for BERT-B/L and DistilBERT, compared to sense expertise reaching 17.86% for RoBERTa-Lm (Table 1). The paper identifies this as an open challenge: "Building pre-trained models inherently able to disambiguate homograph concepts at unit level remains a challenge, and we speculate that such knowledge will help the models generalize even better" (Section 4.3.2).

How This Paper Positions Itself

The paper positions itself at the intersection of model interpretability, model evaluation, and controllable generation, offering a unified method that serves all three goals without requiring training. The key design choices that distinguish this work:

Data-driven concept representation. Rather than defining concepts through rules or single keywords, the paper builds a dataset (Section 3) where each concept is represented by 100–1,000 positive sentences (containing the concept) and 100–1,000 negative sentences (not containing the concept). This flexible definition allows representing diverse concept types — not just concrete nouns, but abstract ideas like inspiration%1:09:02, frustration%1:12:00, and integrity%1:26:00 — all drawn from the OneSec dataset (Scarlini et al., 2019) with WordNet sense annotations. The full concept list (Appendix E, Tables 10–14) spans 1,344 sense concepts and 297 homograph concepts, covering a broad semantic range.

Neuron-as-classifier paradigm. The central methodological move is treating every neuron in a pre-trained Transformer as an independent binary classifier for every concept, using its scalar responses (max-pooled across token positions) as prediction scores. This is fundamentally different from probing (which trains classifiers on top of representations) and from attention analysis (which examines interaction patterns). It is also computationally demanding — evaluating GPT2-L's 414,720 neurons across 1,641 concepts requires storing approximately 2.5 GB of AP scores — but the paper argues this is comparable to or cheaper than the fine-tuning evaluations it aims to replace (Section 7).

Causal intervention, not just correlation. The paper goes beyond identifying expert neurons to demonstrating that they are causal for concept generation. The conditional generation experiments in Section 6 force the top-K expert neurons for a concept to remain active during nucleus sampling (Holtzman et al., 2019), and show that increasing K monotonically increases the presence of the concept in the output — up to a saturation point where the concept dominates and the text becomes degenerate (Table 4). This is a stronger claim than correlation: it shows that the identified neurons control concept expression, not just correlate with it. The paper frames this through a product-of-experts formulation (Hinton, 1999; Nguyen et al., 2017), where p(y = c|x) is the internal condition expert and p(x) is the language model prior — and forcing the top experts amplifies p(y = c|x) relative to p(x) in the joint distribution (Equation 5).

Interpretability without architectural assumptions. Unlike methods that require attention weights or specific layer types, the approach treats all neuron-like units uniformly — the linear layers A, A_proj, B, and B_proj in each Transformer block (Figure 2) — and discovers which ones happen to be experts. This makes it applicable to any Transformer-based model without modification, including encoder-only (BERT, RoBERTa), decoder-only (GPT2), and cross-lingual (XLM) architectures. The results in Figure 3 and Appendix A show that B layers (the second feed-forward expansion in each block) acquire roughly 3.5× more concepts than A layers, suggesting that the method not only finds experts but reveals architectural properties of how Transformers organize knowledge — a finding that would be invisible to methods focused only on attention.

3. Technical Approach

3.1 Reader orientation

This paper builds a system for discovering and exploiting individual neurons in pre-trained Transformer models that act as specialized concept detectors — what the authors call "expert units." The problem it solves is twofold: first, how to find which neurons (out of hundreds of thousands) have learned to detect specific concepts like bird%1:05:00 (warm-blooded egg-laying vertebrates) or chair%1:06:00 (a seat with a back), without training any auxiliary models or modifying the architecture; second, how to use these discovered experts to evaluate model quality, understand concept relationships, and control text generation — all on off-the-shelf pre-trained models with no fine-tuning.

3.2 Big-picture architecture (diagram in words)

The system has five major components:

1. SentenceConcepts Dataset — a collection of 1,641 concepts (1,344 sense + 297 homograph), each represented by 100–1,000 positive sentences (containing the concept) and 100–1,000 negative sentences (not containing it), drawn from the OneSec dataset with WordNet sense annotations.

2. Unit Response Extractor — for every sentence and every neuron in the model (the linear layers A, A_proj, B, B_proj in each Transformer block), this runs a forward pass through the pre-trained model, extracts the intermediate representation at that layer, max-pools across the temporal (token) dimension, and records a single scalar response per neuron per sentence.

3. Expert Unit Ranker — for each concept and each neuron, this treats the neuron's scalar responses across all sentences as prediction scores for a binary classification task (concept present vs. absent) and computes Average Precision (AP) — the area under the precision-recall curve. Neurons are ranked by AP for each concept; those exceeding an acquisition threshold γ are considered "experts" for that concept.

4. Concept Expertise Calculator — aggregates across all concepts to compute X_γ, the percentage of concepts for which the model contains at least one expert neuron exceeding threshold γ. The optimal threshold γ⋆ is learned by maximizing the correlation between X_γ and downstream task performance, producing separate thresholds for sense (γ⋆ = 0.997) and homograph (γ⋆ = 0.985) concepts, and a combined expertise metric X_γ⋆.

5. Conditional Generator — for a given target concept, identifies the top-K expert neurons (by AP), then during nucleus sampling, forces those neurons to fire at their median active value (computed from the positive sentences) at every generation step, amplifying the internal condition expert p(y = c|x) relative to the language model prior p(x) in a product-of-experts formulation.

Information flows as follows: the SentenceConcepts dataset provides positive and negative sentences for each concept → the Unit Response Extractor runs a forward pass for every sentence through a frozen pre-trained model and records max-pooled scalar responses for every neuron in the specified layers → the Expert Unit Ranker computes AP per (neuron, concept) pair, producing a matrix of size M neurons × N concepts → for evaluation, the Concept Expertise Calculator aggregates these scores into a single model-level metric X_γ⋆; for generation, the Conditional Generator selects the top-K neurons for a target concept and clamps their activations during decoding.

3.3 Roadmap for the deep dive

• First, the SentenceConcepts dataset — how concepts are defined, the structure of positive/negative sentences, the two concept categories (sense and homograph), and why this representation is necessary for NLP (unlike vision, where single images suffice). This is the foundation on which all expert discovery depends.

• Second, the unit response extraction procedure — which layers are analyzed (the A, A_proj, B, B_proj linear projections in each Transformer block, illustrated in Figure 2), how activations are converted from per-token sequences to per-sentence scalars (max-pooling), and the resulting representation of each neuron as a vector of scalar responses across sentences.

• Third, the expert unit ranking method — how a neuron is treated as a binary classifier, how Average Precision quantifies its expertise for a concept, why AP rather than accuracy or F1, and the resulting M × N matrix of AP scores that enables all downstream analyses.

• Fourth, the concept expertise metric X_γ and the procedure for finding the optimal acquisition threshold γ⋆ — how γ controls what counts as an "acquired" concept, how γ⋆ is chosen to maximize correlation with downstream task performance, the cross-validation procedure, and the combined expertise formula (Equation 3).

• Fifth, the concept overlap metric Ω — how the top 1% of neurons per concept are used as a binary concept representation, how the Jaccard index between these representations quantifies co-learning, and what this reveals about semantic relationships in the model's internal organization.

• Sixth, the conditional generation mechanism — the product-of-experts formulation (Equation 5), how expert forcing works (clamping top-K neurons to their median active value during decoding), why this provides causal evidence for expertise, and the practical implementation details (nucleus sampling, choice of K, saturation behavior).

3.4 Detailed, sentence-based technical breakdown

This is primarily a methodological and empirical analysis paper whose core idea is that individual neurons in pre-trained Transformers act as specialized concept classifiers, that the density of such specialists predicts model generalization, and that these specialists can be directly manipulated to control generation — all without any training or architectural modification.

---

The SentenceConcepts Dataset: Representing Concepts via Sentence Sets

The fundamental challenge in NLP interpretability is that a concept cannot be adequately represented by a single sentence. If you want to know whether a neuron detects the concept "financial institution," showing it one sentence containing "bank" is insufficient — the sentence might use "bank" in the river sense, and even if it's the financial sense, one example doesn't establish that the neuron generalizes. The paper solves this by representing each concept as a set of positive sentences (containing the concept) and a set of negative sentences (not containing it), constructed from the OneSec dataset (Scarlini et al., 2019).

Data source and annotation structure. The OneSec dataset provides sentences where one keyword has been annotated with a WordNet sense identifier of the form lemma%A:BB:CC (Section 3). For example, a sentence containing the word "shelter" might be annotated with shelter%1:06:00::, where the part before the % is the lemma and the remaining numbers uniquely identify the concept in WordNet. The authors parse all sentences for a given sense ID to create the positive set for each concept, only retaining those sense IDs with more than 100 available sentences.

Two concept categories. The paper distinguishes two fundamentally different types of concepts (Section 3):

1. Sense concepts (1,344 total): Positive sentences contain a keyword with a specific WordNet sense; negative sentences contain different keywords entirely (different lemmas). For example, positive sentences for note%1:10:00 (a brief written record) contain the word "note" used in the "written record" sense; negative sentences are randomly sampled from all sentences whose annotated keyword has a completely different lemma — they might contain "dog," "run," "theory," etc., but not "note" at all. This makes sense concepts relatively easy to detect: the neuron just needs to learn that the presence of the word "note" (or related context words) predicts the concept.

2. Homograph concepts (297 total): Positive sentences contain a keyword with a specific WordNet sense; negative sentences contain the same keyword but with a different meaning. For example, the homograph concept note%1:10:00 VS. note%1:10:01 has positive sentences where "note" means "a brief written record" and negative sentences where "note" means "a musical tone." The keyword is the same string in both cases. This makes homograph concepts fundamentally harder: the neuron cannot rely on the mere presence or absence of a word — it must detect the contextual meaning to distinguish the two senses.

The paper explicitly notes that "homograph concepts are harder to disambiguate than sense concepts" (Section 3), and this is borne out in the results: across all models, the highest homograph expertise achieved is 5.72% (for BERT-B and DistilBERT, with γ = 0.985), compared to 17.86% for sense concepts with RoBERTa-Lm (Table 1). The ability to disambiguate homographs — requiring genuine semantic understanding rather than keyword detection — is identified as an open challenge.

Dataset size constraints. The number of sentences per concept is "constrained by availability in the source dataset" and limited to N_c⁺, N_c⁻ ∈ [100, 1000], with random subsampling when more than 1,000 sentences are available (Section 3). This means each concept has between 200 and 2,000 total sentences, with an average of approximately 1,500 (the paper states the dataset contains ~2.5M sentences for 1,641 concepts in Section 7). This is a substantial dataset — 2.5 million sentences — but it is "weakly annotated" (Section 7), meaning the WordNet sense labels come from the OneSec annotation process, not from manual verification of every sentence. The paper acknowledges that "there are inconsistencies inherent in the source OneSec dataset" and that "the more diverse and accurate the concept dataset, the better it will help evidence the generalization power of TMs."

Why this representation matters. The sentence-sets approach is what makes the entire expert unit discovery pipeline possible. Without a dataset of positive and negative examples per concept, there is no way to measure whether a neuron's responses discriminate the concept. The flexibility of this definition — any concept that can be represented by a set of containing and not-containing sentences — allows the method to cover diverse concept types: concrete objects (bird%1:05:00, elevator%1:06:00), abstract ideas (inspiration%1:09:02, frustration%1:12:00), professions (journalist%1:18:00, farmer%1:18:00), locations (brazil%1:15:00, london%1:15:00), and actions (fishing%1:04:00, drinking%1:04:00). The complete list in Appendix E (Tables 10–14) spans 1,344 sense concepts and 297 homograph concepts, sorted by the AP⋆_c achieved by GPT2-L.

---

Unit Response Extraction: From Tokens to Scalars

Having defined concepts as sets of sentences, the next step is to extract from the pre-trained Transformer a single scalar value per neuron per sentence, representing how strongly that neuron responds to that sentence. This requires specifying which units to analyze, how to aggregate across the variable-length token dimension, and what the resulting data structure looks like.

Which units are analyzed. The paper analyzes the neurons in the linear (fully-connected) layers within each Transformer block, not the attention weights or the embedding layer. Specifically, as illustrated in Figure 2, each Transformer block contains four linear projection layers:

• A and B: The two expanding feed-forward layers (the first expands from the embedding dimension D to a larger intermediate dimension, typically 4D; the second projects back from 4D to D).
• A_proj and B_proj: The linear projections within the multi-head self-attention mechanism.

For a model like GPT2-L (D = 1,280, 36 blocks), the total number of analyzed units is 36 blocks × 9D = 36 × 9 × 1,280 = 414,720 neurons (the factor of 9D comes from the specific layer structure: A, A_proj, B, and B_proj together contribute 9D neurons per block; the paper doesn't break this down further but the total is stated explicitly in the caption of Figure 1). For RoBERTa-L (D = 1,024, 24 blocks in the base configuration, but large uses a different architecture), the paper analyzes 221,184 units (stated in Section 4.3.2).

Why these layers? The paper focuses on the linear layers because they contain the most basic computational units — individual neurons that apply a learned weight vector to their input and produce a scalar output (before the non-linearity). Attention layers, by contrast, produce attention weight matrices that describe interactions between tokens rather than single-neuron responses. The linear layer neurons are the closest analog to the CNN filters studied in Network Dissection (Bau et al., 2017) and GAN Dissection (Bau et al., 2019) — they are scalar feature detectors that can be evaluated independently.

The max-pooling operation. A sentence $x_i$ consists of an arbitrary number $T$ of tokens $x_{i,t} \in \mathbb{R}^D$, where $D$ is the token embedding dimensionality. When this sentence passes through layer $\ell$ of the Transformer, it produces an intermediate representation $z^\ell_i = [z^\ell_{i,1}, \cdots, z^\ell_{i,T}] \in \mathbb{R}^{D_\ell \times T}$, where $D_\ell$ is the dimensionality of that layer's output (typically a multiple of $D$). Each column of this matrix is the layer's activation vector at a specific token position.

To convert this per-token representation into a per-sentence scalar for each neuron, the paper applies max-pooling across the temporal dimension:

$[u^{(\ell,1)}_i, \cdots, u^{(\ell,D_\ell)}_i]^\top = \text{maxpool}(z^\ell_i, \text{axis}=1) \in \mathbb{R}^{D_\ell}$

where $u^{(\ell,k)}_i \in \mathbb{R}$ is the response of unit $k$ in layer $\ell$ to sentence $i$, $z^\ell_i$ is the layer's output tensor of shape $D_\ell \times T$, and $\text{maxpool}(\cdot, \text{axis}=1)$ takes the maximum value across all $T$ token positions independently for each of the $D_\ell$ feature dimensions.

What it computes: For each neuron in the layer, look at its activation value at every token position in the sentence, take the maximum of those values, and record that maximum as the neuron's response to the sentence. The result is a vector of length $D_\ell$ — one scalar per neuron — representing the strongest response each neuron produced anywhere in the sentence.

Why max-pooling? The paper states the reason explicitly: "since sentences can be of arbitrary length, we maxpool the unit responses in time to be invariant to length" (Section 4.1). Alternative choices would have different properties:

• Mean pooling would average the neuron's response over all tokens, which would dilute strong but localized responses — a neuron that fires intensely on one token but is silent on all others would have a low mean, even though it clearly detected something relevant.
• Taking the response at a specific position (e.g., the [CLS] token in BERT) would discard information from other positions and wouldn't work for decoder-only models like GPT2 that don't have a designated summary token.
• Concatenating all positions would produce variable-length representations that can't be compared across sentences.

Max-pooling makes the assumption that if a concept is present in a sentence, the expert neuron for that concept should fire strongly on at least one token — likely the token representing the concept word or a related contextual cue. This is a reasonable assumption for keyword-associated concepts: a neuron that detects "financial bank" should activate somewhere in a sentence about banking, even if it's not active on every token.

Indexing simplification. For notational convenience, the paper replaces the $(\ell, k)$ two-dimensional indexing with a single index $m = 1..M$, where $M$ is the total number of units analyzed across all layers. So $u^m_i$ is the response of unit $m$ to sentence $i$, and $u^m_c \in \mathbb{R}^{N_c}$ (where $N_c = N^+_c + N^-_c$) is the vector of responses of unit $m$ to all sentences representing concept $c$. This flattening makes it possible to treat the entire model as a flat collection of $M$ binary classifiers, each producing a scalar score for every sentence.

What the extraction produces. After running forward passes for all sentences across all concepts, the system has, for each model, a data structure that can be thought of as a matrix of size $M \times \sum_c N_c$: one row per neuron, one column per sentence, with the entry being the max-pooled response of that neuron to that sentence. For GPT2-L with $M = 414,720$ and the dataset containing ~2.5M sentences, this matrix would be ~4.1 TB if stored densely, but in practice the AP computation (described next) can be done incrementally per concept and per neuron, requiring only the responses for one neuron across one concept's sentences at a time. The paper states that the computed AP scores require "414,720 × 1,641 floats ≈ 2.5 GB plus overhead" for GPT2-L (Section 7).

Compute requirements. The forward pass over the dataset is the most computationally demanding step. The paper provides a concrete estimate: "According to the benchmark in the Transformers repository, the average GPU inference time for BERT-B for sentences of 128 tokens is 9ms, which translates into 6.15 hours for the whole dataset" (Section 7). With 8 GPUs and per-concept parallelization, this reduces to 45 minutes. For comparison, "a single evaluation of BERT-B on SQuAD v1.1 takes 24min. But several evaluations are required for hyper-parameter tuning and statistical significance. Summing up, evaluating on SQuAD v1.1/2.0 plus all GLUE tasks is more demanding than our proposed evaluation" — so the method is not only more informative (it produces concept-level interpretability, not just a task score) but also computationally competitive with the standard evaluation pipeline.

---

Expert Unit Ranking: Neurons as Binary Classifiers

Once the scalar responses $u^m_c \in \mathbb{R}^{N_c}$ are extracted for unit $m$ on concept $c$'s sentences, the paper treats the neuron as a binary classifier and the responses as prediction scores. The binary labels $b_c \in \mathbb{Z}_2^{N_c}$ are 1 for positive sentences (containing concept $c$) and 0 for negative sentences (not containing concept $c$, or containing it in a different sense for homographs).

$AP^m_c = AP(u^m_c, b_c) \in [0, 1]$

where $u^m_c$ is the vector of $N_c$ scalar responses from unit $m$ to the sentences of concept $c$, $b_c$ is the binary ground-truth vector of length $N_c$, and $AP(\cdot, \cdot)$ is the function that computes the area under the precision-recall curve.

What it computes: For a single neuron and a single concept, sort all $N_c$ sentences by the neuron's response (highest to lowest), then sweep a decision threshold from the highest response to the lowest. At each threshold, compute precision (fraction of sentences above threshold that are truly positive) and recall (fraction of all positive sentences that are above threshold). The average precision is the area under this precision-recall curve — a value between 0 and 1, where 1.0 means the neuron perfectly separates positive from negative sentences (all positives have higher responses than all negatives) and 0.5 means random guessing. The result is a single scalar per (neuron, concept) pair.

Why Average Precision rather than accuracy or F1? The paper doesn't explicitly justify this choice, but the reasoning follows from the problem structure:

• Accuracy would require choosing a single threshold for what response counts as a "positive prediction," which is arbitrary — what threshold should count as "the neuron is active"? The precision-recall curve summarizes performance across all possible thresholds, providing a threshold-free measure of discriminability.
• F1 score similarly depends on a chosen threshold and balances precision and recall at a specific operating point. For expert discovery, we want to know whether there exists any threshold at which the neuron is a good discriminator, not whether it performs well at a pre-specified threshold.
• ROC-AUC (area under the ROC curve) would be an alternative, but precision-recall is more informative when the classes are imbalanced — if there are many more negative than positive sentences (which can happen given the 100–1000 sentence range), a neuron might achieve high ROC-AUC by simply ranking a few positives correctly while ignoring many false positives, whereas AP penalizes false positives more heavily in imbalanced settings.

The choice of AP means that a neuron earns a high score only if it can achieve both high precision and high recall simultaneously across the full range of possible decision thresholds — it must not only detect the concept when it's present (high recall) but also avoid false alarms when it's absent (high precision).

What the ranking produces. For each concept $c$, the system computes $AP^m_c$ for every neuron $m = 1..M$ in the model, then ranks neurons by their AP for that concept. The best expert for concept $c$ has:

$AP^\star_c = \max_m\{AP^m_c\}$

where $AP^\star_c$ is the AP of the single best neuron for concept $c$, and the max is taken over all $M$ neurons in the model.

This per-concept best-expert score $AP^\star_c$ is the key quantity for all subsequent analyses. It tells us, for each concept, whether the model contains at least one neuron that can reliably discriminate that concept. Concepts with high $AP^\star_c$ are "acquired" by the model in the sense that some neuron has specialized to detect them; concepts with low $AP^\star_c$ are not represented at the individual neuron level (though they might be represented in distributed form across multiple neurons — a limitation the paper acknowledges in Section 7: "It is possible, but not yet explored, that more complex concepts such as homograph, require a more complex expert such as a set of units").

The search space is enormous. For GPT2-L with $M = 414,720$ neurons and the dataset containing 1,641 concepts, there are $414,720 \times 1,641 \approx 680$ million (neuron, concept) pairs to evaluate. The paper frames this enormous search space as a feature, not a bug: "We expect that, given the large search space, certain classifiers will perform well on specific concepts: the expert units" (Section 4.1). The vast majority of neurons will have near-random AP for any given concept; the existence of any neuron with high AP is what makes a concept "acquired." This is analogous to the lottery ticket hypothesis — most parameters are irrelevant for any particular function, but a sparse subset is highly specialized.

---

Concept Expertise X_γ: Quantifying Model-Level Specialization

The central evaluation metric of the paper is concept expertise $X_\gamma$, defined as the percentage of concepts for which the model contains at least one expert neuron exceeding an acquisition threshold $\gamma$:

$X_\gamma = \frac{|\{c \text{ s.t. } AP^\star_c \geq \gamma\}|}{|C|} \quad \forall c \in C$

where $C$ is the set of all concepts (sense or homograph, treated separately), $AP^\star_c$ is the AP of the best expert for concept $c$, and $\gamma \in [0.5, 1)$ is the acquisition threshold.

What it computes: Count how many concepts have at least one neuron with AP ≥ γ, divide by the total number of concepts. The result is a percentage between 0 and 100%. If $X_\gamma = 15\%$, it means that for 15% of all tested concepts, the model contains at least one neuron that can classify that concept with precision at or above $\gamma$ (averaged across all thresholds via the AP metric).

Why this form matters. The expertise metric captures two important properties simultaneously:

1. Diversity of expertise: A model with high $X_\gamma$ has expert neurons for many different concepts, not just a few — it has developed specialized detectors across a broad semantic range.
2. Quality of expertise: The threshold $\gamma$ enforces a minimum discriminability — a model only gets "credit" for a concept if its best neuron is genuinely good at detecting it, not just marginally better than random. As the paper finds, the optimal $\gamma$ values are extremely high (0.997 for sense, 0.985 for homograph), meaning only near-perfect experts count.

An alternative formulation — say, averaging AP over all concepts — would be dominated by the many concepts with low AP and wouldn't capture whether the model has any good experts for the concepts it handles. The binary threshold at $\gamma$ focuses attention on the presence of high-quality specialists, which is the paper's hypothesis about what matters for generalization.

Why separate sense and homograph? The paper treats sense and homograph concepts independently "since they are fundamentally different" (Section 4.2). A neuron detecting "bank" (financial) vs. completely unrelated words is solving a keyword-detection problem; a neuron detecting "bank" (financial) vs. "bank" (river) is solving a contextual disambiguation problem. Pooling these together would obscure the fact that models might excel at one and fail at the other. The separate treatment allows the paper to report that sense expertise reaches 17.86% for RoBERTa-Lm while homograph expertise peaks at only 5.72%, providing a more nuanced picture of model capabilities.

---

Finding the Optimal Acquisition Threshold γ⋆

The choice of $\gamma$ is critical — set it too low, and every model gets credit for concepts it barely detects; set it too high, and no model has any experts. The paper proposes an empirical method to find the $\gamma$ that makes $X_\gamma$ most predictive of downstream task performance, turning the threshold selection from an arbitrary choice into a data-driven optimization.

The optimization objective. The paper measures generalization as the average performance across diverse downstream tasks: all 10 datasets in GLUE (with their different reported metrics) plus SQuAD v1.1 and v2.0. The performance numbers used are reported in Table 5 (Appendix B) and come from the original model papers — they are not re-evaluated by the authors. The goal is to find $\gamma$ that maximizes the correlation between $X_\gamma$ and these task scores:

$\gamma^\star = \arg\max_\gamma \left( \frac{1}{N_{\text{tasks}}} \sum_{\text{task}} r^2(X_\gamma, \text{task}) \right)$

where $N_{\text{tasks}}$ is the number of downstream tasks/metrics, $r^2$ is the squared Pearson correlation coefficient between $X_\gamma$ and the performance on each task, and $\gamma \in [0.5, 1)$.

What it computes: For each candidate threshold $\gamma$ (presumably swept over a fine grid from 0.5 to 1.0, though the paper doesn't specify the granularity), compute the concept expertise $X_\gamma$ for each model, then compute the squared Pearson correlation between $X_\gamma$ and performance on each individual downstream task. Average these $r^2$ values across all tasks, and pick the $\gamma$ that gives the highest average correlation. The result is $\gamma^\star$ — the threshold that makes expertise most predictive of generalization.

Why squared Pearson correlation ($r^2$)? The $r^2$ measures the fraction of variance in task performance that can be explained by a linear relationship with $X_\gamma$. Using $r^2$ rather than raw Pearson $r$ emphasizes stronger correlations (squaring compresses values near zero and spreads values near ±1). Averaging $r^2$ across tasks (rather than computing a single correlation with the average task score) ensures that the chosen $\gamma$ works well for individual tasks, not just the aggregate — a threshold that correlates well with the average but poorly with several individual tasks would have a lower average $r^2$.

Cross-validation for robustness. To assess whether $\gamma^\star$ overfits to the specific set of tasks used, the paper employs a cross-validation procedure: the tasks are "randomly split into reference and test sets, with a ratio 60/40%." For each of 10 random splits, $\gamma^\star$ is computed on the reference set and its RMSE (root mean square error) relative to the $\gamma^\star$ computed on the test set is measured. The paper reports:

• For sense concepts: $\gamma^\star = 0.997$ with RMSE = 0.0004
• For homograph concepts: $\gamma^\star = 0.985$ with RMSE = 0.0028

The extremely low RMSE values (especially for sense, where the error is 0.0004 — meaning $\gamma^\star$ varies by less than 0.001 across splits) indicate that the optimal threshold is stable and not an artifact of the particular task suite. This is a crucial finding: it means that once $\gamma^\star$ is determined on one set of tasks, it can be used to evaluate new models on new tasks without re-optimizing.

What the high γ⋆ values mean. The fact that $\gamma^\star = 0.997$ for sense concepts is striking — it means that for a concept to "count" toward a model's expertise, the model must contain a neuron that classifies it with near-perfect average precision (0.997 is essentially 1.0, allowing only the tiniest fraction of ranking errors). The homograph threshold of 0.985 is slightly lower, reflecting the greater difficulty of homograph disambiguation, but is still extremely stringent. Together with the strong resulting correlation ($r^2 = 0.833$ with generalization), this suggests that "generalization is related to the presence of extremely good and diverse experts" (Section 4.2) — it's not enough to have mediocre detectors for many concepts; the model needs a collection of near-flawless specialists.

Combined expertise formula. With the optimal thresholds determined separately for sense and homograph, the paper defines a combined expertise metric that weights each concept type by its number of concepts:

$X_{\gamma^\star} = \frac{1}{N_{\text{concepts}}} \sum_{g \in (\text{sense}, \text{homograph})} N_g X^g_{\gamma=\gamma^\star[g]}$

where $N_{\text{concepts}}$ is the total number of concepts (1,641), $g$ iterates over the two concept types, $N_g$ is the number of concepts of type $g$ (1,344 for sense, 297 for homograph), and $\gamma^\star[g]$ is the optimal threshold for that type (0.997 for sense, 0.985 for homograph).

What it computes: A weighted average of the sense expertise at γ = 0.997 and the homograph expertise at γ = 0.985. Since there are many more sense concepts, the combined metric is dominated by sense expertise (by a factor of 1,344/297 ≈ 4.5×), but homograph performance still contributes.

Why a weighted average rather than treating all concepts equally? If all 1,641 concepts were pooled together regardless of type, the optimal $\gamma$ would be a compromise that fits neither type well, and the combined metric would obscure the fact that sense and homograph concepts represent fundamentally different challenges. By fitting separate thresholds and then averaging, the metric captures both capabilities appropriately.

---

Concept Overlap Ω: Quantifying Co-Learning

The expert unit ranking produces more than just a per-concept best-expert score — it produces a full vector of AP scores for every concept across all $M$ neurons. This high-dimensional representation enables the paper to ask: do related concepts share expert neurons? If a neuron is an expert for chair%1:06:00 (a seat with a back), is it also good at detecting table%1:06:01 (a piece of furniture with a flat top)? The concept overlap metric $\Omega$ quantifies this co-learning.

Constructing binary concept representations. For each concept $c$, the paper first sets a threshold to select the top experts:

$\tau_c = \text{percentile}_{99}(AP^m_c)$

This is the 99th percentile of AP scores across all $M$ neurons for concept $c$ — the AP value that only 1% of neurons exceed. Then a binary vector $s_c \in \mathbb{Z}_2^M$ is defined:

$s_c = \{\mathbb{1} \text{ if } AP^m_c > \tau_c \text{ else } 0\}$

where $s_c$ has a 1 in position $m$ if neuron $m$ is in the top 1% of all neurons for classifying concept $c$, and 0 otherwise.

What this represents: Each concept is now represented not by its WordNet definition or its sentences, but by the set of neurons that are its best classifiers — the top 1% of the model's units that most strongly discriminate this concept. Two concepts with similar neural signatures (overlapping top experts) are "co-learnt" by the model — they rely on the same specialized neurons.

The overlap formula. The overlap between concepts $q$ and $v$ is the Jaccard index (intersection over union) of their binary concept representations:

$\Omega(q, v) = \frac{\|s_q \cap s_v\|_1}{\|s_q \cup s_v\|_1} \in [0, 1]$

where $\| \cdot \|_1$ is the L1 norm (counting the number of 1s in the binary vector), $s_q \cap s_v$ is element-wise logical AND (neurons that are top-1% for both concepts), and $s_q \cup s_v$ is element-wise logical OR (neurons that are top-1% for at least one of the two concepts).

What it computes: Count how many neurons are in the top 1% for both concepts, divide by how many neurons are in the top 1% for either concept. If the two sets of top experts are identical, $\Omega = 1$; if they are completely disjoint, $\Omega = 0$. An overlap of 0.458 (as in the chair%1:06:00 / table%1:06:01 example from Table 3) means that nearly half of the union of top experts is shared — these concepts are substantially co-learnt.

Why the Jaccard index? Alternatives would include:

• Simple overlap count $\|s_q \cap s_v\|_1$: This would be sensitive to the absolute number of experts, not the relative overlap. A concept with many experts would show high overlap with everything.
• Cosine similarity between the full AP vectors (not thresholded): This would incorporate all neurons, including the 99% that are poor classifiers, diluting the signal from the true experts.
• Correlation of AP scores: This would capture whether high-AP neurons for one concept also tend to be high-AP for another, but wouldn't focus specifically on the top experts.

The Jaccard index on the top 1% focuses squarely on the question: "Do these two concepts share the same specialist neurons?" The choice of top 1% (rather than, say, top 5% or top 0.1%) is a hyperparameter that trades off between including enough neurons to capture meaningful overlap and being selective enough to focus on genuine experts. The paper does not explore sensitivity to this choice.

Why threshold at the 99th percentile per concept rather than at a fixed AP value? If a fixed AP threshold were used (e.g., AP > 0.95), concepts that are inherently harder to classify would have few or no neurons meeting the threshold, and their binary representations would be mostly zeros — making overlap with easy concepts artificially low. By using the 99th percentile per concept, every concept gets exactly 1% of neurons as its "top experts," ensuring that the representation is comparably sparse for all concepts and that the overlap metric captures relative specialization rather than absolute classification difficulty.

---

Conditional Generation via Expert Forcing

The most striking application of expert units is conditional text generation without any retraining. The paper demonstrates that forcing only the top-K expert neurons for a concept to remain active during nucleus sampling causes the model to generate text containing that concept — providing causal evidence that the identified neurons control concept expression, not just correlate with it.

The product-of-experts formulation. The theoretical foundation comes from Hinton (1999) and its adaptation to image generation by Nguyen et al. (2017). A language model learns the probability of a generated sentence $x$:

$p(x) = p(x_1, \ldots, x_T) = \prod_{k=1}^T p(x_k | x_{<k})$

where $x_k$ is the $k$-th token and $x_{<k}$ are all previous tokens.

A conditional generative model maximizes the joint distribution $p(x, y) = p(y|x)p(x)$, where $y$ is a latent conditional variable representing a specific concept. Interpreting this as a product of experts:

$p(x|y=c) \propto p(y=c|x) \, p(x)$

where $p(y=c|x)$ is the condition expert — a model that estimates how likely it is that concept $c$ is present in sentence $x$ — and $p(x)$ is the language model prior — the standard LM that ensures generated text is fluent and grammatical. The product means that generated sentences must simultaneously satisfy both experts: they must read like natural language (high $p(x)$) and contain concept $c$ (high $p(y=c|x)$).

The key hypothesis. Unlike Nguyen et al. (2017), who trained a separate neural network to model $p(y=c|x)$ as an external condition expert, this paper hypothesizes that "the condition expert $p(y=c|x)$ already exists within the same model" (Section 6) — that the pre-trained Transformer already contains neurons whose activation level reflects the presence of concept $c$, and that the model is capable of maximizing $p(x|y=c)$ by trusting these internal experts when generating.

Identifying the internal condition experts. The expert units discovered by the AP ranking procedure (Section 4.1) are the candidates for modeling $p(y=c|x)$. A neuron $m$ with high $AP^m_c$ has learned to produce high activations when concept $c$ is present and low activations when it is absent. If the model, during generation, can be made to produce high activations in those neurons, then — the hypothesis goes — the generated text will be more likely to contain concept $c$, because the model's own internal condition expert is signaling "this concept should be present."

The forcing mechanism. For a target concept $c$, the top-K expert neurons (those with the highest $AP^m_c$) are selected. During autoregressive decoding with nucleus sampling (Holtzman et al., 2019, with $p = 0.9$), at each generation step, the activations of these K neurons are forced to a specific target value rather than being allowed to vary naturally.

The target value for each expert neuron is computed as the median activation of that neuron across all positive sentences for concept $c$ in the dataset. The paper states: "the forcing value [is] computed as the median active value when concept $c$ is present in the input (not the output)" (Section 6). The median is used rather than the mean to be robust to outliers — if the neuron occasionally fires extremely strongly on a few sentences, the median captures its "typical" active state when the concept is present.

What this does operationally. During each step of autoregressive generation:

1. The model computes the next-token distribution as usual (through all Transformer layers, producing per-neuron activations at each layer).
2. For each of the K forced expert neurons, its activation is overridden: instead of using whatever value the network computed, it is set to its pre-computed median positive value.
3. This modified activation propagates through the rest of the network (for neurons in earlier layers, this affects subsequent layers' computations).
4. The model samples the next token from the nucleus-filtered distribution.

The larger $K$, the more the internal condition expert $p(y=c|x)$ is amplified relative to the language model prior $p(x)$. At low $K$, the generated text is fluent but may not contain the concept; at moderate $K$, the concept appears naturally in fluent text; at high $K$, the concept dominates and the text becomes degenerate (repetitive, ungrammatical) because $p(y=c|x)$ overwhelms $p(x)$.

Why this provides causal evidence. Correlation-based interpretability methods (including the AP ranking itself) can only show that a neuron's activity correlates with a concept. The forcing experiment shows that manipulating those neurons causes the concept to appear in the output — establishing a causal, not just correlational, relationship. If the identified neurons were merely correlated with the concept (e.g., they both respond to some confound like sentence length or word frequency), forcing them would not induce the concept. The fact that forcing as few as 40–200 neurons (0.009–0.048% of GPT2-L's 414,720 units) is sufficient to induce the target concept (Table 4) is strong evidence that these neurons are genuinely causal for concept expression.

The median-active-value choice. The paper adapts the forcing approach from Bau et al. (2019), who computed the mean filter response conditioned on an object being present in a GAN-generated image. The adaptation to language requires two changes:

1. We cannot quantify presence of $c$ in the output during generation, since the output doesn't exist yet — the model generates token by token. Instead, the forcing value is computed from the input sentences known to contain $c$ (the positive set), under the assumption that the same activation pattern that indicates concept presence in the input will also indicate concept presence in the output when the model is generating.
2. The median is used rather than the mean — though the paper doesn't explicitly justify this, it's standard practice for robust statistics: if the neuron occasionally fires at extreme values on some sentences, the mean would be pulled toward those extremes, potentially setting an unrealistic forcing target.

Why not force to the maximum? Forcing to the maximum observed activation would represent the strongest possible "concept present" signal, but might be outside the model's typical operating range, leading to unpredictable behavior. The median represents a "typical" active state — strong enough to signal concept presence, but within the distribution the model encountered during training.

What happens at saturation. As $K$ increases, the effect follows a dose-response curve: initially, the concept gradually appears more strongly in the output; eventually, the output saturates with the concept word, and further increases in $K$ produce degenerate, repetitive text. Table 4 shows this progression for concept bird%1:05:00:

• At K = 0 (no forcing): completely on-topic but unrelated text about teaching English.
• At K = 40 (0.009%): the text shifts to include horse-related content (related to animals, but not birds specifically).
• At K = 60 (0.015%): house sparrows appear, getting closer to the bird concept.
• At K = 80 (0.019%): chicken-like air wings appear — bird-related concepts are emerging.
• At K = 200 (0.048%): the text degenerates into repetitive bird-related words — "bird bird bird bird bird" — showing that $p(y=c|x)$ has completely dominated $p(x)$.

This saturation behavior empirically validates the product-of-experts formulation: increasing $K$ amplifies the condition expert, and at the saturation point, the language model prior is no longer sufficient to maintain fluency.

The forcing value as median. The paper is explicit that the forcing value is computed from input sentences, not output: "we cannot quantify the presence of $c$, however the output and the input of LMs are tightly related given the sequential decoding, which allows the forcing value to be computed as the median active value when concept $c$ is present in the input (not the output)" (Section 6). This leverages the autoregressive property of language models — each generated token becomes part of the input context for the next token — to justify using input statistics for output control.

---

Summary of Design Choices and Their Justifications

• Max-pooling over mean-pooling or position-specific extraction: provides length invariance and captures localized neural responses; a neuron that fires intensely on the concept keyword but is silent elsewhere still registers as responsive.
• Average Precision over accuracy, F1, or ROC-AUC: threshold-free evaluation that penalizes both false positives and false negatives across all possible operating points; appropriate for class-imbalanced settings where positive and negative sentence counts may differ.
• Separate γ⋆ for sense and homograph concepts: acknowledges the fundamental difficulty difference; pooling them would obscure that models handle keyword detection much better than contextual disambiguation.
• Squared Pearson correlation (r²) for threshold optimization: emphasizes strong linear relationships and compresses weak ones; averaging r² across individual tasks (rather than correlating with a task average) ensures γ⋆ works broadly.
• 99th percentile per-concept threshold for overlap analysis: ensures every concept has the same proportion of "top experts," making overlap scores comparable across concepts of different inherent difficulty.
• Median active value for forcing target: robust to outliers and represents a "typical" active state within the model's training distribution; avoids the extremes of mean (outlier-sensitive) or maximum (potentially out-of-distribution).
• Nucleus sampling (p = 0.9) for generation: standard approach (Holtzman et al., 2019) that truncates the low-probability tail of the distribution, balancing fluency and diversity; the forcing mechanism is orthogonal to the decoding strategy and could be combined with other methods.
• Product-of-experts formulation: provides a principled probabilistic framework for combining the language model prior with the internal condition expert; explains why increasing K amplifies concept presence and why saturation leads to degeneration.

4. Key Insights and Innovations

Innovation 1: The Neuron-as-Classifier Paradigm — Finding Interpretable Units Without Training Auxiliary Models

The paper's most fundamental conceptual move is reframing the interpretability problem from "what distributed representation encodes concept X?" to "which single neuron is the classifier for concept X?" Prior work in NLP interpretability overwhelmingly relied on training auxiliary models: probing classifiers (Conneau et al., 2018; Tenney et al., 2019; Liu et al., 2019) that learn linear or non-linear mappings from intermediate representations to task labels, or attention-based visualizations (Clark et al., 2019) that reveal token-token interaction patterns. Both paradigms treat the model as producing features that require external analysis to interpret.

This paper inverts that logic entirely. The central methodological move — treating every neuron's scalar responses across a concept's sentence set as prediction scores for a binary classifier, and evaluating via Average Precision — means that the model's own neurons are the classifiers, and the interpretability method is purely evaluative: it measures how good each neuron already is, without teaching it anything new. No weights are updated. No auxiliary model is trained. The "probe" is the neuron itself.

This is intellectually distinctive for several reasons:

It eliminates the probe-vs-model ambiguity. When a probing classifier achieves high accuracy on a linguistic task, there is a well-known confound: did the model actually encode that information, or did the probe learn to extract it from weak signals that the model itself doesn't use? The probing literature has wrestled with this through control tasks (Hewitt and Liang, 2019) and probe capacity limitations, but the ambiguity remains structural — a probe can always overfit. By contrast, when a single neuron achieves AP = 0.9999 on elevator%1:06:00 (as GPT2-L does; Table 10 in Appendix E), there is no probe to blame — the neuron's raw scalar outputs, thresholded at any value, nearly perfectly separate elevator-containing from non-elevator sentences. The model unambiguously represents this concept in a locally accessible way.

It discovers specialization that distributed methods miss. Probing typically finds that linear combinations of many neurons encode linguistic features — distributed representations. The expert unit approach finds that for many concepts, a single neuron does the job nearly perfectly. The paper's result that "most of the concepts have less than 50 dedicated expert units (0.022%), with a median of 7 experts (0.0032%) per concept" (Section 4.3.2, Figure 4-II) for RoBERTa-L — a model with 221,184 analyzed units — reveals a degree of sparsity and localization that distributed probing approaches cannot express. This is not an incremental finding; it challenges the prevailing view that Transformer representations are purely distributed and suggests a hybrid organization where some concepts are encoded locally, others distributively.

It connects NLP interpretability to the Network Dissection literature in vision. The paper explicitly builds on Bau et al. (2017, 2019), who showed that individual CNN filters and GAN units correspond to specific objects. But adapting this paradigm to NLP required solving three non-trivial problems that the vision approach never faced: (1) concepts can't be defined by single examples (solved by the sentence-sets representation); (2) neurons don't produce spatial maps (solved by max-pooling across the temporal dimension); (3) the evaluation metric must handle variable-length sequences (solved by Average Precision on max-pooled responses). The fact that this adaptation works — finding highly selective neurons across diverse Transformer architectures — suggests that neuron-level specialization is a general property of deep networks trained on structured data, not an artifact of convolutional architectures or spatial inputs.

The "large search space" framing. Rather than viewing the 680 million neuron-concept pairs in GPT2-L as a computational burden, the paper frames the enormous search space as enabling discovery: "We expect that, given the large search space, certain classifiers will perform well on specific concepts: the expert units" (Section 4.1). This reframes sparsity from a problem (pruning unimportant weights) to a discovery principle (with enough neurons and enough concepts, some pairings will inevitably be highly selective by chance, and these are the ones we find). This is a conceptual cousin of the lottery ticket hypothesis (Frankle and Carbin, 2019) but applied at inference time for interpretability rather than at training time for compression.

Evidence anchor: The finding that B layers acquire ~3.5× more concepts than A layers in GPT2-L (Figure 3, Section 4.3.1) — and that A_proj and B_proj layers acquire very few — would be invisible to methods that don't analyze individual neuron responses. This architectural asymmetry (the expanding feed-forward layers are the primary locus of concept specialization, not the attention projections) is a discovery about how Transformers organize knowledge, not just an evaluation of what they know.

Innovation 2: Concept Expertise Xγ as a Training-Free Proxy for Model Generalization

Before this paper, comparing pre-trained models required fine-tuning each one on a suite of downstream tasks (GLUE, SQuAD, etc.) and averaging the scores — a process that is computationally expensive, requires per-task hyperparameter tuning, and introduces benchmark-specific biases (Niven and Kao, 2019). The paper introduces concept expertise X_γ — the percentage of 1,641 diverse concepts for which a model contains at least one neuron with AP ≥ γ — and demonstrates that, at an empirically optimized threshold γ⋆, this metric correlates with downstream performance at r² = 0.833, higher than the average inter-task correlation among the tasks themselves (r² = 0.826; Table 2).

This is not just "another evaluation metric." It is a fundamentally different kind of evaluation — one based on inspecting the model's internal representations for the presence of specialized detectors, rather than measuring behavioral outputs after task-specific adaptation. Its significance lies in what it implies:

It provides evidence for a mechanistic hypothesis about generalization. The paper's core hypothesis is that "the presence of expert units is related to the knowledge acquired by TMs and to their performance" (Section 1). The strong correlation between X_γ⋆ and downstream task scores — and the fact that the optimal threshold γ⋆ requires near-perfect experts (0.997 for sense, 0.985 for homograph) — supports a specific claim: generalization emerges not from weak distributed knowledge spread across many neurons, but from having a diverse collection of extremely reliable, locally encoded concept detectors that can be composed for novel tasks. A model with experts for "financial institution," "transaction," "interest rate," and "loan" can combine these specialists when fine-tuned on a banking-related task; a model without them cannot. This is a testable mechanistic theory, not just a correlation.

It decouples evaluation from specific task choices. The standard practice of evaluating on GLUE + SQuAD bakes in assumptions about which tasks matter and weights them equally. The concept expertise metric is task-agnostic — it measures the diversity and quality of the model's internal concept vocabulary, independent of how that vocabulary gets used downstream. The fact that γ⋆ generalizes across task splits (RMSE = 0.0004 for sense concepts; Section 4.2) means the optimal threshold is not an artifact of the particular GLUE/SQuAD suite — it captures something fundamental about the relationship between concept specialization and generalization.

It reveals differential capability that aggregated benchmarks obscure. Table 1 shows that RoBERTa-Lm achieves the highest sense expertise (17.86%) but only middling homograph expertise (4.04%), while BERT-L achieves lower sense expertise (7.51%) but higher homograph expertise (5.72%). This differential pattern — models excel at different types of concept understanding — is invisible in aggregate GLUE scores, which collapse everything into a single number. The expertise breakdown by concept type provides a more diagnostic picture of model strengths and weaknesses.

The comparison to model size is revealing. Table 2 shows that model size correlates poorly with several tasks (r² < 0.3 for MRPC, STS-B(p), QNLI), while X_γ⋆ maintains r² > 0.75 for all but one (STS-B(p), where the task itself has r² = 0.49 with other tasks — suggesting it's an outlier, not a failure of X_γ⋆). This means concept expertise captures something beyond parameter count — it measures how effectively those parameters have been organized into specialists, which varies across models of similar size depending on training data, objective, and architecture.

Evidence anchor: The correlation results in Table 2 — specifically that X_γ⋆ outperforms the average inter-task correlation on 12 of 16 metrics — establish that expertise is not just correlated with generalization but is arguably a better proxy for overall capability than any single downstream task is for any other.

Innovation 3: Expert Forcing — Causal Control of Pre-Trained LMs Without Retraining

The paper's third major contribution is demonstrating that the discovered expert neurons are not merely correlational — they are causally effective for controlling concept presence in generated text. By forcing as few as 40–200 neurons (0.009–0.048% of GPT2-L's 414,720 units) to remain active at their median concept-positive value during nucleus sampling, the model's output shifts from generic text to text containing the target concept, with the concept presence increasing monotonically with K until saturation (Table 4).

This is distinct from all prior work on controllable generation because it requires no training, no fine-tuning, and no architectural modifications. The paper explicitly contrasts this with:

• Disentangled learning methods (Hu et al., 2017; Romanov et al., 2019; Chen et al., 2019) that require training dedicated architectures or objectives to separate concepts like sentiment, tense, syntax, and semantics.
• Conditional language models like CTRL (Keskar et al., 2019) that train massive models from scratch with control codes.
• Plug-and-play generative networks (Nguyen et al., 2017) that train external condition experts to guide pre-trained generators.

In all these approaches, the ability to condition on a concept is built in during training. The expert forcing approach, by contrast, discovers that the capability already exists in the pre-trained model and simply activates it. The paper claims this is the first demonstration of conditioning an off-the-shelf pre-trained LM without any training or additional parameters.

The intellectual significance goes beyond the practical benefit of avoiding retraining:

It provides causal evidence for the expert unit hypothesis. Correlation-based interpretability (including the AP ranking itself) can always be questioned: maybe neuron N fires on sentences about banking because it detects formal register, not the financial concept itself, and formal register happens to correlate with banking sentences in the dataset. The forcing experiment closes this loop: if artificially activating the neuron causes banking-related text to be generated (and deactivating it suppresses banking content), the neuron is part of the causal mechanism for concept expression, not just a correlated bystander. This is a stronger standard of evidence than what most interpretability work provides.

It reveals that conditional generation is a latent capability of pre-trained LMs. The fact that forcing 50 neurons — 0.012% of the model — is sufficient to induce football%1:04:00 in GPT2-L's output (Figure 1, where the generated text shifts to include "coach," "shooting," "throwing and catching") implies that the model already "knows how" to generate text about football; it just doesn't, because the default language model prior p(x) doesn't favor that concept given the context. The conditional expert p(y = football|x) is already wired into the model's weights; expert forcing simply amplifies its influence relative to the unconditional prior.

It establishes a dose-response relationship. The gradual increase in concept presence with K (Table 4: from no birds at K = 0, to sparrows at K = 60, to "bird bird bird" saturation at K = 200) maps cleanly onto the product-of-experts formulation: K controls the weight of p(y = c|x) relative to p(x). This quantitative relationship — not just "forcing works" but "more forcing produces more concept, up to a point" — is evidence that the mechanism operates through the probabilistic framework described in Equation 5, not through some degenerate or artifact-driven pathway.

The saturation behavior is itself informative. That increasing K eventually produces degenerate, repetitive text ("bird bird bird bird bird") is not a failure — it's a validation. If forcing K neurons produced fluent text regardless of K, we would suspect the mechanism doesn't actually work (maybe the model is ignoring the forced activations). The fact that the condition expert can overwhelm the language model prior confirms that the forced neurons genuinely influence the generation process, and that the two experts — fluency and concept presence — trade off against each other as the product-of-experts formulation predicts.

Evidence anchor: Table 4 and the extended results in Appendix D (Table 8) show successful conditioning across diverse concepts (elevator%1:06:00, smoke%1:19:00, gold%1:21:00, frustration%1:12:00) all with AP⋆_c ≥ γ⋆, while Table 9 shows that concepts with low AP⋆_c (e.g., work%1:06:00 with AP⋆_c = 0.8508) produce illegible output under forcing — confirming that expert quality predicts conditioning success.

Innovation 4: Concept Overlap as a Quantitative Measure of Co-Learning and Semantic Organization

The paper introduces a simple but powerful diagnostic: the Jaccard index Ω(q, v) between the top 1% of expert neurons for two concepts (Section 5, Equation 4). This metric transforms the expert unit discovery pipeline into a tool for analyzing how the model organizes semantic knowledge internally — which concepts it has "co-learnt" by assigning them overlapping sets of specialist neurons.

This is significant not because it measures something new (the idea that related concepts share neural representations is intuitive), but because it provides the first quantitative, neuron-level measure of semantic similarity in pre-trained Transformers that doesn't require training auxiliary models or relying on attention patterns. Prior work could probe whether a representation encodes, say, that "chair" and "table" are similar (via classifier accuracy on a relatedness task), but couldn't say which specific neurons encode that similarity and how much overlap exists. The overlap metric answers both questions precisely.

The results in Table 3 and Appendix C (Tables 6–7) demonstrate that the overlap is semantically meaningful at a fine-grained level:

• chair%1:06:00 (a seat for one person) shows highest overlap with table%1:06:01 (Ω = 0.458), bed%1:06:00 (0.361), cup%1:06:00 (0.341) — all furniture items. The overlap with the homograph concept chair%1:04:00 VS. chair%1:06:00 (the professor position vs. the furniture) is ranked 4th at 0.336 — the model has learned to distinguish the word senses, but the furniture sense shares more experts with other furniture than with the homograph's professor sense.
• chair%1:04:00 (the position of professor) shows a different overlap pattern: fellow%1:18:02 (0.371), director%1:18:03 (0.297), administration%1:04:00 (0.243), member%1:18:00 (0.241) — all professional/academic roles. The homograph concept distinguishing the two chair senses appears at Ω = 0.575, which is the highest overlap but makes sense: the neurons most tuned to the professor sense of "chair" are also those best able to distinguish it from the furniture sense.
• suspension%1:28:00 (a temporary cessation) overlaps with recovery%1:11:00 (0.398), season%1:28:02 (0.396), prospect%1:26:00 (0.387) — all temporal/interruption concepts.
• suspension%1:27:00 (a mixture of fine particles in fluid) overlaps with solution%1:27:00 (0.492), deposit%1:19:00 (0.438), material%1:27:00 (0.432), powder%1:27:00 (0.415) — all chemistry/materials concepts.

The same word (suspension, chair, phone) yields completely different overlap neighborhoods depending on which sense is queried. This is not word-level co-occurrence — it's genuine sense-level semantic organization captured by the pattern of shared expert neurons.

Why this is more than a visualization. The t-SNE projections in Appendix C (Figures 12–13) provide a spatial representation of the concept overlap structure, but the paper's contribution is the overlap metric itself — a single number per concept pair that can be ranked, thresholded, and used for downstream analysis. This enables quantitative claims like "the homograph concept distinguishing two senses of 'chair' is the 4th most overlapping concept with the furniture sense" — a statement that would be impossible with purely qualitative visualization.

Evidence anchor: Table 6 (Appendix C) shows that for five different query concepts (chair%1:06:00, chair%1:04:00, suspension%1:28:00, suspension%1:27:00, phone%1:06:00, phone%1:10:00), the top-5 overlapping concepts are all semantically coherent with the query's WordNet definition — and importantly, the overlap neighborhoods for different senses of the same word are completely different, confirming that the model's expert organization respects meaning, not surface form.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses a custom dataset called SentenceConcepts, constructed from the OneSec dataset (Scarlini et al., 2019). It contains 1,641 concepts total — 1,344 sense concepts and 297 homograph concepts — each represented by 100–1,000 positive sentences (containing the concept) and 100–1,000 negative sentences (not containing it, or containing the same keyword in a different sense for homographs). The complete concept list is provided in Appendix E (Tables 10–14). The total dataset size is approximately 2.5 million sentences.

• Base model(s). The paper analyzes 10 pre-trained Transformer models obtained from the HuggingFace Transformers repository (version 2.1.1): BERT-Base and BERT-Large (Devlin et al., 2018), RoBERTa-Base, RoBERTa-Large, and RoBERTa-Large-mnli (Liu et al., 2019), DistilBERT (Sanh et al., 2019), GPT2-Small, GPT2-Medium, and GPT2-Large (Radford et al., 2019), and XLM (Lample and Conneau, 2019). Model sizes range from 66M parameters (DistilBERT) to 774M (GPT2-L). The models span encoder-only (BERT, RoBERTa, DistilBERT), decoder-only (GPT2), and cross-lingual (XLM) architectures, chosen to test whether the expert unit phenomenon generalizes across architectural families. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4).

• Metrics. The primary interpretability metric is Average Precision (AP) — the area under the precision-recall curve — computed per (neuron, concept) pair by treating the neuron's max-pooled scalar responses across a concept's sentences as prediction scores and the binary concept-presence labels as ground truth (Section 4.1). The model-level evaluation metric is concept expertise X_γ — the percentage of concepts for which the model contains at least one neuron with AP ≥ γ (Equation 1). The combined expertise X_γ⋆ uses empirically optimized thresholds γ⋆ = 0.997 for sense concepts and γ⋆ = 0.985 for homograph concepts (Equation 3). For correlation with downstream performance, the paper uses squared Pearson correlation coefficient (r²) between X_γ⋆ and the reported performance on GLUE (10 datasets with multiple metrics, totaling 16 metrics) and SQuAD v1.1/2.0 (F1 scores). For concept co-learning, the metric is concept overlap Ω — the Jaccard index between the top 1% of expert neurons for two concepts (Equation 4). For conditional generation, evaluation is qualitative (human inspection of generated text for concept presence), with no automated metric reported.

• Baselines. The paper does not compare against alternative interpretability methods in a quantitative head-to-head framework, since the expert unit discovery method is a new paradigm. Instead, the baselines are internal comparisons: model size as a predictor of generalization (Table 2, column S), and average inter-task correlation as the standard for how well any single downstream task predicts others (Table 2, column T). For the conditional generation experiments, the baseline is unforced nucleus sampling (K = 0, the first row in Table 4), representing the language model's default generation behavior without expert intervention. The paper explicitly contrasts its approach with probing classifiers (Conneau et al., 2018; Adi et al., 2017; Tenney et al., 2019; Liu et al., 2019), saliency methods (Clark et al., 2019; Ghaeini et al., 2018), and disentangled learning methods (Hu et al., 2017; Romanov et al., 2019; Chen et al., 2019) as conceptual alternatives, but does not implement or numerically compare against them.

• Generation budget / compute accounting. The expert discovery pipeline involves a single forward pass over all sentences for all concepts. The paper provides concrete compute estimates (Section 7): for BERT-Base at 9ms per 128-token sentence, the full dataset requires 6.15 GPU-hours on a single GPU, reducible to 45 minutes with 8-way per-concept parallelization. The AP scores for GPT2-L require storing 414,720 × 1,641 ≈ 2.5 GB of floating-point values. For comparison, the paper notes that "a single evaluation of BERT-B on SQuAD v1.1 takes 24min. But several evaluations are required for hyper-parameter tuning and statistical significance. Summing up, evaluating on SQuAD v1.1/2.0 plus all GLUE tasks is more demanding than our proposed evaluation" (Section 7). For the conditional generation experiments, the budget is measured in number of forced expert neurons K, ranging from 40 to 200 (0.009–0.048% of GPT2-L's 414,720 units). The decoding strategy is nucleus sampling with p = 0.9, and the paper does not report generation times or token counts.

• Cross-validation / statistical protocol. For determining the optimal acquisition threshold γ⋆, the paper uses a cross-validation procedure: downstream tasks are "randomly split into reference and test sets, with a ratio 60/40%" (Section 4.2), and the RMSE between γ⋆ values computed on reference and test sets is measured over 10 random splits. The reported RMSE values are 0.0004 for sense concepts and 0.0028 for homograph concepts, indicating high stability. For the concept overlap analysis, the choice of top 1% threshold (99th percentile) is applied uniformly to all concepts, ensuring each concept contributes the same proportion of experts to the binary representation. For the conditional generation experiments, no statistical protocol is reported — the results are qualitative demonstrations with selected examples. The paper does not report confidence intervals for the correlation coefficients, expertise values, or overlap scores.

Main Quantitative Results

Concept Distribution Across Layers and Model Architectures

The paper first characterizes where expert units reside within Transformer models. Using a relaxed acquisition threshold of γ = 0.95 (chosen "for visualization purposes, γ⋆ being too restrictive"; Section 4.3.1), the per-layer distribution of acquired concepts reveals several architectural regularities (Figure 3 for GPT2-L and RoBERTa-L; Appendix A, Figures 5–11 for all other models):

Shallow layers dominate deep layers for sense concept acquisition. In GPT2-L (Figure 3-I), the number of acquired sense concepts decreases substantially from early to late layers, with the earliest Transformer blocks acquiring the most concepts. RoBERTa-L (Figure 3-III) shows a similar but less pronounced trend — the drop from shallow to deep layers is present but gentler. The paper notes this is "the first observation of this phenomenon in the NLP domain" and draws a parallel to GAN Dissection (Bau et al., 2019), where early layers in image GANs also accumulate more object concepts than later layers.

B layers acquire substantially more concepts than other layer types. Within each Transformer block (Figure 2), the B layers (the second feed-forward expansion) acquire approximately 3.5× more concepts than A layers in GPT2-L, and more than 10× more than A_proj and B_proj layers (the attention projections). This pattern holds qualitatively across RoBERTa-L as well, though with less extreme ratios. The paper interprets this as evidence that "the expanding layers (A and B) in the Transformer block are more prone to learn concepts" (Section 4.3.1), and specifically that B layers — which project from the intermediate dimension back down to the embedding dimension — are the primary locus of concept specialization.

Homograph concepts are sparsely distributed and few in early layers. Figures 3-II and 3-IV show that homograph concepts (disambiguating different senses of the same word) are spread more thinly across layers, with very few detected in the first several Transformer blocks. Unlike sense concepts, which cluster in early layers, homograph concepts appear more uniformly (albeit at low absolute numbers) across the depth of the model. This suggests that contextual disambiguation — which requires integrating broader context to distinguish word senses — is not handled by the earliest layers, consistent with the intuition that early layers detect local patterns while later layers perform more complex contextual reasoning.

Cross-model consistency. Appendix A (Figures 5–11) shows that these patterns are broadly consistent across BERT-Base, BERT-Large, DistilBERT, GPT2-Small, GPT2-Medium, RoBERTa-Large-mnli, and XLM. The qualitative finding that B layers are the primary locus of concept expertise and that generative models (GPT2) show a stronger early-to-late decay than encoder models (BERT, RoBERTa) appears to be a general property of Transformer architectures, not specific to any single model.

Key takeaway from the distribution analysis: The architecture of where concepts are stored is non-uniform and non-random. Concepts concentrate in specific layer types (B layers over A layers, and both over attention projections) and in specific depth regimes (shallow for sense concepts, more distributed for homographs). This provides the first systematic characterization of where pre-trained Transformers store conceptual knowledge at the neuron level.

Concept Expertise Across Models

Table 1 reports the concept expertise for all 10 models at the optimally derived thresholds (γ⋆ = 0.997 for sense, 0.985 for homograph), along with the combined expertise X_γ⋆. The results reveal substantial variation across models:

Sense expertise spans a 17× range across models. RoBERTa-Large-mnli achieves the highest sense expertise at 17.86% (240 concepts), followed by GPT2-L at 15.03% (202 concepts) and RoBERTa-L at 14.66% (197 concepts). At the low end, BERT-Base achieves only 1.04% (14 concepts) and GPT2-Small achieves 1.79% (24 concepts). Notably, RoBERTa-L (355M parameters) achieves 14.66%, while BERT-L (330M parameters, comparable size) achieves only 7.51% — a nearly 2× difference despite similar parameter counts. This is attributed to "modifying the training procedure and the data" (Section 4.3.2), specifically RoBERTa's dynamic masking, larger training data, and longer training.

Homograph expertise is uniformly low, peaking at 5.72%. The highest homograph expertise is achieved by BERT-Base, BERT-Large, and DistilBERT — all at 5.72% (17 concepts). RoBERTa-L achieves 5.05% (15 concepts), and GPT2-L achieves only 3.37% (10 concepts). This confirms the paper's expectation that homograph concepts are fundamentally harder: no model acquires more than ~6% of the 297 homograph concepts, even at the relaxed threshold of γ = 0.985. The paper identifies this as an open challenge: "Building pre-trained models inherently able to disambiguate homograph concepts at unit level remains a challenge, and we speculate that such knowledge will help the models generalize even better" (Section 4.3.2).

Combined expertise X_γ⋆ is dominated by sense performance. Because there are 1,344 sense concepts vs. 297 homograph concepts (a 4.5:1 ratio), the combined metric (Equation 3) heavily weights sense expertise. RoBERTa-Large-mnli leads at 15.36%, followed by RoBERTa-L and GPT2-L at 12.92%, then XLM at 8.59%, and BERT-Base at 1.89%. The combined metric provides a single-number summary but obscures the interesting pattern that some models (BERT-L, DistilBERT) achieve relatively high homograph expertise despite mediocre sense expertise.

Expert units are extremely sparse. Figure 4-II shows the histogram of the number of expert units per concept (at γ = 0.95) for RoBERTa-L. "Most of the concepts have less than 50 dedicated expert units (0.022%), with a median of 7 experts (0.0032%) per concept" — out of 221,184 analyzed units. This extreme sparsity (the median concept is detected by only 7 out of 221,184 neurons) supports the paper's framing of experts as highly specialized, locally encoded detectors rather than distributed representations. Even for the best-detected concepts, the number of expert neurons is a tiny fraction of the total.

Expertise as a Proxy for Model Generalization

Table 2 presents the paper's central evaluation claim: the squared Pearson correlation (r²) between concept expertise X_γ⋆ and downstream task performance, compared against model size and inter-task correlations.

The headline number: r² = 0.833 average correlation with downstream tasks. Across the 16 GLUE and SQuAD metrics, X_γ⋆ achieves an average r² of 0.833, which is higher than the average inter-task correlation of 0.826. This means that concept expertise predicts performance on a held-out downstream task about as well as performance on one GLUE task predicts performance on another — and it does so without any fine-tuning.

Per-task breakdown reveals consistent strength. X_γ⋆ achieves r² > 0.75 on 12 of 16 metrics. The strongest correlations are with AX (r² = 0.956), QQP accuracy (0.944), MNLI-mm (0.944), SQuAD 2.0 F1 (0.937), and QNLI (0.923). The weakest correlations are with STS-B Pearson (0.401) and WNLI (0.619). Critically, for tasks where model size correlates poorly (r² < 0.3 for MRPC accuracy, MRPC F1, STS-B Pearson, QNLI), X_γ⋆ maintains r² > 0.75 for all except STS-B Pearson — and STS-B Pearson itself shows r² = 0.49 with other tasks, suggesting it is an outlier in the task suite rather than a failure specific to expertise.

Comparison to model size. The average correlation with model size is only 0.408, compared to 0.833 for X_γ⋆. Model size correlates well with some tasks (SQuAD 1.1 F1: r² = 0.899; SQuAD 2.0 F1: 0.961) but poorly with many GLUE tasks (MRPC F1: 0.162; STS-B Pearson: 0.000; QNLI: 0.286). This demonstrates that parameter count alone is a poor proxy for generalization, while concept expertise — which measures how effectively those parameters have been organized into specialists — captures meaningful variation in model quality.

Comparison to inter-task correlation. The average inter-task correlation of 0.826 represents the baseline: how well does knowing a model's performance on one GLUE/SQuAD task predict its performance on another? X_γ⋆ exceeds this average (0.833 vs. 0.826), and outperforms inter-task correlation on 12 of 16 individual metrics. This means that concept expertise is not just correlated with downstream performance — it is arguably a better proxy for overall capability than any single fine-tuned evaluation is for any other.

The models used in the correlation. Table 2 notes that "only those models reporting results on each dataset are used" — meaning different subsets of models appear in different rows. BERT-Base, BERT-Large, DistilBERT, and RoBERTa-L are used for all tasks; XLM is added where it reports results. No GPT2 models appear in this table because GPT2 is a generative model without reported GLUE/SQuAD scores (it would require fine-tuning with a task-specific head, which the paper does not perform). This limits the correlation analysis to 4–5 encoder-only models — a small sample that makes the r² values potentially sensitive to individual model performance.

The AP⋆_c histogram context. Figure 4-I shows the histogram of AP⋆_c (the AP of the best expert for each concept) for RoBERTa-L. Most sense concepts have AP⋆_c > 0.90, with a strong mode near 1.0, while homograph concepts show a broad distribution from ~0.5 to ~1.0. The vertical lines at γ⋆ = 0.997 (sense) and 0.985 (homograph) show that the optimal thresholds fall in the extreme upper tail — only concepts with near-perfect best experts count toward expertise. This explains why the expertise percentages in Table 1 are relatively low (max 17.86%) despite Figure 4-I showing that most sense concepts have some expert with AP > 0.90: the threshold is so stringent that only ~15–18% of concepts clear it.

Concept Overlap Quantifies Semantic Co-Learning

Table 3 and Appendix C (Tables 6–7) present the core concept overlap results. The analysis demonstrates that the Jaccard index between top-1% expert sets produces semantically coherent neighborhoods.

Different senses of the same word yield different overlap neighborhoods. For chair%1:06:00 (furniture), the top-5 overlapping concepts are table%1:06:01 (Ω = 0.458), bed%1:06:00 (0.361), cup%1:06:00 (0.341), a homograph disambiguating the two table senses (0.336), and floor%1:06:00 (0.328) — all household objects or furniture items. For chair%1:04:00 (professor position), the top-5 overlaps are the homograph distinguishing the two chair senses (0.575), fellow%1:18:02 (0.371), director%1:18:03 (0.297), administration%1:04:00 (0.243), and member%1:18:00 (0.241) — all professional or organizational roles. The fact that the same word produces completely different overlap neighborhoods depending on sense is strong evidence that the expert organization respects meaning, not surface form.

Chemically-related concepts cluster together. For suspension%1:27:00 (a mixture of fine particles), the top overlaps are solution%1:27:00 (0.492), deposit%1:19:00 (0.438), material%1:27:00 (0.432), powder%1:27:00 (0.415), and crystal%1:27:00 (0.413) — all substances and materials. For suspension%1:28:00 (temporary cessation), the overlaps are recovery%1:11:00 (0.398), season%1:28:02 (0.396), prospect%1:26:00 (0.387), attempt%1:04:00 (0.380) — all temporal or process-oriented concepts. Again, the same word with different senses yields neighborhoods that are semantically coherent and non-overlapping with each other.

Market concepts show mutual reinforcement. Table 7 (Appendix C) shows an interesting case: market%1:04:00 (commercial activity) and market%1:14:00 (customers for a product) have high overlap with each other and with economy%1:14:00, capital%1:21:01, labor%1:14:00, and wealth%1:26:00. The homograph concept market%1:14:00 VS. market%1:04:00 achieves low AP⋆_c = 0.523, indicating that while the two senses share many experts (they're semantically close), no single neuron perfectly distinguishes them — the model has not developed a specialized homograph disambiguator for these two senses.

t-SNE visualization provides spatial confirmation. Figures 12–13 (Appendix C) show t-SNE projections of the binary concept representations s_c, with zoomed-in views on chair, right, and their neighborhoods. The visualizations confirm that concepts with similar meanings (as judged by human interpretation of WordNet definitions) cluster together in the expert overlap space, and that homograph concepts (shown in orange) often appear near the sense concepts they disambiguate — reflecting that the most relevant experts for telling two senses apart are the ones sensitive to the distinctions between them.

Key takeaway: The overlap metric Ω provides a quantitative, reproducible measure of semantic similarity as represented in the model's internal organization. Unlike word embeddings or attention patterns, this similarity is grounded in which specific neurons the model has assigned to each concept, making it a direct window into the model's conceptual taxonomy.

Conditional Generation via Expert Forcing

Table 4 and the extended results in Appendix D (Tables 8–9) demonstrate that forcing top expert neurons during nucleus sampling induces the target concept in generated text.

The dose-response relationship. For concept bird%1:05:00 (warm-blooded egg-laying vertebrates) using GPT2-L with context "Once upon a time":

• K = 0 (no forcing, 0% of units): Generic text about teaching high school English — no bird-related content.
• K = 40 (0.009%): Text shifts to include "horse" and "treasures" — animal-adjacent but not bird-specific.
• K = 60 (0.015%): "house sparrows" appears — a specific bird, directly on-target.
• K = 80 (0.019%): "chicken-like air wings" — bird-related concepts strengthening.
• K = 200 (0.048%): Degeneration into "bird bird bird bird bird bird" — saturation, p(y = c|x) dominates p(x).

This monotonic increase in concept presence with K, culminating in saturation, empirically validates the product-of-experts formulation (Equation 5). The gradual shift from generic → adjacent → on-target → degenerate demonstrates that the forcing mechanism provides continuous control over concept strength, not just a binary on/off switch.

Concepts with AP⋆_c ≥ γ⋆ condition successfully. Table 8 (Appendix D) shows four additional concepts (elevator%1:06:00, smoke%1:19:00, gold%1:21:00, frustration%1:12:00) with AP⋆_c ranging from 0.9981 to 0.9999, all at K = 60 forced experts. In each case, the generated text from the context "In a shocking finding, scientist discovered a herd of unicorns..." incorporates the target concept naturally: "levitation and deceleration, which blew them up a few hundred feet" for elevator; "three fresh, fresh inhalations to extract all of the smoke" for smoke; "magical 'Slab Silver', which is one of the most beautiful forms of gold" for gold; "we did not have success" and "wearing brightly red patches of clothing" for frustration. While the generation quality varies (some outputs are more coherent than others), the concept presence is unambiguous.

Concepts with AP⋆_c ≪ γ⋆ fail to condition. Table 9 (Appendix D) demonstrates the failure case with work%1:06:00 (AP⋆_c = 0.8508, far below γ⋆ = 0.997). At K = 40, the generated text is "a was on. Magic, symbolism, decadence and tragedy" — largely unrelated to the work concept. At K = 70, the output becomes "Now I met my tools„„, work, work" — the word "work" appears but the text is fragmentary. At K = 100 and K = 200, the output degenerates into repetitive "WORK WORK WORK" and "work work work" strings, entirely illegible. The model fails to generate fluent text containing the concept because it lacks genuine expert units for work%1:06:00 — forcing random or weakly-correlated neurons produces noise, not concept induction.

The figure 1 example. The paper's opening example (Figure 1) shows GPT2-L conditioned on football%1:04:001 by forcing its top 50 expert units (0.012% of 414,720 units), with the beginning of the paper's abstract as context. The generated text reads: "In this work we study the presence of expert units in pre-trained Transformer Models (TMs), and how they impact a model's performance. We define expert units to be neurons that are able to classify a specific concept with a given average precision, hence being able to correctly classify a future ball's passing or shooting angle. We find that many multi-category TMs are specifically trained to indicate the accuracy of a given concept when following the instructions of a pre-specified skill coach and/or observers. In this experiment we test this hypothesis, using TMs that specialize in the shooting skills of throwing and catching." The text blends the academic paper context ("Transformer Models," "classify a specific concept") with football terminology ("ball's passing or shooting angle," "skill coach," "shooting skills of throwing and catching") — demonstrating that expert forcing can induce the concept while respecting the provided context.

Key takeaway from generation experiments: The forcing results provide three pieces of evidence for the expert unit hypothesis: (1) concepts with high-AP experts can be reliably induced with very few forced neurons (40–200, or 0.01–0.05% of units); (2) concepts without high-AP experts cannot be induced; (3) the dose-response relationship follows the product-of-experts formulation, with K controlling the weight of the condition expert relative to the language model prior. This establishes that the discovered experts are causal for concept expression, not merely correlational.

Ablation Studies and Robustness Checks

The paper's ablation studies are limited in scope — there is no systematic hyperparameter sweep for the expert forcing mechanism, no comparison of alternative max-pooling strategies, and no sensitivity analysis for the top-1% threshold in the overlap metric. What ablations exist are primarily architectural (comparing across models and layers) and threshold-related (the γ⋆ optimization procedure). I report what is present and note what is absent.

Architectural comparison across 10 models: The concept distribution analysis (Figure 3, Appendix A Figures 5–11) and expertise comparison (Table 1) collectively serve as a model-family ablation, showing that the expert unit phenomenon is not unique to any single architecture or training procedure. BERT, RoBERTa, GPT2, DistilBERT, and XLM all exhibit expert units, with broadly consistent layer-type preferences (B layers > A layers ≫ attention projections). However, the quantitative expertise varies substantially — RoBERTa-L (355M) achieves 2× the sense expertise of BERT-L (330M) despite similar size — confirming that training procedure and data matter significantly. The paper attributes RoBERTa's advantage to "modifying the training procedure and the data" but does not ablate which specific changes (dynamic masking, larger corpus, longer training, removal of next-sentence prediction) drive the improvement.

γ⋆ cross-validation across task splits: The 60/40% split procedure with 10 random seeds produces RMSE values of 0.0004 (sense) and 0.0028 (homograph), demonstrating that the optimal threshold is stable across different task subsets. This is the paper's primary robustness check for its central evaluation metric. However, the cross-validation is performed within the set of GLUE+SQuAD tasks — it does not test whether γ⋆ would generalize to entirely different task families (e.g., summarization, translation, dialogue). The stability across GLUE/SQuAD splits tells us the threshold isn't overfit to specific tasks within that suite, but doesn't guarantee it would be optimal for evaluating models on fundamentally different capabilities.

Separate vs. pooled analysis of sense and homograph concepts: Table 1 reports expertise separately for sense and homograph, with different γ⋆ thresholds. This is an implicit ablation: treating all 1,641 concepts as one pool would obscure the qualitatively different behavior (sense expertise ranges from 1–18%; homograph expertise is capped at ~6%) and would produce a compromised γ⋆ that fits neither type well. The paper's choice to separate them is justified by the RMSE results, but the paper does not report what the pooled alternative would look like (what γ⋆ would emerge, what correlation it would achieve).

Layer-type comparison within Transformer blocks: Figure 3 and Appendix A show per-layer-type concept acquisition counts, demonstrating that B layers acquire ~3.5× more concepts than A layers in GPT2-L, and that A_proj and B_proj layers acquire very few. This is a natural ablation within the architecture: by analyzing all four linear layer types separately, the paper discovers that concept specialization is not uniformly distributed — it concentrates in specific computational stages. The finding that attention projections (A_proj, B_proj) contribute minimally to concept expertise suggests that concept detection is primarily a feed-forward computation, not an attention-based one. The paper does not explore why this asymmetry exists (e.g., are the expanding dimensions of A and B layers better suited for storing sparse concept detectors?).

AP⋆_c histogram vs. γ⋆ threshold: Figure 4-I shows the distribution of best-expert AP across concepts, with γ⋆ values marked as vertical lines. The fact that γ⋆ = 0.997 falls in the extreme upper tail of the sense AP⋆_c distribution (where most concepts cluster near 1.0) validates that the optimal threshold is stringent but not impossibly so — many concepts have AP⋆_c in the 0.99–1.0 range, so γ⋆ = 0.997 does identify a meaningful subset. For homographs, γ⋆ = 0.985 is lower but still in the upper region of a much broader distribution, reflecting the greater difficulty.

Missing ablation: Max-pooling vs. alternative aggregation strategies. The paper uses max-pooling across the temporal dimension to convert per-token activations to per-sentence scalars. Alternative strategies — mean pooling, sum pooling, taking the [CLS] token activation, or using the response at the keyword's token position — are not compared. It is possible that max-pooling inflates AP scores by capturing spurious high activations on unrelated tokens, and that a different aggregation method would produce different expert rankings. The paper does not investigate this.

Missing ablation: Sensitivity to the number of sentences per concept. The dataset uses 100–1,000 sentences per concept, but the paper does not ablate whether AP rankings stabilize at smaller sample sizes. This is practically important: if reliable expert discovery requires only 50 sentences per concept, the dataset could be expanded much more cheaply. If it requires 500+, the cost of concept dataset construction is a significant barrier to extending the method to new domains.

Missing ablation: Top-1% threshold sensitivity for concept overlap. The overlap metric Ω uses the 99th percentile per concept to define binary concept representations. The paper does not explore sensitivity to this choice — would top-5% or top-0.1% produce different overlap neighborhoods? The threshold captures the intuition that only the very best experts matter for co-learning, but the robustness of this choice is untested.

Missing ablation: Forcing value sensitivity. The conditional generation uses the median active value from positive sentences as the forcing target. Alternatives — mean active value, maximum active value, fixed percentile (e.g., 75th or 90th percentile), or a learned optimal forcing value — are not explored. The paper does not report whether the generation quality is sensitive to this choice, or whether some concepts require higher or lower forcing values for optimal induction.

Negative result with ReST^EM: While not presented as an ablation, the paper's discussion of revision model training failure (noted in the reference example but actually from a different paper — I should not include this here as it's not in this paper). There are no negative ablation results in this paper — the expert forcing failure for low-AP⋆ concepts (Table 9) is the closest to a negative result, but it's presented as confirmation of the method rather than as an ablation.

Critical Assessment

The experiments provide solid support for the paper's central claims, but with important boundary conditions and methodological limitations that constrain the strength of those claims.

Claim: Expert units exist in pre-trained Transformers and can be identified by treating neurons as binary classifiers.

This is the paper's most foundational claim, and the evidence is strong. The distribution of AP⋆_c scores (Figure 4-I) shows that for the majority of the 1,344 sense concepts, there exists at least one neuron with AP > 0.90, and for many, AP approaches 1.0. The per-layer concept distributions (Figure 3, Appendix A) are consistent across 10 models spanning three architectural families, demonstrating that the phenomenon is not model-specific. The sparsity of experts (median of 7 per concept in RoBERTa-L, out of 221,184 units; Figure 4-II) supports the "expert" framing — these are genuinely selective units, not artifacts of the evaluation method.

However, the evidence is bounded by the concept dataset. The 1,641 concepts are all drawn from WordNet and are predominantly concrete nouns, verbs, and adjectives with clear lexical anchors. The method identifies neurons that detect concepts closely tied to specific keywords (for sense concepts) or to contextual cues distinguishing word senses (for homograph concepts). It does not establish whether expert units exist for more abstract or compositional concepts (e.g., "sarcasm," "narrative tension," "mathematical proof structure") that cannot be cleanly represented by sets of positive and negative sentences with binary labels. The paper acknowledges this implicitly: "It is possible, but not yet explored, that more complex concepts such as homograph, require a more complex expert such as a set of units" (Section 7). The dataset-constrained nature of the findings means we should be cautious about extrapolating to "Transformers learn specialized concept detectors for all types of knowledge."

Furthermore, the paper conflates the discovery of any expert with all relevant experts. The AP metric identifies the single best neuron per concept, but there may be hundreds of additional neurons with AP just below the best — distributed expertise that the binary "acquired vs. not" threshold obscures. The focus on individual neurons (rather than linear combinations or circuits) is a deliberate design choice, but it means the method is blind to concepts encoded in distributed form. The paper acknowledges this limitation: "It is possible, but not yet explored, that more complex concepts such as homograph, require a more complex expert such as a set of units."

Claim: Concept expertise X_γ⋆ correlates with downstream task performance (r² = 0.833) and can rank models without fine-tuning.

The correlation result in Table 2 is statistically impressive and supports the claim that expertise predicts generalization. However, several aspects of the experimental design limit the strength of the conclusion:

The correlation is based on only 4–5 models for most tasks. Table 2's first column lists the models used: BERT-Base, BERT-Large, DistilBERT, and RoBERTa-L (4 models) for most tasks, with XLM added where it reports results. An r² computed on 4 data points is extremely sensitive to individual model performance — a single outlier can drive the correlation up or down substantially. The paper does not report confidence intervals or p-values for the r² values, so we cannot assess whether these correlations are statistically significant given the tiny sample size. This is the single most important limitation of the evaluation results: the finding that X_γ⋆ is a good proxy for generalization rests on 4–5 encoder models, all trained on similar English corpora with similar objectives (masked language modeling).

No GPT2 models are included in the correlation analysis. GPT2-Small, GPT2-Medium, and GPT2-Large are analyzed for expert units (Table 1) but excluded from the generalization correlation because they don't have reported GLUE/SQuAD scores. This is understandable — GPT2 is a generative model that requires task-specific fine-tuning heads to be evaluated on GLUE, and the paper's goal is to avoid fine-tuning — but it means we don't know whether the X_γ⋆–generalization relationship holds for autoregressive language models. GPT2-L achieves 12.92% combined expertise (Table 1), comparable to RoBERTa-L, but we can't verify whether its downstream performance (if fine-tuned) would match that prediction.

The optimal threshold γ⋆ is fit to the same tasks used for evaluation. The procedure in Section 4.2 optimizes γ⋆ to maximize the average r² across all GLUE and SQuAD tasks. The combined expertise X_γ⋆ is then correlated with those same tasks in Table 2. While the cross-validation across task splits (60/40%) shows that γ⋆ is stable, this only demonstrates that the threshold generalizes across subsets of the same task suite — it does not demonstrate generalization to entirely held-out tasks. A stronger evaluation would hold out one or more tasks entirely during γ⋆ optimization, then test whether X_γ⋆ (computed using the held-out-determined γ⋆) predicts performance on the held-out tasks. The paper does not perform this test.

The comparison to model size is somewhat unfair. Model size correlates at r² = 0.408 on average, but the model set includes DistilBERT (66M) and XLM (667M), creating a wide size range that should help size correlate if it were a reliable predictor. The fact that size correlates poorly (r² < 0.3 on several tasks) while X_γ⋆ maintains r² > 0.75 on those same tasks is genuinely informative — expertise captures something beyond scale. However, the paper doesn't compare X_γ⋆ to other training-free metrics that might also predict generalization (e.g., vocabulary size, training data quantity, perplexity on a held-out corpus). Without these baselines, we can't assess whether expertise is uniquely good or just better than the weakest plausible baseline (parameter count).

Claim: Concept overlap Ω quantifies semantic co-learning in a semantically meaningful way.

The results in Table 3 and Appendix C are qualitatively convincing — the neighborhoods of overlapping concepts are semantically coherent and sense-specific. However, the evaluation is purely qualitative (human inspection of top-5 lists and t-SNE plots). There is no quantitative benchmark for "does the overlap structure match human semantic similarity judgments?" — for example, correlating Ω with WordNet path similarity, human similarity ratings, or performance on a word sense disambiguation task. The paper presents overlap as a tool for explainability ("given a concept with unknown definition, the overlapping concepts can help explain it"; Section 5) but does not validate this use case (e.g., by hiding concept definitions and asking human judges whether the overlap neighborhood recovers the definition).

Additionally, the top-1% threshold is arbitrary. The paper chooses the 99th percentile to ensure each concept contributes the same number of experts to the binary representation, which is a reasonable normalization. But without a sensitivity analysis (what if top-5%? top-0.1%? a fixed AP threshold of, say, 0.95?), we don't know whether the semantically coherent neighborhoods are robust to this choice or an artifact of it.

Claim: Expert forcing induces target concepts in generated text, demonstrating causal control.

The qualitative results in Table 4, Figure 1, and Appendix D are striking and effectively communicate the core idea. However, the evaluation of this claim has significant limitations:

No automated evaluation of concept presence. The paper relies entirely on human inspection of a few hand-picked examples. There is no quantitative measurement of conditioning success — no metric for "how strongly is concept c present in the generated text?" (e.g., using a concept classifier, keyword counting, or human rating studies). Without automated evaluation, we cannot assess: (a) what fraction of generated samples actually contain the target concept (the examples shown are presumably cherry-picked successes); (b) how the success rate varies across concepts and K values; (c) whether the generated text is diverse or collapses to a few templates.

The number of tested concepts is small and selected. The paper shows forcing results for ~10 concepts total across all tables and figures, out of 1,641 available concepts. The selection is not described as systematic — the shown concepts all have very high AP⋆_c (except the failure case of work%1:06:00). We don't know whether forcing works for concepts with AP⋆_c in the 0.95–0.99 range (below γ⋆ but still good), or whether the success rate drops smoothly or discontinuously with AP⋆_c. The claim that "concepts with AP⋆_c ≥ γ⋆ condition successfully" is supported by the examples shown but not by any statistical evidence.

No comparison to alternative conditioning methods. The paper claims this is the first method to condition pre-trained LMs without retraining, which is a genuine novelty. But without comparing to any alternative — even a simple baseline like prepending the concept word to the prompt — we can't assess whether expert forcing is better than simpler approaches or just different. For example, if the prompt "Write about birds: Once upon a time" already induces bird-related text with GPT2-L, then expert forcing's contribution is incremental, not transformative. The paper does not run this baseline.

The degeneracy at high K is informative but not quantitatively characterized. The saturation behavior ("bird bird bird...") is a validation of the product-of-experts framing, but the paper doesn't measure where saturation occurs as a function of AP⋆_c, concept type, or model size. A systematic characterization — "saturation begins at K ≈ N_experts × 2" or "saturation K correlates with AP⋆_c" — would strengthen the mechanistic interpretation.

What experiments would have strengthened the paper:

1. A held-out task evaluation for γ⋆: Optimize γ⋆ on a subset of GLUE tasks, then test X_γ⋆ against the remaining held-out tasks (not just cross-validated within the full set). This would demonstrate genuine predictive power rather than within-suite correlation.

2. Statistical significance testing for the r² values: With only 4–5 models, the r² = 0.833 could be driven by one or two data points. Reporting confidence intervals or p-values against a null distribution (e.g., permuting model labels) would clarify how robust the correlation is to the small sample size.

3. Inclusion of GPT2 models in a generalization evaluation: Fine-tune GPT2-S/M/L on GLUE/SQuAD (or a subset) and test whether their X_γ⋆ predicts the resulting scores. This would extend the claimed generality of expertise as a generalization proxy beyond encoder-only models.

4. A baseline for conditional generation: Compare expert forcing against (a) prepending the concept word to the prompt, (b) using the concept word as the first generated token, (c) naive activation maximization (forcing all neurons with positive weights for the concept word embedding). If expert forcing substantially outperforms these simple baselines, the causal claim is strengthened; if not, the method may be rediscovering something the model already does through standard prompting.

5. Automated evaluation of conditioning success: Use a held-out concept classifier (or a separate pre-trained model) to measure the probability that generated text contains the target concept, and report success@K curves across a representative sample of concepts. This would replace cherry-picked examples with systematic evidence.

6. Sensitivity analysis for key hyperparameters: The max-pooling strategy, the top-1% threshold for overlap, the median forcing value, and the minimum sentence count per concept are all fixed choices. Ablating these would clarify which design decisions are load-bearing and which are incidental.

Overall assessment: The paper's experiments successfully demonstrate the existence of expert units, their correlation with generalization (in encoder models, on GLUE/SQuAD), and their causal role in concept expression (for selected high-AP concepts). These are non-trivial findings that advance the field's understanding of how pre-trained Transformers organize knowledge internally. However, the quantitative strength of the claims is constrained by small sample sizes (4–5 models for the correlation analysis), lack of held-out task evaluation, and purely qualitative assessment of the generation results. The paper establishes a promising paradigm — neuron-as-classifier interpretability — and provides compelling initial evidence, but the generalizability and practical utility of the approach remain to be demonstrated at scale.

6. Limitations and Trade-offs

Capability Ceiling: Hard Problems Derive No Benefit from Expert Forcing

The constraint. The expert forcing approach to conditional generation only works when the model contains high-quality expert neurons for the target concept. The paper explicitly demonstrates this with work%1:06:00 (AP⋆_c = 0.8508, far below the sense threshold γ⋆ = 0.997), where forcing top experts produces illegible, repetitive output rather than concept-containing text (Table 9, Appendix D). As K increases from 40 to 200, the output degrades from fragmented phrases to "WORK WORK WORK WORK WORK" — the model has no genuine expert for this concept, so forcing weakly-correlated neurons injects noise rather than signal.

The consequence. This creates a hard capability boundary: expert forcing can only induce concepts that the model has already learned to represent at the individual neuron level with near-perfect precision. For any concept where AP⋆_c < γ⋆ — which includes the majority of concepts even for the best models (RoBERTa-Lm acquires only 17.86% of sense concepts at γ⋆ = 0.997; Table 1) — the method either fails outright or produces degenerate output. A practitioner cannot use expert forcing to induce arbitrary new concepts; they can only amplify concepts the model already "knows." This is fundamentally different from training-based conditioning methods (CTRL, Keskar et al., 2019; or fine-tuning with control codes) which can teach the model new concept-generation associations. Expert forcing discovers and exploits existing knowledge but cannot create it.

Evidence in the paper. Table 4 demonstrates the success case for bird%1:05:00 (AP⋆_c = 0.9995) with a clean dose-response curve. Table 9 demonstrates the failure case for work%1:06:00 (AP⋆_c = 0.8508) with output degeneration. Figure 4-I shows the full distribution of AP⋆_c values: many concepts, especially homographs, fall well below the γ⋆ thresholds. Table 1 shows that even the best model (RoBERTa-Lm) acquires only 240 of 1,344 sense concepts (17.86%) and 12 of 297 homograph concepts (4.04%) at the stringent γ⋆ thresholds.

Mitigation status. The paper is candid about this limitation, noting that "building pre-trained models inherently able to disambiguate homograph concepts at unit level remains a challenge, and we speculate that such knowledge will help the models generalize even better" (Section 4.3.2). The limitation is inherent to the "discovery, not training" paradigm — it cannot be mitigated without either improving the base model's pre-training (to create better experts) or relaxing the definition of expertise (to use weaker or more distributed representations, which the paper explicitly does not explore; Section 7: "It is possible, but not yet explored, that more complex concepts such as homograph, require a more complex expert such as a set of units").

Difficulty Estimation Cost Is Unaccounted For in the Headline Efficiency Numbers

The constraint. Discovering expert units requires running a forward pass over the entire SentenceConcepts dataset — approximately 2.5 million sentences across 1,641 concepts — for every model being analyzed. The paper estimates this at 6.15 GPU-hours for BERT-Base on a single GPU (Section 7). For larger models like GPT2-L (414,720 units analyzed vs. BERT-Base's fewer units), the cost would be proportionally higher. Furthermore, the expert forcing procedure requires computing the median active value for the top-K experts from the positive sentence set for each target concept — this is a one-time cost per concept but must be done before generation can begin.

The consequence. The paper positions concept expertise X_γ⋆ as a cheaper alternative to fine-tuning evaluation ("evaluating on SQuAD v1.1/2.0 plus all GLUE tasks is more demanding than our proposed evaluation"; Section 7). But this comparison is misleading in an important way: the expert discovery cost is a fixed upfront cost per model, while fine-tuning evaluation costs scale with the number of downstream tasks. For evaluating a single new model, the expert discovery approach requires 6+ GPU-hours on the full concept dataset. A practitioner evaluating one new model checkpoint would find it faster to fine-tune on one or two representative GLUE tasks (SST-2 takes minutes; MNLI takes a few hours) than to run the full expert discovery pipeline. The claimed efficiency advantage only materializes when evaluating many models (amortizing the concept dataset forward passes) or when the alternative is the full GLUE+SQuAD suite with hyperparameter tuning.

For the conditional generation use case, the upfront cost of computing AP scores for all 414,720 × 1,641 neuron-concept pairs (~680 million evaluations for GPT2-L) must be paid before a single sentence can be conditioned on any concept. If a practitioner wants to condition on a concept not in the SentenceConcepts dataset, they must first construct a new dataset of positive and negative sentences for that concept, run forward passes, compute AP scores for all neurons, and identify the top-K experts — a process that likely takes hours. This makes expert forcing impractical for ad hoc or interactive concept conditioning, where training-free methods like prompt engineering produce results in seconds.

Evidence in the paper. Section 7 provides the compute estimates: 6.15 GPU-hours for BERT-Base, reducible to 45 minutes with 8 GPUs. The AP score storage is ~2.5 GB for GPT2-L. The paper acknowledges that "finding experts in TMs is an exhaustive task that implies some memory and compute requirements" (Section 7). However, it does not include this cost in any of the headline comparisons (e.g., the r² = 0.833 correlation is reported without noting that producing X_γ⋆ required 6+ GPU-hours of computation per model).

Mitigation status. The paper acknowledges the compute requirement but does not attempt to reduce it. The discussion frames the cost as acceptable relative to full GLUE+SQuAD evaluation (Section 7), but this framing conflates the one-time concept dataset forward pass (amortizable across models) with the per-model AP computation (not amortizable). The paper suggests parallelization (8 GPUs → 45 minutes) as a practical mitigation but does not explore whether fewer sentences per concept, fewer concepts, or a learned difficulty predictor could reduce the cost while preserving the correlation with generalization. No future work is suggested on efficient expert discovery.

Single Benchmark, Single Model Family, Small Evaluation Set for the Central Correlation Claim

The constraint. The paper's headline finding — that concept expertise X_γ⋆ correlates with downstream task performance at r² = 0.833 — is established on a narrow experimental foundation:

• 4–5 encoder-only models are used in the correlation analysis (BERT-Base, BERT-Large, DistilBERT, RoBERTa-L, with XLM added where it reports scores; Table 2, column 1). The GPT2 models (Small, Medium, Large) are evaluated for expert units (Table 1) but excluded from the generalization correlation because they lack reported GLUE/SQuAD scores.
• A single benchmark family (GLUE + SQuAD) is used both to optimize γ⋆ (Section 4.2) and to evaluate the correlation (Table 2). The 60/40% cross-validation across tasks demonstrates stability within this suite but provides no evidence that γ⋆ or the X_γ⋆–generalization relationship would hold for other task types (summarization, translation, dialogue, code generation).
• A single concept dataset (SentenceConcepts, derived from OneSec with WordNet senses) is used to compute expertise. All 1,641 concepts are English WordNet senses, predominantly concrete nouns and verbs with clear lexical anchors.

The consequence. An r² computed on 4–5 data points is statistically fragile — a single model with unusual properties could drive the correlation up or down substantially. The paper reports no confidence intervals, p-values, or permutation tests for the r² values, making it impossible to assess whether r² = 0.833 is reliably different from, say, r² = 0.6 given the tiny sample size. If RoBERTa-L were removed from the analysis (it has the highest or second-highest X_γ⋆ and the highest GLUE score by a substantial margin; Table 1 and Table 5), the correlation might change dramatically — but we cannot know because the paper doesn't report leave-one-out sensitivity.

The exclusion of GPT2 models from the correlation analysis means we don't know whether expertise predicts generalization for autoregressive language models, which now dominate the NLP landscape. GPT2-L achieves X_γ⋆ = 12.92% (Table 1) — comparable to RoBERTa-L at 12.92% — but without downstream task scores for GPT2, we can't verify whether this expertise translates to comparable fine-tuned performance. The relationship might be architecture-specific, with masked language models (BERT, RoBERTa) developing expert units that transfer differently to downstream tasks than autoregressive models (GPT2).

The reliance on WordNet concepts means the expertise metric captures the model's ability to represent WordNet-style lexical concepts — concrete objects, actions, properties with clear definitional boundaries. It may not capture expertise for relational concepts, event structures, discourse-level phenomena, or any knowledge not expressible as "this sentence contains concept X." A model could have low X_γ⋆ because it represents knowledge in forms not captured by WordNet sense distinctions, not because it generalizes poorly.

Evidence in the paper. Table 2 explicitly lists which models are used for each task (column 1): BERT-B/L, DistilBERT, and RoBERTa-L for most tasks, with XLM added where available. Table 1 shows that GPT2 models have been analyzed for expert units but are absent from Table 2. Section 4.2 describes the γ⋆ optimization using "the 10 datasets composing GLUE with their different reported metrics and SQuAD v1.1/2.0" — no other task families are considered. The paper does not report confidence intervals for any correlation coefficient.

Mitigation status. The paper does not acknowledge the small sample size as a limitation. The cross-validation across 60/40% task splits (Section 4.2) addresses the stability of γ⋆ across task subsets but does not address the stability of the correlation itself across model subsets. The paper states the analyzed models are "representative of the capabilities of many contemporary LLMs" (Section 4) but provides no evidence for this representativeness claim. Section 8 suggests "the proposed concept dataset can be easily enriched for finer model comparison" but does not propose testing on non-GLUE tasks or generative model benchmarks.

The Correlation vs. Causation Gap: Expertise May Reflect Rather Than Cause Generalization

The constraint. The paper demonstrates a correlation between concept expertise X_γ⋆ and downstream task performance (r² = 0.833; Table 2), and frames this as evidence that "the presence of expert units is related to the knowledge acquired by TMs and to their performance" (Section 1). However, correlation does not establish the direction of causation — and the paper's experimental design cannot distinguish between several competing explanations:

• Expertise causes generalization: The paper's implicit model — diverse, high-quality concept detectors compose to solve novel tasks.
• Generalization causes expertise: Models that generalize well (due to better architecture, training data, or optimization) happen to develop more interpretable, locally-encoded concept representations as a byproduct, but these experts are not causally responsible for the generalization.
• A third factor causes both: Factors like training data diversity, model capacity, or optimization quality independently improve both generalization and the formation of expert units, creating a spurious correlation.

The consequence. If expertise is merely a symptom of good training rather than a mechanism of generalization, then X_γ⋆ is still useful as an evaluation metric (it correlates with what we care about), but the paper's stronger mechanistic claims — "generalization is related to the presence of extremely good and diverse experts" (Section 4.2) and the implication that improving expertise would improve generalization — are unsupported. A practitioner might invest effort in training models to develop more expert units, expecting this to boost downstream performance, when in fact both are driven by an unmeasured confound (e.g., training data quality) and manipulating one would not affect the other.

The expert forcing experiments (Section 6) provide causal evidence that expert neurons control concept expression in generation, but this is a different causal claim — it shows that experts are causal for concept-specific output, not that the density of experts across many concepts is causal for generalization across tasks. A model could have many causally effective concept experts but still fail to compose them effectively for a novel task; conversely, a model could generalize well through distributed, non-expert representations that the X_γ⋆ metric fails to capture.

Evidence in the paper. The paper's primary evidence for the expertise–generalization link is the correlation in Table 2. The paper also notes that model size correlates poorly with several tasks (r² < 0.3 for MRPC, STS-B, QNLI) while X_γ⋆ maintains r² > 0.75 (Table 2), arguing that expertise captures something beyond scale. However, this comparison rules out only one potential confound (parameter count); it does not rule out training data quality, training objective, or architectural inductive biases as alternative explanations.

Mitigation status. The paper does not address the causal direction question. The expert forcing experiments (Section 6) establish that individual experts are causal for their specific concepts, and the paper seems to treat this as sufficient evidence that expertise in aggregate causes generalization — but this is a logical leap from "these neurons cause this concept to appear in text" to "the number of such neurons causes the model to perform well on diverse tasks." No intervention experiment manipulates expertise and measures the effect on generalization (e.g., by pruning non-expert neurons and showing preserved performance, or by showing that models with matched size but different expertise — through different training recipes — show expertise-predicted performance differences). The paper does not discuss this distinction.

The Expert Forcing Mechanism Is Brittle and Uncharacterized

The constraint. The conditional generation results (Section 6; Table 4; Figure 1; Appendix D) demonstrate that expert forcing works for selected high-AP concepts, but the paper provides almost no characterization of when and how reliably it works:

• The optimal K varies per concept and is found by human inspection. For bird%1:05:00, saturation occurs around K = 200 (Table 4). For football%1:04:001, K = 50 produces the Figure 1 output. For Table 8 concepts, K = 60 is used uniformly. There is no principled method for choosing K given a concept's AP⋆_c or the number of available experts — a practitioner must guess and inspect.
• The forcing value (median active value from positive sentences) has unknown sensitivity. The paper uses the median with no justification beyond robustness to outliers, and does not explore whether some concepts require higher or lower forcing targets, or whether the median from the input sentences (which is what's used; Section 6) is appropriate for controlling output generation.
• Generation quality varies substantially across concepts. Table 8 outputs range from coherent (gold%1:21:00 produces readable text about Canadian Gold Corporation) to borderline (frustration%1:12:00 produces text about a "lone unicorn on the walls of a remote mountain, wearing brightly red patches of clothing" — the connection to frustration is tenuous). The paper provides no measure of fluency, coherence, or concept relevance beyond the selected examples.
• The evaluation is purely qualitative on a handful of hand-picked examples. No automated metric (concept classifier accuracy, keyword counting, human rating) is applied. No success rate is reported. We don't know whether the examples represent typical performance or cherry-picked best cases.

The consequence. A practitioner attempting to use expert forcing for conditional generation faces several unresolved practical questions: How many experts should I force for concept X? Will the generated text be coherent, or will it degenerate into repetition? Does this work for concepts I care about, or only for the specific WordNet senses where AP⋆_c is very high? Without answers to these questions — which the paper does not provide — expert forcing remains a compelling demonstration rather than a deployable technique.

More fundamentally, the paper does not compare expert forcing against any baseline conditioning method. A simple alternative — prepending "Write about birds:" to the context — might achieve comparable or better concept induction with no expert discovery cost. Without this comparison, we cannot assess whether expert forcing is better than trivial prompting or merely different. The paper claims novelty ("the first to condition an off-the-shelf pre-trained LM without fine-tuning, re-training or using additional parameters"; Section 1), but novelty does not imply practical superiority.

Evidence in the paper. Table 4 shows K values of 0, 40, 60, 80, and 200 for bird%1:05:00, with no explanation for why these specific values were chosen. Table 8 uses K = 60 for four concepts. Figure 1 uses K = 50 for football%1:04:001. The paper states "the larger K, the more concept c will be present in the output, provided that the expert has learnt the concept. Too large K will result in illegible sentences, since p(y = c|x) will dominate p(x) in Eq. (5)" (Section 6), but provides no guidance for finding the sweet spot between "not enough concept" and "illegible." No baseline conditioning method is tested.

Mitigation status. The paper does not acknowledge these as limitations. The generation experiments are presented as proof-of-concept demonstrations of the causal efficacy of expert units, not as a fully characterized conditioning method. Section 8 lists "conditioning language models on their own knowledge" as a future research avenue, implicitly acknowledging that the technique needs further development. However, the lack of quantitative evaluation, K-selection guidance, or baseline comparisons means the practical utility of expert forcing for conditional generation is unproven — the results establish that experts are causal but do not establish that expert forcing is a useful conditioning method compared to simpler alternatives.

Homograph Disambiguation Remains Unsolved, and the Paper Offers No Path Forward

The constraint. The paper explicitly introduces homograph concepts as the harder case — distinguishing different senses of the same word — and the results confirm this difficulty. Across all 10 models analyzed, the highest homograph expertise achieved is 5.72% (for BERT-Base, BERT-Large, and DistilBERT at γ⋆ = 0.985; Table 1). This means that for ~94% of the 297 homograph concepts, no model contains a single neuron that can distinguish the two senses with AP ≥ 0.985. RoBERTa-Lm, the model with the highest sense expertise (17.86%), achieves only 4.04% homograph expertise.

The consequence. The expert unit discovery method — treating individual neurons as binary classifiers — fundamentally struggles with concepts that require contextual disambiguation rather than keyword detection. This is not a minor edge case: homograph disambiguation is a core linguistic competence that humans perform effortlessly (understanding that "bank" means financial institution in "I deposited money at the bank" vs. river edge in "I sat on the bank"). The paper's method provides no insight into how models actually perform homograph disambiguation in practice (since pre-trained models like BERT do disambiguate homographs successfully on downstream tasks, just not via single-neuron experts), and offers no technique for improving it. The paper speculates that "more complex concepts such as homograph require a more complex expert such as a set of units" (Section 7) but does not explore what such a multi-unit expert would look like or how to discover it.

For the conditional generation use case, the homograph failure is particularly acute: the forcing method cannot reliably induce a specific sense of an ambiguous word. Forcing experts for bank%1:14:00 (financial institution) would likely fail because the model lacks high-AP single-neuron experts for this concept — and even if it partially succeeded, the generated text might oscillate between financial and river senses because the forcing mechanism has no way to suppress the incorrect sense.

Evidence in the paper. Table 1 reports homograph expertise values ranging from 1.35% (GPT2-Small) to 5.72% (BERT-B/L, DistilBERT). Figure 4-I shows the AP⋆_c distribution for homographs: it is broad and shifted substantially lower than the sense distribution, with many concepts having AP⋆_c < 0.8. Figure 3-II and 3-IV show that homograph concepts are sparsely distributed across layers, with very few in early Transformer blocks — suggesting that whatever mechanism handles homograph disambiguation is not localized in the same way as sense concept detection. The paper acknowledges this as an open challenge: "Building pre-trained models inherently able to disambiguate homograph concepts at unit level remains a challenge" (Section 4.3.2).

Mitigation status. The paper identifies the homograph problem clearly and does not overclaim success. However, it offers no mitigation strategy beyond the speculation that multi-unit experts might help (Section 7). The limitation is fundamental to the individual-neuron-as-classifier paradigm: if homograph disambiguation requires comparing contextual evidence distributed across many neurons (e.g., one set of neurons detects financial vocabulary, another detects nature vocabulary, and their relative activation determines the sense), then no single neuron will achieve high AP, and the method's binary "acquired vs. not acquired" framework cannot capture this capability. Future work would need to extend the expert definition to groups or circuits of neurons, which would substantially increase the search complexity (from O(M) per concept to O(M^k) for groups of size k) and require a different evaluation framework than per-neuron AP.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a fundamentally different approach to understanding pre-trained Transformer models: rather than probing representations with external classifiers or visualizing attention patterns, it treats individual neurons as the classifiers themselves and measures how well they already detect specific concepts. This is not an incremental improvement to existing interpretability tools — it is a conceptual reframing that changes what questions we can ask about model internals.

From "what does the model know?" to "which specific neurons know what?" Prior probing work (Conneau et al., 2018; Tenney et al., 2019; Liu et al., 2019) could establish that a model's representations encode linguistic properties, but only by training auxiliary classifiers on top of those representations. This leaves a persistent ambiguity: did the probe extract knowledge the model actually uses, or did it learn to detect weak signals the model itself ignores? The neuron-as-classifier paradigm eliminates this ambiguity for the subset of concepts where single-neuron experts exist — when neuron 4,217 in layer 8 achieves AP = 0.9999 on elevator%1:06:00 (Table 10, Appendix E), there is no probe to blame. The neuron is the elevator detector. This transforms the interpretability question from "does the model know X?" (answer: probably yes, somewhere in the distributed representation) to "does the model have a dedicated detector for X, and if so, where exactly?" The shift from distributed to localized interpretability enables a new class of analyses — per-layer concept distributions (Figure 3), per-concept expert counts (Figure 4-II), concept overlap metrics (Section 5) — that would be impossible if knowledge were assumed to be purely distributed.

A new axis for model evaluation that bypasses fine-tuning. The finding that concept expertise X_γ⋆ correlates with downstream task performance at r² = 0.833 — higher than the average inter-task correlation among GLUE/SQuAD tasks themselves (r² = 0.826; Table 2) — provides the first evidence that model quality can be estimated by inspecting internal representations for the presence of specialized detectors, without any task-specific adaptation. This is not just a cheaper evaluation method (though the paper argues it is computationally competitive with full GLUE+SQuAD evaluation; Section 7). It is a different kind of evaluation that measures a model's internal conceptual vocabulary rather than its behavioral outputs after fine-tuning. This decoupling is significant because it means model developers could potentially assess a pre-trained checkpoint's quality before deciding whether to invest in downstream fine-tuning, and could diagnose which types of concepts a model has learned well (sense concepts vs. homographs; Table 1) rather than receiving a single aggregate score. The comparison to model size (average r² = 0.408; Table 2) demonstrates that parameter count alone is a poor substitute for this kind of internal inspection — expertise captures how effectively parameters have been organized into specialists, not just how many parameters exist.

Reconciling contradictory findings about interpretability in NLP. The paper does not explicitly frame this as a reconciliation, but it implicitly resolves a tension in the interpretability literature: some works find that individual components are interpretable (attention heads attending to syntactic relations; Clark et al., 2019), while others find that representations are irreducibly distributed (probing classifiers finding information in linear combinations of many neurons; Tenney et al., 2019). The expert unit results suggest both views are correct, depending on the concept type and the model quality. For sense concepts (keyword detection) in well-trained models like RoBERTa-Lm, individual neurons achieve near-perfect classification (AP⋆_c ≥ 0.997 for 17.86% of concepts), supporting the localized-interpretability view. For homograph concepts (contextual disambiguation) and for weaker models, expertise is rare or nonexistent (max 5.72% homograph expertise across all models; Table 1), supporting the distributed-representation view. The paper thus provides a unified framework where the degree of localization is not a property of the architecture or the interpretability method, but of the specific concept-model pair — some concepts are encoded locally, others distributively, and the threshold γ⋆ tells you how localized "localized" needs to be for expertise to matter for generalization.

Redirecting research attention toward internal model organization. The paper's architectural findings — that B layers acquire ~3.5× more concepts than A layers, that A_proj and B_proj layers acquire very few, and that sense concepts concentrate in shallow layers while homographs are more distributed (Figure 3; Appendix A) — shift interpretability research away from attention patterns (the dominant focus; Clark et al., 2019; Ghaeini et al., 2018) and toward the feed-forward layers as the primary locus of conceptual knowledge. The finding that attention projections (A_proj, B_proj) contribute minimally to concept expertise challenges the implicit assumption in much attention-analysis work that self-attention is where semantic knowledge resides. Instead, the expanding feed-forward layers (A and especially B) appear to be where concepts are encoded at the neuron level, with attention serving a routing or contextualization function. This reframes the Transformer not as "attention + feed-forward" where attention does the interesting work, but as "feed-forward concept detectors + attention-based context integration" — a more specific mechanistic hypothesis that future work can test.

Establishing that pre-trained LMs contain latent conditional generation capabilities. The expert forcing results (Section 6; Table 4; Figure 1) demonstrate something genuinely surprising: an off-the-shelf GPT2 model, with no training for conditional generation, can be made to generate text about football%1:04:001 by forcing just 50 neurons (0.012% of its units) to remain active. This means the model already "knows how" to condition on concepts — the conditional distribution p(x|y = c) is implicitly represented in its weights — and expert forcing merely amplifies the existing conditional signal relative to the unconditional language model prior. This finding should change how the field thinks about pre-trained models: they are not just unconditional text generators awaiting fine-tuning to become conditional; they are latently conditional, and the challenge is discovering how to access that conditionality without retraining. This opens a new research direction distinct from both fine-tuning (which changes the model) and prompting (which changes the input): manipulating internal activations to access capabilities already present in the weights.

Research directions that become more attractive:

• Neuron-level interpretability as a standard model diagnostic. The paper provides a complete recipe — dataset construction, AP computation, threshold optimization, expertise scoring — that other researchers can apply to new models, new concept datasets, and new domains. This lowers the barrier to neuron-level analysis from "design a new probing task and train a classifier" to "run forward passes and compute AP."
• Feed-forward layer analysis over attention analysis. The finding that concept expertise concentrates in A and B layers (especially B layers) suggests that future interpretability work should focus more on the feed-forward computations and less on attention patterns, reversing the emphasis of the past several years.
• Causal interventions as the standard of evidence for interpretability claims. The expert forcing experiments demonstrate that causal manipulation of identified neurons produces the expected behavioral change, setting a higher evidentiary bar than correlation-based methods. Future work claiming to have "found the X detector" should demonstrate that manipulating it controls X in the output.

Research directions that become less attractive:

• Purely correlational probing without causal validation. The paper's forcing results show that correlation (high AP) and causation (concept induction under forcing) align for high-AP concepts — but this alignment is demonstrated, not assumed. Probing papers that stop at "we found a classifier that achieves high accuracy on property X" without testing whether manipulating the identified representations affects behavior leave the causal question unanswered.
• Uniform treatment of all Transformer layers as equally important for knowledge storage. The stark asymmetry between A/B layers and A_proj/B_proj layers (Figure 3) suggests that methods aggregating across all layers without distinguishing layer types are averaging away important structure.
• One-size-fits-all interpretability methods that don't distinguish concept difficulty. The 17× gap between sense and homograph expertise (Table 1) shows that any method claiming "Transformers represent concepts in way W" without distinguishing concept types is likely oversimplifying — distributed for homographs, localized for sense concepts in good models.

Follow-Up Research This Work Enables

Extending expert unit discovery to concept groups (circuits) for homograph disambiguation. The paper's individual-neuron approach fails for homograph concepts (max 5.72% expertise; Table 1), and the authors speculate that "more complex concepts such as homograph require a more complex expert such as a set of units" (Section 7). A natural follow-up would search for pairs or small groups of neurons whose joint activation pattern distinguishes homograph senses, even though no single neuron does. Concretely: for concept bank%1:14:00 VS. bank%1:17:01 (financial institution vs. river edge), compute AP for every pair of neurons (u_i, u_j) by using a simple combination rule — e.g., u_i − u_j or logistic regression on [u_i, u_j] — and measure whether any pair achieves AP ≥ γ⋆. This scales quadratically (M²/2 ≈ 86 billion pairs for GPT2-L's 414,720 neurons, though layer-wise restrictions could reduce this), but could reveal whether homograph disambiguation is implemented by opponent pairs (one neuron detects financial context, another detects nature context, and their difference encodes the sense). A successful demonstration — finding neuron pairs with AP ≥ 0.985 for, say, 30% of homograph concepts — would extend the expert unit framework from "localized experts" to "localized expert circuits" and substantially close the gap between sense and homograph expertise. A negative result (no pairs achieve high AP) would suggest homograph disambiguation relies on more distributed mechanisms, which would itself be an informative finding about the limits of localized interpretability.

Systematic characterization of the expert forcing dose-response curve, with automated evaluation. The paper's conditioning results (Table 4; Appendix D) are qualitative demonstrations on a handful of hand-picked concepts. A rigorous follow-up would automate the evaluation of conditioning success and characterize how success depends on K, AP⋆_c, concept type, and model. Specifically: for a representative sample of 100 concepts spanning the full range of AP⋆_c values (from 0.5 to 1.0, including both sense and homograph), sweep K from 0 to 500 (or until saturation), generate 50 samples per (concept, K) pair with different random seeds, and measure concept presence using a held-out concept classifier (e.g., a fine-tuned BERT trained on the SentenceConcepts dataset, or keyword counting for unambiguous terms). The key output would be a success@K curve per concept, showing the probability that generated text contains the target concept as a function of K. This would answer several open questions from the paper: (1) Is there a threshold AP⋆_c below which forcing never works, and is that threshold close to γ⋆? (2) Does the optimal K (highest concept presence before fluency degrades) correlate with the number of available experts above some AP threshold? (3) Do homograph concepts ever condition successfully (forcing one sense while suppressing the other), or is the failure in Table 1 insurmountable for current methods? A strong result — e.g., success@K > 0.8 for all concepts with AP⋆_c > 0.95 at K ≈ N_experts × 2 — would transform expert forcing from a compelling demo into a reliable technique. A negative result — e.g., high variance in success rates even for high-AP concepts, or frequent degeneration before reliable concept induction — would suggest that additional mechanisms beyond simple activation clamping are needed for practical conditional generation.

Held-out task evaluation of X_γ⋆ as a generalization predictor. The paper's correlation result (r² = 0.833; Table 2) is established on the same task suite (GLUE + SQuAD) used to optimize γ⋆, with only 4–5 encoder-only models, and without statistical significance testing. A critical stress-test would: (1) optimize γ⋆ on GLUE tasks, then test X_γ⋆ against entirely held-out tasks not used in optimization — e.g., SuperGLUE, a suite of reading comprehension datasets (NarrativeQA, RACE), or summarization metrics (ROUGE on CNN/DailyMail). (2) Expand the model set to include autoregressive models (GPT2-S/M/L, GPT-Neo, OPT variants) by fine-tuning them on the held-out tasks to obtain performance scores, even though this reintroduces fine-tuning (the expertise computation itself remains training-free; only the validation requires fine-tuning). (3) Report leave-one-out r² with confidence intervals to assess sensitivity to individual models. A positive result — r² remains > 0.7 on held-out tasks with 10+ models and confidence intervals exclude zero — would establish X_γ⋆ as a genuinely task-independent proxy for model quality, making it a standard diagnostic for pre-trained model releases. A negative result — r² drops substantially on held-out tasks or becomes non-significant with more models — would suggest that the relationship is specific to the GLUE/SQuAD task family or to encoder-only architectures, substantially limiting the metric's generality. Either outcome is valuable: the positive case enables cheaper model evaluation; the negative case clarifies that expertise captures something about GLUE-style natural language understanding specifically, not general-purpose language capability.

Difficulty prediction from the question text alone, amortizing the expert discovery cost. The paper's method for computing concept expertise requires running forward passes over 2.5 million sentences per model (6.15 GPU-hours for BERT-Base; Section 7), and the conditional generation pipeline requires identifying top-K experts per concept by computing AP for all 680 million neuron-concept pairs in GPT2-L. Both costs are upfront and substantial. A practical follow-up would train a lightweight model to predict, from a concept's WordNet definition or example sentences alone, whether a given model contains high-AP experts for that concept — essentially learning to predict AP⋆_c without running the full expert discovery pipeline. The training data already exists: for each of the 1,641 concepts and 10 models, the paper has computed AP⋆_c values (Appendix E, Tables 10–14 provide per-concept AP⋆_c for GPT2-L; similar data exists for other models in the paper's computed results). A researcher could train a linear classifier or small transformer that takes as input (a) the WordNet definition embedding, (b) embeddings of 10–20 example positive sentences, (c) the model architecture and size, and predicts whether AP⋆_c ≥ γ⋆. If such a predictor achieves, say, 80% accuracy in held-out concept evaluation, it could serve as a cheap pre-filter: spend the full forward-pass budget only on concepts predicted to have experts, dramatically reducing the cost of applying the method to new models or new concept datasets. This would address the paper's most significant practical limitation — the upfront compute cost — and make expert-based evaluation and conditioning feasible for interactive or resource-constrained settings.

Comparing expert forcing against prompting baselines for conditional generation. The paper claims expert forcing is the first method to condition pre-trained LMs without training (Section 1), but it never compares against the simplest training-free alternative: prompting. A rigorous comparison would take the same concepts used in the paper's generation experiments (bird%1:05:00, elevator%1:06:00, smoke%1:19:00, gold%1:21:00, frustration%1:12:00, football%1:04:001, and the failure case work%1:06:00) and generate text using several prompting strategies: (a) prepending "Write about {concept word}:" to the context, (b) using the concept as the first generated token, (c) providing a definition-then-example prompt ("{WordNet definition}. For example:"). For each strategy, generate 100 samples and measure concept presence using the automated classifier from the dose-response follow-up described above. Compare against expert forcing at the optimal K for each concept (determined by the dose-response characterization). The key question: does expert forcing produce more or more reliable concept induction than simple prompting, and at what cost in fluency? If prompting achieves comparable concept presence for high-AP concepts (e.g., "Write about birds:" already induces bird-related text in GPT2-L), then expert forcing's contribution is incremental rather than transformative, and its practical value lies primarily in enabling fine-grained control (gradual K-adjustment) rather than concept presence per se. If prompting fails for concepts where expert forcing succeeds (e.g., abstract concepts like frustration%1:12:00 where the concept word alone doesn't reliably induce the concept), expert forcing fills a genuine gap. Either outcome provides essential context for the paper's claims and guides practitioners on when to use each approach.

Manipulating expert units for model editing: erasing or amplifying specific concepts. The paper demonstrates that amplifying expert neurons induces their concept in output. The natural inverse is to ask: can we suppress a concept by clamping its expert neurons to low (concept-absent) values? And can we do this selectively — suppressing bank%1:14:00 (financial) while leaving bank%1:17:01 (river) unaffected? This would extend expert forcing from conditional generation to controlled generation with negative constraints: generate text that is about topic A but not about topic B. A concrete experiment: for a pair of related concepts like dog%1:05:00 (the animal) and cat%1:05:00 (the animal), identify the top-K experts for each concept. Then generate text with context "I have a pet" under three conditions: (1) no forcing (baseline), (2) force dog experts active + cat experts suppressed, (3) force cat experts active + dog experts suppressed. Measure whether generated text mentions the intended animal more often and the suppressed animal less often compared to baseline. This tests whether the model's concept experts are independent enough to allow selective manipulation — if dog and cat experts overlap substantially (high Ω(dog, cat) in the overlap metric; Section 5), simultaneous activation of one and suppression of the other may be impossible, revealing a fundamental limit on selective control. A positive result (selective control works for concepts with low expert overlap) would open the door to "concept editing" in pre-trained models — enhancing or suppressing specific knowledge without retraining — which has direct applications in bias mitigation (suppress stereotyped associations) and content filtering.

Practical Applications and Downstream Use Cases

Model selection and quality assurance for pre-trained model releases. Organizations releasing pre-trained models (HuggingFace, Meta, Google, etc.) currently report performance on standardized benchmarks (GLUE, SuperGLUE, MMLU) that require task-specific fine-tuning heads and hyperparameter optimization. The concept expertise metric X_γ⋆ offers a complementary evaluation that requires no fine-tuning — just forward passes over the SentenceConcepts dataset (or an extended version) and AP computation. A model hub could compute X_γ⋆ for every uploaded checkpoint (at a cost of ~45 GPU-minutes with 8-way parallelization; Section 7) and display it alongside traditional benchmarks, providing users with an immediate signal of model quality that doesn't depend on the specific fine-tuning procedure used. The per-concept-type breakdown (sense expertise vs. homograph expertise; Table 1) provides richer diagnostic information than a single aggregate score: a user needing strong word-sense disambiguation (e.g., for legal document analysis) could select models with high homograph expertise, while a user doing keyword-based retrieval might prioritize sense expertise. The paper's finding that RoBERTa-L doubles BERT-L's sense expertise (14.66% vs. 7.51%) despite similar parameter counts (355M vs. 330M) demonstrates that X_γ⋆ captures training-quality differences that parameter count alone misses — making it a genuine value-add for model comparison.

Cost-efficient evaluation in model development cycles. During pre-training, researchers typically evaluate intermediate checkpoints by fine-tuning on downstream tasks — a process that, for large models, can take longer than the pre-training itself for a single evaluation. The expertise metric offers a lightweight alternative: run the expert discovery pipeline on intermediate checkpoints (e.g., every 10K training steps) and track X_γ⋆ over the course of training. If X_γ⋆ plateaus while training loss continues to decrease, this might indicate that the model is memorizing surface statistics without developing new conceptual specialists — a sign of overfitting or training data saturation that loss curves alone wouldn't reveal. Conversely, if X_γ⋆ continues to increase late in training, it suggests the model is still building useful conceptual structure and training should continue. This use case doesn't require X_γ⋆ to perfectly predict downstream performance — it only needs to be a useful relative signal within a single training run, which is a weaker requirement than the paper's cross-model correlation claim. Development teams could integrate expertise tracking into their training loops with moderate compute overhead (6 GPU-hours per evaluation point, parallelizable), potentially saving far more compute by avoiding unnecessary fine-tuning evaluations or by catching training failures earlier.

"Concept probe" for debugging model failures in production. When a deployed model (dialogue system, content moderator, information extraction pipeline) produces an error on a specific input, operators currently have limited tools for understanding why: they can examine the input text, the model's output, and maybe attention visualizations, but they cannot ask "does this model understand concept X?" in a systematic way. The expert unit framework provides a concrete diagnostic: for any concept of interest (e.g., a failure mode where the model confuses "complaint" as a legal filing vs. "complaint" as an expression of dissatisfaction), an operator can construct a small dataset of positive and negative sentences (100 each would likely suffice; the paper uses 100–1000), run forward passes, compute AP for all neurons, and determine whether the model contains a high-AP expert for the disambiguating concept. If AP⋆_c < 0.9, the model simply doesn't have a specialized detector for this distinction — the failure is a capability gap, not a one-off error, and the fix is likely further pre-training or fine-tuning on disambiguation data. If AP⋆_c > 0.99 but the model still fails, the issue is in how the expert's signal is used downstream (e.g., the classification head isn't attending to the right features), and the fix might involve architectural changes or fine-tuning the task-specific layers while keeping the expert intact. This transforms error analysis from guesswork into a systematic capability audit.

Lightweight concept-conditioned text generation for creative and assistive applications. While the paper's forcing results are preliminary, the demonstrated ability to induce concepts with as few as 40–200 forced neurons (0.009–0.048% of units; Table 4, Figure 1) suggests a practical tool for applications where full model retraining is infeasible but concept-conditioned generation is desired. For example: a writing assistant that helps authors maintain consistent thematic elements ("generate the next paragraph, but make sure it mentions the character's background as a journalist%1:18:00") could use expert forcing on the journalist-concept neurons during decoding, without requiring a fine-tuned conditional model for every possible character profession. The upfront cost — computing AP for the concepts of interest — would be paid once per concept and amortized across many generation requests. The key practical requirement is that the concepts of interest have high AP⋆_c (likely true for concrete professions, objects, and actions, given the sense expertise results in Table 1; likely false for abstract or relational concepts). This use case would benefit substantially from the dose-response characterization proposed above, allowing the system to automatically select K to achieve a target concept strength without human trial-and-error.