Language Models as Knowledge Bases?

ArXiv: 1909.01066

🎯 Pitch

BERT-large recalls factual knowledge at a level that rivals a supervised relation extraction system equipped with an oracle entity linker, yet it achieves this without any fine-tuning or schema engineering. However, this capability is highly uneven—BERT correctly answers 74.5% of one-to-one relation queries but only 24.3% of many-to-many relations, exposing fundamental limitations in how standard pretraining encodes different types of world knowledge.

1. Executive Summary

This paper analyzes whether pretrained language models—without any fine-tuning—already store relational knowledge that can be recovered by querying them with cloze-style prompts. Using the LAMA probe across four knowledge sources (Google-RE, T-REx, ConceptNet, and SQuAD) and comparing models including BERT, ELMo, Transformer-XL, and fairseq-fconv, the work evaluates how well models recall facts by predicting masked objects in templates like "Dante was born in [MASK]." The central finding is that BERT-large contains relational knowledge competitive with a traditional relation extraction baseline augmented with an oracle entity linker—achieving 32.3% mean precision@1 on T-REx compared to 33.8% for the supervised baseline—and reaches 57.1% P@10 on open-domain QA, close to DrQA's supervised 63.5%, establishing that language models can serve as unsupervised knowledge bases only for certain relation types, with performance varying dramatically between 1-to-1 relations (74.5% P@1) and N-to-M relations (24.3% P@1).

2. Context and Motivation

The Core Problem: Can Language Models Replace Structured Knowledge Bases?

The fundamental question this paper tackles is disarmingly direct: do pretrained language models already store factual and commonsense relational knowledge in their parameters, and if so, can we recover it simply by querying them with natural language prompts? This is not a question about fine-tuning or task-specific adaptation—it asks what knowledge is already there in off-the-shelf models that researchers download and use as starting points for downstream work.

The practical stakes are high. Traditional knowledge bases like Wikidata, Freebase, or domain-specific KGs require enormous engineering effort: schema design, entity extraction pipelines, coreference resolution, entity linking, and relation extraction—each component needing supervised data, manual annotation, and careful tuning. Errors compound across pipeline stages. As the authors note in their introduction, "errors can easily propagate and accumulate throughout the pipeline." In contrast, language models offer an alluring shortcut: no schema engineering, no human annotation, no explicit entity linking, and support for an open set of relations. If a language model trained on raw text can serve as a knowledge base simply by answering fill-in-the-blank questions, the entire pipeline collapses into a single pretrained model.

This question also matters for how we understand what these models actually learn. By 2019, when this paper was published, BERT and similar models had revolutionized NLP through transfer learning—but the field's understanding of what knowledge they acquired was largely limited to linguistic and syntactic properties (Peters et al., 2018b; Goldberg, 2019; Tenney et al., 2019). Whether they also encoded world knowledge—facts about entities, commonsense relationships, event sequences—was an open question with far-reaching implications for how we should design systems that need to reason about the world.

Where Prior Work Fell Short: A Taxonomy of Gaps

The paper identifies several distinct shortcomings in the existing literature:

Gap 1: The linguistic knowledge focus. Most prior analysis of pretrained models concentrated on syntax, semantics, and transfer learning to NLP benchmarks. Marvin and Linzen (2018) tested whether LSTMs learned grammaticality distinctions. Peters et al. (2018b) showed that ELMo's lower layers capture local syntax while upper layers model long-range dependencies. Goldberg (2019) demonstrated BERT's strong grasp of English syntactic phenomena. Tenney et al. (2019) found that language models excel at encoding sentence structure for syntactic phenomena but provide limited gains for semantic tasks. All of this work asked "do these models understand language structure?"—not "do they know facts about the world?"

The authors are careful to position their work as complementary: "while this provides insights into the linguistic knowledge of language models, it does not provide insights into their factual and commonsense knowledge." This is not a criticism of prior work but an observation that a crucial dimension of model capability remained entirely unexamined.

Gap 2: Transfer learning doesn't equal knowledge recovery. The GLUE benchmark (Wang et al., 2018) and similar evaluation suites tested whether pretrained representations were useful feature extractors for downstream tasks like sentiment analysis, natural language inference, and paraphrase detection. But strong transfer performance doesn't necessarily mean the model knows facts—it could mean the model has learned useful linguistic features that make it easier for a task-specific head to learn from limited supervised data. The distinction is between "this model provides good representations" and "this model can answer factual questions." The latter requires the model to have actually encoded specific world knowledge in its weights, not just generalizable linguistic patterns.

Gap 3: GPT-2's hints were tantalizing but incomplete. The closest prior work to this paper's investigation was Radford et al. (2019)'s analysis of GPT-2's zero-shot capabilities. GPT-2 achieved an F1 of 55 on CoQA and 4.1% accuracy on Natural Questions without any fine-tuning or information retrieval, suggesting that very large language models do memorize factual knowledge. However, as the authors point out, the large GPT-2 model was not publicly available, and the publicly released small version achieved less than 1% on Natural Questions—"5.3 times worse than the large model." This created a gap: the community had suggestive evidence from an inaccessible model but no systematic study of what publicly available models actually know. Moreover, GPT-2's zero-shot evaluation provided only aggregate accuracy numbers, not the fine-grained, relation-by-relation analysis that would reveal which kinds of knowledge are stored and which are not.

Gap 4: No direct comparison to traditional knowledge extraction. Even if language models can answer some factual questions, it was unclear whether they could compete with dedicated knowledge extraction pipelines. A relation extraction system—trained to identify relational triples from text and store them in a structured knowledge base—represents the traditional approach. If a language model's factual recall is far worse than what an off-the-shelf RE system can extract from the same text, then the language model isn't a practical alternative. Prior work provided no such comparison, making it impossible to assess whether language models represent a viable path toward unsupervised knowledge bases.

Gap 5: Difficulty estimation across relation types was absent. Existing work treated factual knowledge as monolithic. In practice, relations vary enormously in character: 1-to-1 relations (each country has one capital) are fundamentally different from N-to-M relations (an author writes many books, a book has many authors). A language model might excel at the former while failing completely at the latter, but no prior study had broken down performance by relation type to characterize this variation. Without this breakdown, aggregate accuracy numbers could be deeply misleading—high performance on a few simple relation types could mask near-zero performance on the rest.

How This Paper Positions Itself

The paper's positioning is carefully constructed along several axes:

As a probe, not a proposed system. The LAMA probe is introduced as an analysis tool, not as a deployment-ready knowledge base or a claim that language models should replace KGs. The paper repeatedly uses language like "we are interested in the relational knowledge already present" and "without any fine-tuning." This is deliberate: the goal is to measure what's already there, not to demonstrate that a particular technique recovers knowledge better than others. The probe nature also justifies design choices that might seem suboptimal for a production system—manual templates, single-token objects, unified vocabulary—because simplicity and interpretability matter more than maximizing scores.

As complementary to the linguistic analysis literature. The paper explicitly frames itself as extending the line of work that investigates what pretrained models learn, rather than proposing a new architecture or training method. It cites Peters et al. (2018b), Goldberg (2019), Tenney et al. (2019), and McCoy et al. (2019) as the relevant intellectual context, then states that it fills the gap left by these studies: factual and commonsense knowledge rather than linguistic knowledge. This framing matters because it positions the contribution within an established, respected line of research rather than as an isolated empirical study.

As a systematic, multi-dimensional evaluation. Unlike GPT-2's aggregate zero-shot scores, the LAMA probe is designed to provide fine-grained results: per-relation accuracy, per-relation-type breakdowns (1-to-1, N-to-1, N-to-M), rank distributions, sensitivity to query phrasing, and correlations between accuracy and properties like mention frequency. This enables the paper to make nuanced claims about which knowledge is stored and under what conditions it can be recovered, rather than blanket statements about whether language models "know facts."

As establishing a lower bound, not an upper bound. The authors explicitly argue that their manual templates and single-token restriction make their evaluation a conservative estimate of what language models know: "we argue that this means we are measuring a lower bound for what language models know." The analogy is to traditional knowledge bases, which also have a single query interface (the relation ID) and can't handle paraphrases. If a fact can be recovered under these constraints, it's strong evidence it's stored; if it can't, it might still be stored but expressed differently. This rhetorical move anticipates and deflects the criticism that template choice biases the results.

As opining on a possible future, not claiming the present. The paper's conclusion is carefully measured: "language models trained on ever growing corpora might become a viable alternative to traditional knowledge bases extracted from text in the future." This is not a claim that BERT replaces KGs today—the results showing poor performance on N-to-M relations and the 38.5% gap to DrQA on P@1 for SQuAD make that clear. Rather, it's an argument that the trend line is promising and that researchers should track this capability as models scale, using probes like LAMA to measure progress.

The Conceptual Framework: A Direct Competitor to Symbolic Knowledge Bases

Beyond the specific gaps, the paper operates within a broader intellectual tension that was especially acute in 2019 NLP: the competition between symbolic and neural approaches to knowledge representation. Traditional knowledge bases (Freebase, Wikidata, ConceptNet) represent facts as explicit triples (subject, relation, object) stored in structured databases. They're precise, interpretable, and support exact queries—but they're expensive to build and maintain. Neural language models represent knowledge implicitly in high-dimensional parameter vectors. They're cheap to train (relatively), support fuzzy queries through natural language, and automatically cover whatever relationships are in the training data—but they're opaque, inconsistent, and their knowledge is entangled with linguistic patterns.

The paper essentially asks: how far has the neural approach already come, without anyone explicitly trying to make it into a knowledge base? The LAMA probe provides the first rigorous empirical comparison point between these two paradigms, using the same data to test both. If BERT—trained only on a language modeling objective, never explicitly taught to extract or store relational triples—can match a purpose-built relation extraction pipeline with oracle entity linking, that's strong evidence that the neural knowledge representation has progressed further than the field realized. If it fails badly on certain relation types, that reveals fundamental limitations of implicit knowledge storage that hybrid or symbolic approaches might need to address.

Why 2019 Was the Right Moment for This Investigation

Several converging factors made this study timely:

BERT had just been released and was rapidly becoming the default starting point for NLP research. Understanding what it already knew—before fine-tuning—was critical for thousands of downstream practitioners.
Model sizes were growing rapidly, with BERT-large at 340M parameters training on 3.3B words, and even larger models (GPT-2 at 1.5B parameters) on the horizon. If knowledge storage scales with model size, we needed baselines to measure that scaling.
The field was debating what "understanding" means for neural models. McCoy et al. (2019) had just shown that BERT relies on fallible syntactic heuristics for natural language inference rather than genuine understanding. Knowing whether and how models store factual knowledge was part of this larger conversation about the nature of neural representations.
Unsupervised learning was ascendant. The success of BERT and GPT had demonstrated that pretraining on unlabeled text could replace supervised feature engineering. If factual knowledge was also emerging from unsupervised pretraining, that would extend the reach of unsupervised learning from linguistic features to world knowledge.

The paper's core question—"Language Models as Knowledge Bases?"—was thus perfectly positioned at the intersection of empirical model analysis, knowledge representation, and the growing excitement about emergent capabilities of large-scale pretraining.

3. Technical Approach

3.1 Reader Orientation

This paper introduces LAMA (LAnguage Model Analysis probe)—an evaluation framework that measures how much factual and commonsense knowledge pretrained language models store in their parameters, without any fine-tuning. The "system" is not a model being built, but a measurement apparatus: a set of fact databases, a method for converting facts into natural language queries, and a scoring protocol that reveals whether models can recall correct answers. The core problem is that we don't know what world knowledge exists in off-the-shelf pretrained models, and the solution takes the shape of a standardized test suite that queries models with cloze statements (fill-in-the-blank sentences) and ranks their predictions against a common vocabulary—enabling direct comparison across model architectures, training corpora, and knowledge types.

3.2 Big-Picture Architecture

The LAMA framework has four major components connected in a pipeline:

Knowledge Sources — Four databases of facts (Google-RE, T-REx, ConceptNet, SQuAD) that provide ground-truth (subject, relation, object) triples or question-answer pairs. Each source captures a different type of knowledge: entity relationships from Wikidata, commonsense relationships from ConceptNet, and natural language question answers from SQuAD.
Template Converter — For each fact, a manually defined cloze sentence template (e.g., "Dante was born in [MASK]") is filled with the subject and has the object position masked. This converts a structured triple into a natural language query that the language model can answer by predicting the masked token. Each relation gets its own template pattern.
Language Models — The set of pretrained models being tested (fairseq-fconv, Transformer-XL, ELMo original, ELMo 5.5B, BERT-base, BERT-large). Each model processes cloze sentences using its native architecture (forward generation for autoregressive models, masked-token prediction for BERT) and produces a probability distribution over its vocabulary at the masked position.
Evaluation Metrics — The system maps each model's vocabulary to a unified vocabulary (the intersection of all models' vocabularies, ~21K tokens), ranks the ground-truth object against all other tokens in this vocabulary, and computes precision-at-k (P@k): the fraction of facts for which the correct answer appears in the top-k predictions.

Information flows as follows: a fact triple enters the system → the template converter produces a cloze sentence → the language model predicts a probability distribution at the masked position → the evaluation module restricts predictions to the common vocabulary, removes known alternative correct answers, ranks the ground-truth token, and computes P@k → results are aggregated per relation, per knowledge source, and per relation type (1-to-1, N-to-1, N-to-M).

3.3 Roadmap for the Deep Dive

First, the Knowledge Sources — what data is used, how it is structured, and what makes each source distinct. Understanding the data is essential because the probe's ability to measure knowledge depends critically on the quality and coverage of the fact databases.
Second, the Template Construction — how structured triples become cloze sentences, including the design decisions around manual templates, single-token answers, and why these constraints are deliberate choices rather than limitations.
Third, the Language Model Query Mechanisms — how each model architecture (unidirectional, ELMo's bidirectional LSTM, BERT's masked LM) generates predictions at a specified position, including the crucial difference between autoregressive and masked prediction.
Fourth, the Vocabulary Unification — why models are mapped to a common vocabulary of ~21K tokens, how the intersection is computed, and what this means for fair comparison.
Fifth, the Evaluation Protocol and Metrics — how precision-at-k is computed, how the "filtered" setting handles multiple valid answers, and what P@k actually measures operationally.
Sixth, the Baseline Systems — how the relation extraction baseline (RE with naïve and oracle entity linking), the frequency baseline, and DrQA are constructed, since understanding what language models are compared against is essential to interpreting the results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an analysis and measurement paper whose core idea is that we can probe the implicit world knowledge in pretrained language models by converting relational facts into fill-in-the-blank queries and measuring whether the model ranks the correct object token highly among all possible vocabulary items. The LAMA probe is not a proposed system for deployment but rather a diagnostic instrument—a standardized test that reveals what a model "knows" in a format that enables controlled comparison across architectures, training regimes, and knowledge types.

The LAMA Probe: What It Measures and How

The probe's fundamental operation is deceptively simple. Given a fact expressed as a triple (subject, relation, object)—for example, (Dante, born-in, Florence)—the system constructs a natural language sentence that expresses this relationship with the object position masked. For the "born-in" relation, the template might be "[S] was born in [O]", which becomes "Dante was born in [MASK]" when the subject is substituted. The language model is then asked to predict what token should fill the masked position. If the model assigns high probability to "Florence" (or ranks it highly among all vocabulary items), we infer that the model has stored this relational knowledge.

This simple operation masks enormous complexity in the design choices. Every component—the knowledge sources, the templates, the vocabulary mapping, the evaluation metric—embodies a trade-off between measurement fidelity, fairness across models, and practicality. Below, I walk through each component in the order that facts flow through the system.

Knowledge Sources: Where the Ground Truth Comes From

The LAMA probe uses four distinct knowledge sources, each testing a different kind of relational knowledge.

Google-RE (3 relations, ~5,527 facts). This corpus contains approximately 60,000 facts manually extracted from Wikipedia by human annotators, though the paper considers only three relations: "place of birth," "date of birth," and "place of death." Two relations are excluded because "they contain mainly multi-tokens objects that are not supported in our evaluation." Each fact in Google-RE is, by design, manually aligned to a specific short passage of Wikipedia text that supports it—meaning we know with certainty that the relevant fact is expressed in a sentence the model may have seen during training. This makes Google-RE the highest-quality, most precisely aligned knowledge source and serves as an upper bound on what a probe can measure.

T-REx (41 relations, ~34,039 facts). Derived from the T-REx dataset of Elsahar et al. (2018), this is a much larger subset of Wikidata triples with broader relational coverage. The paper considers 41 Wikidata relations and subsamples at most 1,000 facts per relation. Critically, unlike Google-RE, T-REx facts were automatically aligned to Wikipedia sentences (not manually verified), introducing potential noise. However, the authors cite Elsahar et al.'s reported "accuracy of 97.8% for the alignment technique over a test set," giving confidence that most alignments are correct.

The paper categorizes T-REx relations into three structural types, following the knowledge base completion literature (Bordes et al., 2013):

1-to-1 relations (2 relations, 937 facts): Each subject maps to exactly one object, and each object maps to exactly one subject. Example: "capital of" — a country has one capital city, and a city is the capital of one country. This is the simplest relational pattern because there is no ambiguity.
N-to-1 relations (23 relations, 20,006 facts): Many subjects can map to the same object, but each subject maps to only one object. Example: "place of birth" — many people can be born in the same city, but each person has only one birthplace. The subject side has cardinality N, the object side has cardinality 1.
N-to-M relations (16 relations, 13,096 facts): Both subjects and objects can appear in multiple facts. Example: "member of political party" — a person can belong to multiple parties over time, and a party has many members. Both sides have unbounded cardinality.

This categorization is essential because it predicts differential difficulty: 1-to-1 relations should be easiest (the model needs to associate one unique object with each subject), while N-to-M relations should be hardest (multiple objects are valid for a single subject, creating ambiguity and diluting the statistical signal during pretraining).

ConceptNet (16 relations, ~11,458 facts). ConceptNet is a multilingual commonsense knowledge base built on top of Open Mind Common Sense (OMCS) sentences—human-contributed statements about everyday relationships between concepts. The paper uses the English portion and considers only facts with single-token objects across 16 relations (e.g., "CapableOf," "HasProperty," "CausesDesire," "UsedFor"). The template construction for ConceptNet differs fundamentally from the Wikidata-based sources: instead of manually defined templates, the system finds the original OMCS sentence that contains both the subject and object, masks the object within that sentence, and uses the masked sentence as the query. If multiple sentences exist for a given triple, one is chosen at random.

Two important differences from Google-RE and T-REx: first, ConceptNet facts have no explicit alignment to Wikipedia sentences—the query sentences come from the OMCS corpus, not Wikipedia—meaning the language models may not have seen these specific sentences during training. Second, commonsense knowledge is qualitatively different from entity-centric factual knowledge: "birds can fly" or "ponds are used for fishing" involve prototypical properties and typical relationships rather than named-entity facts. This tests whether language models capture a different layer of knowledge beyond encyclopedic facts.

SQuAD (305 questions). The Stanford Question Answering Dataset is a popular reading comprehension benchmark where questions are answerable from a specific Wikipedia paragraph. The paper selects a subset of 305 "context-insensitive" questions from the SQuAD development set with single-token answers—questions that can be answered without reading the surrounding paragraph because they refer to general knowledge. For each question-answer pair, the authors manually create a cloze-style statement. For example, "Who developed the theory of relativity?" is rewritten as "The theory of relativity was developed by [MASK]." Since SQuAD was created from Wikipedia articles, each fact is known to be expressed somewhere in Wikipedia, providing the same alignment property as Google-RE: somewhere in the training data, the model could have encountered a sentence expressing this fact.

This source is distinct because it tests whether language models can answer naturalistic questions after they've been converted to a declarative form—bridging the gap between the cloze format used for structured triples and the question-answering format that downstream users actually want.

Template Construction: Converting Triples to Cloze Queries

The conversion of a structured fact into a natural language query is the most delicate design choice in the LAMA probe, and every aspect of it is deliberate.

Manual, relation-specific templates. For the Wikidata-based sources (Google-RE and T-REx), the authors manually define a single template for each relation. Table 3 in the paper shows examples: for T-REx relation P19 ("place of birth"), the template is "[S] was born in [O]"; for P20 ("place of death"), "[S] died in [O]"; for P279 ("subclass of"), "[S] is a subclass of [O]." The authors explicitly acknowledge the limitation: "One can expect that the choice of templates has an impact on the results, and this is indeed the case: for some relations we find both worse and better ways to query for the same information (with respect to a given model) by using an alternate template."

Why not use multiple templates and average? The authors make an analogy to traditional knowledge bases: a symbolic KB has only one way to query knowledge for a specific relation—namely, by using the relation ID. If you query with works-for and someone asks a different surface form like is-working-for, the KB returns nothing. The single-template design in LAMA mirrors this property: it tests whether the knowledge is accessible through a specific, predetermined access pattern, just as a structured query against a KB tests accessibility through a specific relation identifier. The authors argue this makes their evaluation a lower bound: if a model can recover the fact through this one specific template, it's strong evidence the knowledge is stored. If it can't, the knowledge might still be stored but expressed differently.

Single-token object constraint. The probe only evaluates predictions where the correct answer is a single token (e.g., "Florence," "Paris," "English"). This is not because the authors believe multi-token answers are irrelevant—they explicitly discuss the issue—but because multi-token generation introduces confounding variables: "beam size, candidate scoring weights, length normalization, n-gram repetition penalties, etc. that obscure the knowledge we are trying to measure." At the time of publication (2019), well-calibrated multi-token generation was an active research problem, particularly for bidirectional models like BERT which were not designed for autoregressive generation. The authors cite Welleck et al. (2019) on non-monotonic text generation as evidence of ongoing challenges in this area. By restricting to single-token answers, the probe isolates the knowledge retrieval capability from the separate problem of text generation quality.

Object-slot-only queries. The probe always queries for the object position in a triple, never the subject or relation. This is partly mitigated by including "reverse relations" in the dataset—for example, both contains and contained-by appear, so a subject query on one becomes an object query on the other. The authors give two reasons for not querying relation slots: first, "surface form realisations of relations will span several tokens," which violates the single-token constraint. Second, even if multi-token generation were solved, "relations can generally be expressed with many different wordings, making it unclear what the gold standard pattern for a relation should be, and how to measure accuracy in this context."

ConceptNet's sentence-based templates. The ConceptNet source uses a fundamentally different template strategy. Instead of manually defining a pattern, the system retrieves the original OMCS sentence that was used to create the ConceptNet triple, then masks the object token within that sentence. For example, for the triple (virus, Causes, infection), the OMCS sentence might be "Sometimes virus causes [MASK]." This has the advantage of using naturally occurring human language rather than researcher-defined templates, but it introduces variability: the same fact might be expressed in different sentences, and the randomly chosen one may not be the phrasing the language model finds most natural. The authors do not report systematic analysis of this variability for ConceptNet.

SQuAD's manual cloze conversion. For the 305 SQuAD questions, the authors manually rewrite each natural language question into a declarative cloze statement. This is a creative but subjective process—there is no algorithmic procedure for converting "Who developed X?" into "X was developed by [MASK]"—and the paper does not provide inter-annotator agreement or systematic guidelines for this conversion. The resulting templates are essentially researcher-designed, similar to the Wikidata templates, but informed by question-answering conventions rather than relation schemas.

Language Model Query Mechanisms: How Each Model Generates Predictions

A critical design requirement of the LAMA probe is that each language model must be queried in its "natural" way—using the generation mechanism that matches its training objective—to avoid artificially advantaging or disadvantaging any architecture. This section explains the prediction mechanism for each model type in the study.

Unidirectional Language Models (fairseq-fconv and Transformer-XL)

For standard autoregressive language models trained to predict $p(w_t \mid w_1, \dots, w_{t-1})$ , the query mechanism is straightforward. Given an input sequence where the object token is at position $t$ , the model naturally computes a probability distribution over the vocabulary at that position using the hidden state from the immediately preceding position.

Formally, the model computes:

$p(w_t = v \mid w_1, \dots, w_{t-1}) = \text{softmax}(W h_{t-1} + b)_v$

where $h_{t-1} \in \mathbb{R}^k$ is the output vector of the neural network at position $t-1$ (the position just before the token being predicted), $W \in \mathbb{R}^{|V| \times k}$ is a learned weight matrix that maps hidden states to vocabulary scores, $b$ is a bias vector, and $v$ indexes into the vocabulary $V$ .

What it computes: for each possible token in the vocabulary, the model produces a scalar score based on the hidden representation of the preceding context, then normalizes these scores into a probability distribution via softmax. The hidden state $h_{t-1}$ encodes the entire history $[w_1, \dots, w_{t-1}]$ through the model's architecture (gated convolutions for fairseq-fconv, self-attention with recurrence for Transformer-XL). The output is a vector of length $|V|$ where each entry is the model's estimated probability that token $v$ appears at position $t$ given the preceding context.

Why this form: this is the standard neural language model factorization from Bengio et al. (2003), where each word's probability is conditioned on all previous words. The linear map $W h_{t-1} + b$ followed by softmax ensures that outputs are non-negative and sum to one, making them valid probabilities. For cloze queries, the "context before the mask" fills in $w_1, \dots, w_{t-1}$ , and the model predicts what comes next—exactly matching its training objective.

Practical note: for a query like "Dante was born in [MASK]," the model must see the sentence up to the token just before the mask. The probe feeds the sequence "Dante was born in" to the model, obtains the output distribution at the final position, and treats this as the prediction for what should follow. The model never "sees" any tokens after the mask position, consistent with its left-to-right training.

ELMo's Bidirectional LSTM

ELMo uses a fundamentally different architecture and must be queried differently. ELMo is not a single language model but a combination of two independent LSTM language models: a forward LSTM that computes $p(w_t \mid w_1, \dots, w_{t-1})$ (reading left-to-right) and a backward LSTM that computes $p(w_t \mid w_{t+1}, \dots, w_N)$ (reading right-to-left). During pretraining, these two models are trained separately, and their log-likelihoods are combined.

To query ELMo for the token at position $t$ , the probe:

Runs the forward LSTM over the sequence $[w_1, \dots, w_{t-1}]$ and obtains the output hidden state $\overrightarrow{h_{t-1}}$ from the position just before $t$ .
Runs the backward LSTM over the reverse sequence $[w_N, \dots, w_{t+1}]$ and obtains the output hidden state $\overleftarrow{h_{t+1}}$ from the position just after $t$ .
Computes the forward probability distribution: $p_{\text{fwd}}(v) = \text{softmax}(W_{\text{fwd}} \overrightarrow{h_{t-1}} + b_{\text{fwd}})_v$ .
Computes the backward probability distribution: $p_{\text{bwd}}(v) = \text{softmax}(W_{\text{bwd}} \overleftarrow{h_{t+1}} + b_{\text{bwd}})_v$ .
Averages the two distributions: $p(v) = \frac{1}{2}(p_{\text{fwd}}(v) + p_{\text{bwd}}(v))$ .

What it computes: a single probability for each vocabulary item that averages the predictions from both directions. The forward component captures what word is likely given the left context (e.g., "Dante was born in __"); the backward component captures what word is likely given the right context (e.g., "Things written before 1300" or simply the end of the sentence). Averaging them gives a prediction that uses both sides of the mask.

Why this form: this averaging mirrors the ELMo pretraining objective, which maximizes the joint log-likelihood $\sum_{t} (\log p_{\text{fwd}}(w_t \mid w_{<t}) + \log p_{\text{bwd}}(w_t \mid w_{>t}))$ . The two models are independently parameterized; combining their output distributions via simple averaging rather than a learned mixture weight preserves the unsupervised nature of the probe (no additional parameters are trained). The authors follow "the loss definition in Peters et al. (2018a)" for this averaging procedure.

Key difference from BERT: ELMo runs two separate forward passes (one forward LSTM, one backward LSTM) and averages their independent probability estimates. BERT, in contrast, conditions on both directions simultaneously in a single forward pass through the Transformer. This means ELMo's "bidirectionality" is approximating true bidirectional context through model averaging, while BERT's is architecturally integrated. The LAMA probe's evaluation should be sensitive to this difference—BERT can condition on correlations between left and right context, while ELMo sees them independently.

BERT's Masked Language Model

BERT is queried fundamentally differently from both unidirectional models and ELMo because its training objective is not to predict words sequentially but to fill in randomly masked positions. The procedure for querying BERT on a cloze statement is:

The input sequence is constructed with an explicit [MASK] token at the position $t$ where the object should appear. For example: "Dante was born in [MASK] ."
The entire sequence (including the mask token) is fed through BERT's Transformer encoder in a single forward pass. Every token attends to every other token (both left and right of the mask) through the self-attention mechanism.
The output vector $h_t$ at the masked position is extracted from the final Transformer layer. This vector has dimensionality $k$ (768 for BERT-base, 1024 for BERT-large) and encodes information from the entire sequence context—both the words before the mask and the words after it.
The probability distribution over the vocabulary is computed as $\text{softmax}(W h_t + b)$ , where $W \in \mathbb{R}^{|V| \times k}$ are the output embedding weights (tied with the input embeddings) and $b$ is a bias.

What it computes: unlike unidirectional models that predict based only on preceding context, BERT's prediction at the [MASK] position is conditioned on the entire surrounding sequence simultaneously. The self-attention mechanism at each Transformer layer allows the vector $h_t$ to gather information from every other position, meaning the model sees both "Dante was born in" (left context) and "." or subsequent words (right context) when making its prediction. For the cloze templates used in LAMA, there is typically no substantial right context—the mask appears at the end of the template—so BERT's advantage from bidirectionality is limited. However, for sentence-internal masks (as in some ConceptNet sentences), BERT can use both sides.

Why this form: this exactly matches BERT's pretraining objective as described in Devlin et al. (2018b). During pretraining, 15% of tokens are randomly selected; 80% of those are replaced with [MASK], 10% with a random token, and 10% left unchanged. The model is trained to predict the original token at masked positions using the final hidden vector. By using the same [MASK] token insertion and prediction head, the LAMA probe queries BERT in precisely the way it was trained to be queried, maximizing the likelihood that stored knowledge will be elicited.

Practical implication for knowledge probing: BERT's bidirectional conditioning is theoretically advantageous for cloze tasks because it can use both preceding and following context. However, the LAMA templates are structured so that the mask is typically at or near the end of the template sentence, meaning the right context is minimal (often just a period). This reduces BERT's architectural advantage over unidirectional models in practice, making the comparison somewhat fairer than it might first appear.

The Unified Vocabulary: Enabling Fair Comparison Across Models

One of the most subtle but consequential design decisions in LAMA is the use of a common vocabulary across all models. This is necessary because the models in the study were trained with radically different tokenization schemes and vocabulary sizes, and the size of the vocabulary directly affects ranking difficulty.

Consider the problem: if Model A has a vocabulary of 30,000 tokens and Model B has a vocabulary of 800,000 tokens, and both correctly assign high probability to the ground-truth answer "Florence," Model B will naturally rank "Florence" at a better percentile simply because there are more competitor tokens to beat. Comparing raw ranks across different vocabulary sizes is meaningless—a rank of 50 out of 30K is fundamentally different from a rank of 50 out of 800K.

The paper's solution is to intersect all models' vocabularies:

"To allow a fair comparison, we let models generate over a unified vocabulary, which is the intersection of the vocabularies for all considered models (~21K case-sensitive tokens)."

The procedure works as follows:

For each model, extract its full case-sensitive vocabulary (the set of tokens it can produce in its output softmax layer). For ELMo, this is approximately 800,000 tokens; for BERT, approximately 30,000 tokens; for fairseq-fconv and Transformer-XL, their respective vocabularies.
Compute the intersection—the set of tokens that appear in every model's vocabulary. The paper reports this yields approximately 21,000 case-sensitive tokens.
At evaluation time, each model computes its full probability distribution over its native vocabulary. The probe then masks out (removes from consideration) all tokens that are not in the common vocabulary, renormalizing the probabilities over the remaining tokens. Only the ~21K common tokens are eligible to be ranked.

What this achieves: every model is evaluated on exactly the same set of possible answers. If "Florence" is in the common vocabulary (which it is, since it's a common word), all models must rank it against the same competitor set. If "Florence" is not in the common vocabulary—perhaps because one model uses an unusual tokenization that splits it into subwords—that fact cannot be tested across all models and is effectively excluded.

What this costs: the intersection vocabulary is substantially smaller than any individual model's vocabulary. ELMo loses over 97% of its output vocabulary, BERT loses about 30% of its output vocabulary. This means that some facts become untestable because the correct answer token isn't in the common set—the paper doesn't report what fraction of facts this affects, but it's a direct consequence of the fairness constraint. The smaller vocabulary also means that rankings are easier for all models than they would be with full vocabularies, since there are fewer competitor tokens to beat. This is acceptable because the probe's goal is relative comparison between models, not absolute measurement.

Why intersection rather than union or model-specific vocabularies: a union vocabulary (allowing each model to rank against its full set) would make the evaluation unfair, as argued above. Model-specific evaluation would prevent cross-model comparison entirely. The intersection is the only choice that guarantees each model faces exactly the same ranking difficulty.

A subtle consequence for tokenization: models that use subword tokenization (like BERT with WordPiece) may represent a multi-character word like "Florence" as a single token, while character-level or larger-vocabulary models might split it. The common vocabulary intersection ensures that only tokens which are indivisible units in every model are tested. This introduces a systematic bias against facts whose standard object tokenization differs across models—but since the bias applies equally to all facts (any fact whose object token isn't universal is simply excluded), it doesn't advantage any particular model in the comparison.

Evaluation Protocol and Metrics: Computing Precision-at-k

The core metric in LAMA is mean precision at k (P@k), a ranking-based measure adapted from the knowledge base completion literature (Bordes et al., 2013). Here is the step-by-step evaluation protocol for a single fact.

Step 1: Generate predictions. The language model processes the cloze template and produces a probability distribution over its native vocabulary at the masked position. This distribution is then filtered to the common vocabulary of ~21K tokens.

Step 2: Filter known alternative answers. For N-to-M relations (where multiple objects can be valid for a given subject-relation pair), the evaluation must account for the fact that there isn't a single correct answer. The paper follows the "filtered" setting from Bordes et al. (2013):

"To account for multiple valid objects for a subject-relation pair (i.e., for N-M relations), we follow Bordes et al. (2013) and remove from the candidates when ranking at test time all other valid objects in the training data other than the one we test."

Operationally: if we're testing the fact (Dante, influenced-by, Virgil) and the training data also contains (Dante, influenced-by, Ovid) and (Dante, influenced-by, Cicero) as valid facts, the evaluation removes "Ovid" and "Cicero" from the vocabulary before ranking. The model doesn't get penalized for predicting another correct answer—only for predicting incorrect ones. However, it's important to note what this filtering does NOT do: it does not remove alternative answers that are valid but not in the training data. If (Dante, influenced-by, Guido Cavalcanti) is also a true fact but not in the probe's fact database, the model still gets "penalized" for ranking Cavalcanti above Virgil, even though it's a correct answer.

Step 3: Rank the ground-truth token. The remaining vocabulary items are sorted by their assigned probability (or log probability, which is equivalent for ranking). The probe finds the position (rank) of the ground-truth object token in this sorted list. Rank 1 means the ground truth has the highest probability; rank 100 means 99 other tokens are considered more likely.

Step 4: Compute P@k for the fact. For a given fact $f$ with ground-truth object $o$ and a cutoff $k$ , the P@k value is:

$\text{P@k}(f) = \begin{cases} 1 & \text{if rank}(o) \leq k \\ 0 & \text{otherwise} \end{cases}$

where $\text{rank}(o)$ is the position of the ground-truth token in the sorted, filterted vocabulary.

What it computes: a binary indicator of whether the correct answer appears among the model's top-k predictions. P@1 is the strictest: did the model rank the correct answer first? P@10 is more lenient: was the correct answer anywhere in the top 10? P@100 is very lenient and is used primarily to assess whether the model has any signal at all—if the answer isn't in the top 100, the model effectively has no useful knowledge of that fact.

Why this form: the knowledge base completion field has long used ranking-based metrics because they capture the practical use case: you have a knowledge base with a missing object, you query the system, and you look at the top-k returned answers. If the correct answer is in that list, the system is useful; if not, it isn't. Accuracy (did the top-1 prediction match the ground truth?) is just P@1. Mean reciprocal rank (MRR) would weight ranks smoothly (1/rank rather than a hard cutoff), but the paper opts for P@k because it's more interpretable: "BERT gets the right answer in its top 10 predictions for 60% of T-REx facts" is a clear statement.

Step 5: Aggregate across facts. The mean P@k across all facts in a relation, knowledge source, or the entire probe is computed as:

$\text{mean P@k} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}[\text{rank}(o_i) \leq k]$

where $N$ is the number of facts being aggregated over, and $\mathbb{1}[\cdot]$ is the indicator function.

What it computes: the fraction of facts for which the correct answer appears in the top-k predictions, averaged over all facts under consideration. A mean P@1 of 32.3% on T-REx for BERT-large means that for approximately one-third of all Wikidata facts tested, BERT-large assigned the highest probability to exactly the correct object—despite never being trained on fact extraction and never seeing a knowledge base schema.

Baseline Systems: What Language Models Are Compared Against

To contextualize the language model results, the paper constructs several baselines that represent alternative ways of obtaining relational knowledge. These baselines are carefully designed to provide both lower bounds (what a trivial method achieves) and upper bounds (what dedicated systems with supervised training and oracle information can achieve).

Frequency Baseline (Freq)

This is the simplest possible baseline: for each relation, compute how frequently each object token appears across all facts of that relation in the test data, then always predict the most common object for any subject query.

"For a subject and relation pair, this baseline ranks words based on how frequently they appear as objects for the given relation in the test data."

What it computes: for the relation "place of birth," the frequency baseline looks at all birthplace facts in the test set, counts that "New York" appears 50 times, "London" appears 40 times, etc., and always predicts "New York" as the top answer regardless of the subject. This is a relation-level prior: it captures the marginal distribution of objects for each relation type without looking at the subject at all.

What it represents: this baseline shows what performance you can get by simply knowing that certain objects are common for a relation type, ignoring the subject entirely. For highly skewed relations (e.g., most people are born in a small set of common locations), this baseline can be surprisingly high. Anything below the frequency baseline is worse than knowing nothing about the specific subject.

Why it matters: if a language model's performance is close to the frequency baseline, it suggests the model might be exploiting the same marginal statistics (learning that "countries often have capitals that are well-known cities" rather than learning specific country-capital associations). The paper reports a frequency baseline of 22.03% mean P@1 on T-REx, meaning that roughly one-fifth of facts can be "answered" just by guessing the most common object for each relation. This is a high bar in absolute terms and makes the language model results more meaningful: BERT-large's 32.3% beats the frequency baseline by about 10 percentage points, indicating knowledge beyond simple marginal statistics.

Relation Extraction with Entity Linking (REn and REo)

This baseline represents the traditional approach to building a knowledge base from text: run a relation extraction system over relevant documents, construct a graph of extracted triples, and query it at test time. The paper uses the pretrained relation extraction model of Sorokin and Gurevych (2017), which was "trained on a subcorpus of Wikipedia annotated with Wikidata relations" and extracts triples using an LSTM-based encoder with an attention mechanism.

The critical design element is how subject entities are linked between the test query and the extracted triples:

REn (naïve entity linking): uses exact string matching to find the subject in the extracted graph. If we query for "Dante" and the RE system extracted triples with subject "Dante," they are matched. If the system extracted "Dante Alighieri" or "Durante degli Alighieri," the match fails.
REo (oracle entity linking): in addition to string matching, uses an oracle that knows the correct alignment between test subjects and extracted subjects. As the authors explain: "if RE has extracted any triple $(s', r, o')$ from that sentence $x$ , $s'$ will be linked to $s$ and $o'$ to $o$ . In practice, this means RE can return the correct solution $o$ if any relation instance of the right type was extracted from $x$ , regardless of whether it has a wrong subject or object."

The consequence of the oracle is dramatic: REo will be credited with "knowing" a fact if it extracted any correct relation from the sentence known to contain that fact, even if it misidentified the subject or object. This is an extremely generous evaluation—essentially, if the RE system can identify that a sentence of type "place of birth" exists and extract any triple of that type, the oracle maps it to the correct answer. This makes REo a very strong upper bound on what traditional extraction can achieve when entity linking is perfect.

The construction pipeline:

For each knowledge source with Wikipedia alignment (Google-RE and T-REx), identify the specific Wikipedia sentences that are known to express each test fact. This uses the manual alignment for Google-RE and the automatic alignment for T-REx.
Run the RE model on these sentences to extract triples. The RE model extracts (subject, relation, object) triples with confidence scores.
Build a knowledge graph from all extracted triples. At test time, for a query (s, r, ?), find the subject in the graph (using string matching for REn, oracle for REo), then rank all objects connected by relation $r$ based on the RE model's confidence scores.
Compute P@1 using the same ranking procedure as for language models.

What this baseline controls for: the RE baseline uses the exact same sentences that are known to express the facts, with a purpose-built extraction system trained on relation extraction data. It represents the best that traditional pipeline NLP can do on this data, given supervised training and (in the oracle case) perfect entity linking. If BERT outperforms REo, that's a strong signal that language models are competitive with—and in some cases better than—traditional extraction pipelines.

A crucial data limitation: the RE baseline only has access to the specific aligned Wikipedia sentences, not all of Wikipedia. The language models, in contrast, were trained on much larger corpora (e.g., BERT on 3.3B words from Wikipedia and BookCorpus). This means the RE baseline is operating in a lower-data regime. The authors explicitly address this: "We suspected BERT might have an advantage due to the larger amount of data it has processed, so we added Wikitext-103 as additional data to the relation extraction system and observed no significant change in performance." This suggests that simply giving the RE system more text doesn't help—relation extraction performance may saturate with relatively small amounts of target-domain data, while language models continue to absorb knowledge from larger corpora.

DrQA for Open-Domain Question Answering

For the SQuAD knowledge source, the baseline is DrQA (Chen et al., 2017), a supervised open-domain QA system. DrQA uses a two-stage pipeline:

Document retrieval: a TF-IDF information retrieval component searches Wikipedia for articles relevant to the question. The top $k$ articles are returned.
Reading comprehension: a neural reading comprehension model (trained on SQuAD) reads each retrieved article and extracts answer spans. The model is a multi-layer bidirectional LSTM that encodes both the question and the passage, then predicts start and end positions for the answer span within the passage.

To ensure fair comparison with the single-token language model predictions, "we constrain the predictions of DrQA to single-token answers." This means DrQA can only output answers that are a single token long, even if its reading comprehension model would prefer a multi-token span.

What this baseline represents: a state-of-the-art (for 2019) supervised QA system that has access to (a) the entire Wikipedia corpus at retrieval time, (b) a dedicated TF-IDF retrieval index, and (c) a neural reading comprehension model trained on over 100,000 SQuAD question-answer pairs. This is a very strong baseline—DrQA was one of the best-performing open-domain QA systems when the paper was written.

The comparison is asymmetric in important ways: DrQA has supervised training on the exact task (SQuAD question answering), access to a retrieval system, and the ability to read specific Wikipedia paragraphs. BERT-large, in contrast, has no fine-tuning, no retrieval mechanism, and cannot "read" documents—it must answer from whatever knowledge is stored in its parameters. The fact that BERT-large achieves P@10 of 57.1% compared to DrQA's 63.5% is therefore remarkable: an unsupervised model with no task-specific training is within 6.4 percentage points of a fully supervised system.

Summary of Design Choices and Their Justifications

Manual single templates rather than multiple paraphrases: mirrors the structured query interface of traditional knowledge bases, establishing a conservative lower bound on model knowledge. If the fact is recoverable through this one template, the knowledge is robustly stored.
Single-token objects only: avoids confounding the measurement of knowledge retrieval with the separate problem of multi-token generation quality, which was an unresolved research challenge in 2019, especially for bidirectional models.
Object-slot-only queries: avoids the ambiguity of multi-token relation surface forms and the unclear gold standard for what constitutes a correct relation prediction. Reverse relations partially compensate for the one-sidedness.
Common vocabulary intersection: ensures every model is evaluated against exactly the same set of possible answers, eliminating vocabulary size as a confounding variable. The cost is reduced coverage (facts whose object token isn't universal are excluded) and easier ranking (fewer competitors), but these affect all models equally.
Filtered evaluation for N-to-M relations: removes known alternative correct answers from the ranking, preventing models from being penalized for predicting other valid objects. However, unlisted valid answers still count as "errors," which may underestimate model knowledge.
Relation extraction baseline with oracle entity linking: provides an extremely generous upper bound on traditional pipeline performance, making the language model results more impressive when they approach or exceed this baseline.
DrQA baseline for SQuAD: provides a fully supervised comparison point that has access to a retrieval system and task-specific training, making the language models' unsupervised performance more meaningful.
Rank-based metrics (P@k) rather than accuracy: aligns with the knowledge base completion literature and captures the practical use case of inspecting top-k predictions. P@k at multiple values of $k$ reveals whether the model's knowledge is precise (high P@1) or distributed (low P@1 but high P@100).

4. Key Insights and Innovations

Innovation 1: The Knowledge Probe as a Distinct Diagnostic Category Separating "What Is Stored" from "What Can Be Fine-Tuned"

The paper's deepest conceptual move is creating a sharp methodological distinction between two questions the field had been conflating: "Does this model store X knowledge in its pretrained weights?" versus "Can this model be fine-tuned to perform task Y?" Before LAMA, the dominant paradigm for evaluating pretrained models was transfer learning performance—initialize with pretrained weights, fine-tune on a downstream task, measure accuracy. This conflates the model's stored knowledge with its capacity to adapt when given task-specific supervision. A model could score well on a relation extraction benchmark after fine-tuning not because it knew the facts beforehand, but because pretraining gave it good linguistic features that made supervised learning from limited labels easier.

LAMA breaks this conflation by imposing a strict zero-shot, no-fine-tuning discipline. The probe queries models in their pretrained state, using only their native generation mechanisms (masked token prediction for BERT, next-word prediction for autoregressive models, bidirectional averaging for ELMo), with no task-specific heads, no gradient updates, and no exposure to the probe's fact distribution. This creates a clean measurement of stored knowledge as distinct from knowledge-acquisition capacity. The distinction is not merely methodological hygiene—it has direct implications for deployment: if you want to answer factual questions today from a model you downloaded yesterday, fine-tuning is irrelevant; what matters is what's already encoded.

This framing reframes the entire "BERT as knowledge base" question. The KB comparison is not "BERT after fine-tuning on relation extraction vs. a traditional KB" but rather "BERT as-is vs. a KB as-is." Both are static stores: the KB has explicit triples, BERT has implicit parameter encodings. Both can be queried without further training. The probe makes this comparison meaningful by ensuring BERT is evaluated under conditions analogous to KB querying—a single access mechanism (the template), no learning at test time, and a fixed vocabulary. The manual template choice, which initially seems like an arbitrary limitation, becomes a deliberate parallel to the KB's single relation-ID query interface.

This diagnostic move is fundamental, not incremental. It created an entire subfield of "knowledge probing" that treats pretrained models as objects of study rather than as starting points for supervised learning. Subsequent work on factual knowledge in LMs, knowledge editing, and model cards for factual reliability all descend from this conceptual separation.

Innovation 2: The Relationship Between Relation Type and Knowledge Recoverability as a Diagnostic of Implicit Storage Mechanisms

The paper's breakdown of T-REx results by relation structural type—1-to-1, N-to-1, N-to-M—is not merely a granular evaluation. It is a diagnostic instrument that reveals something fundamental about how language models store relational knowledge implicitly through statistical co-occurrence rather than through explicit symbolic encoding.

The empirical pattern is stark (Table 2): BERT-large achieves 74.5% P@1 on 1-to-1 relations (e.g., "capital of") but only 24.3% on N-to-M relations (e.g., "member of political party"). This is not a random variance. It is a signature of the underlying storage mechanism. In a 1-to-1 relation, each subject appears almost exclusively with one object in the training corpus—"Paris" co-occurs with "capital of France" in predictable, consistent patterns. The language model's training objective (predict the masked word given context) directly incentivizes learning this deterministic mapping. The model doesn't need to understand what a capital is; it only needs to learn that when it sees "The capital of France is [MASK]," the answer is overwhelmingly likely to be "Paris."

In an N-to-M relation, the signal is fundamentally different. A politician might be a member of multiple parties over a career; a party has thousands of members. The co-occurrence statistics are diffuse: "Gordon Scholes" appears alongside "Labor" in some contexts, "Labour" in others (as the generation example in Table 3 shows: BERT predicts both "Labour" and "Labor" for the "member of political party" query). The model cannot learn a deterministic mapping because no deterministic mapping exists in the training distribution—the same subject appears with different objects depending on the sentence context (which the template, by design, strips away). The LAMA probe reveals this limitation not as a "failure" of BERT but as a direct consequence of how statistical language modeling encodes knowledge: through co-occurrence strength, not through structured relational reasoning.

This is a fundamental insight, not an incremental finding. Prior to LAMA, the question "do language models know facts?" was treated as a yes/no question. The relation-type breakdown transforms it into a how question: facts are stored through co-occurrence patterns, and the recoverability of any fact type depends on how strongly those patterns concentrate. This predicts—correctly, as subsequent work would confirm—that language models will excel at functional, deterministic relationships (capitals, birthplaces, official languages) and struggle with one-to-many, context-dependent, or temporally-scoped relationships (membership, employment, family relations). The probe doesn't just measure knowledge; it reveals the representational signature of the storage mechanism itself.

The negative result for N-to-M relations is thus as important as the positive result for 1-to-1 relations. It establishes a boundary condition: language models trained on next-word prediction can serve as knowledge bases only for knowledge types whose surface realization follows predictable, low-entropy co-occurrence patterns. This was not obvious before LAMA. The framing of "language models as knowledge bases" without this caveat would be misleading; with it, the paper provides a precise characterization of where the analogy holds and where it breaks down.

Innovation 3: The Oracle-Enhanced Relation Extraction Baseline as a Meaningful, Non-Trivial Comparison Point

The most counterintuitive design choice in LAMA is the construction of the REo (relation extraction with oracle entity linking) baseline. On its face, this looks like an unfair comparison—giving the traditional system access to ground-truth information that the language model doesn't have. But the choice is deeply strategic and reveals a sophisticated understanding of what the comparison is actually measuring.

The RE system is given two enormous advantages: (1) it operates on the exact sentences known to express each test fact (from the manual alignment in Google-RE and the automatic alignment in T-REx), eliminating the retrieval problem entirely, and (2) the oracle entity linker maps any extracted triple of the correct relation type to the test fact, regardless of whether the RE system correctly identified the subject or object. This means REo succeeds if the RE model can merely detect that some instance of the target relation is expressed in the aligned sentence—it doesn't need to get the arguments right.

Why make the baseline this generous? Because the comparison is not "can BERT extract relations better than an RE system?" (the answer to that is obvious: BERT wasn't trained to extract relations and has no extraction mechanism). The comparison is: given ideal extraction conditions (known relevant sentences, perfect entity linking), how much relational knowledge can a traditional pipeline recover from text versus how much a language model absorbed during pretraining? The RE baseline represents the upper bound of what symbolic extraction can achieve from the same information source (Wikipedia sentences). If BERT, which saw these same sentences during pretraining but was never instructed to extract facts from them, can match or exceed this upper bound, then the implicit absorption of knowledge through language modeling is competitive with explicit extraction—despite being completely unsupervised and requiring no pipeline engineering.

The empirical result—BERT-large achieves 32.3% P@1 on T-REx compared to REo's 33.8% (Table 2)—is therefore genuinely striking. BERT is within 1.5 percentage points of a system that was (a) trained on supervised relation extraction data, (b) given only the sentences known to contain the test facts, and (c) granted perfect entity linking. This is not "BERT beats a weak baseline." It is "BERT matches a system operating with oracle knowledge under idealized extraction conditions." The comparison is fundamental because it establishes the viability of the implicit knowledge storage paradigm: if the best explicit extraction can only achieve 33.8% on this data, and an unsupervised language model achieves 32.3%, then the language model approach is not just promising—it is already competitive with the dominant engineering paradigm.

The addition of more training data (WikiText-103) to the RE system producing no significant improvement reinforces this point. It suggests that explicit extraction hits a performance ceiling that implicit absorption may not share—because language models can continue to absorb weak statistical signals from vastly larger corpora, while extraction systems require high-precision patterns that saturate with limited training data. This is a forward-looking argument that the paper makes carefully in its conclusion: if this trend holds, future larger language models trained on more data may steadily pull ahead of traditional extraction, not just match it.

Innovation 4: The Query-Sensitivity Analysis Demonstrating That Memorization, Not Reasoning, Underlies Factual Recall

The experiment in Figure 4—where the authors take up to 100 random facts per relation, find 10 different Wikipedia sentences known to express each fact, mask the object in each sentence, and measure the distribution of ranks—is a glass-box diagnostic disguised as a robustness check. It reveals something uncomfortable and important about what "knowledge" means in these models.

The key finding is that BERT and ELMo 5.5B show the lowest variability in ranking the correct object across different surface realizations of the same fact, while also being the models most likely to have seen the specific query sentences during training (since they were trained on Wikipedia, unlike ELMo original or fairseq-fconv). The authors note this explicitly: "Note that BERT and ELMo 5.5B have been trained on a larger portion of Wikipedia than fairseq-fconv and Transformer-XL and may have seen more sentences containing the test queries during training."

This correlation between low query-sensitivity and likely training-set exposure is not coincidental. It suggests that what the probe is measuring is not an abstract, reasoning-based understanding of facts (where a model that truly "knows" that "Dante was born in Florence" would produce "Florence" regardless of how the question is phrased) but rather memorization of specific surface patterns from the training data. The models that perform best on LAMA are precisely those most likely to have encountered near-exact matches to the probe's query templates during training. Their robustness to paraphrasing—as measured by the low variance in Figure 4—may reflect not genuine understanding but exposure to a wider variety of surface realizations of the same facts in the training corpus, enabling pattern matching against a larger set of memorized templates.

This is a negative result with profound implications. It reframes the paper's title question—"Language Models as Knowledge Bases?"—from an empirical comparison to a conceptual challenge. A real knowledge base stores facts independently of how they're queried: SELECT object WHERE subject="Dante" AND relation="born-in" returns "Florence" regardless of whether you wrote the query in uppercase, with a different variable name, or with an extra whitespace. But the language model's "knowledge" appears to be entangled with the surface form of the query. Change "Dante was born in [MASK]" to "The birthplace of Dante is [MASK]" and the model might fail—not because it doesn't "know" the fact, but because it never saw that specific co-occurrence pattern during training.

This insight anticipates and explains many subsequent findings in the knowledge probing literature: that probes are sensitive to prompt phrasing (Jiang et al., 2020), that "knowledge" can appear to vanish or appear based on how questions are asked (Elazar et al., 2021), and that language models may be better understood as stores of surface-level co-occurrence statistics rather than structured relational knowledge. The paper doesn't make this argument explicitly—it presents Figure 4 as a robustness analysis—but the data points unmistakably in this direction. The diagnostic move of measuring not just accuracy but accuracy variance across paraphrases is what makes this innovation subtle but fundamental. It transforms LAMA from a benchmark into a tool for questioning what "knowledge" even means in the context of neural language models.

5. Experimental Analysis

Evaluation Methodology

Dataset. The LAMA probe uses four knowledge sources: Google-RE (~5,527 facts across 3 relations, manually extracted from Wikipedia with explicit sentence-level alignment), T-REx (~34,039 facts across 41 Wikidata relations, automatically aligned to Wikipedia sentences with 97.8% reported alignment accuracy), ConceptNet (~11,458 facts across 16 commonsense relations, derived from Open Mind Common Sense sentences), and SQuAD (305 context-insensitive questions from the development set with single-token answers, manually converted to cloze statements). For the Wikidata-based sources, facts are subject-relation-object triples; for SQuAD, they are question-answer pairs. The paper subsamples at most 1,000 facts per T-REx relation to keep evaluation tractable.
Base models. Six pretrained language models are evaluated: fairseq-fconv (324M parameters, gated convolutional architecture, trained on WikiText-103 with 103M words), Transformer-XL large (257M parameters, Transformer with recurrence, also trained on WikiText-103), ELMo original (93.6M parameters, bidirectional LSTM, trained on Google Billion Word with 800M words), ELMo 5.5B (same architecture as ELMo original but trained on English Wikipedia and WMT 2008–2012 news data totaling 5.5B words), BERT-base (110M parameters, Transformer, trained on English Wikipedia and BookCorpus totaling 3.3B words), and BERT-large (340M parameters, same architecture and training data as BERT-base). The specific models were chosen to span diverse architectures (convolutional, Transformer, LSTM), training objectives (autoregressive, bidirectional masked LM, forward-backward LM), and training data scales and compositions, enabling analysis of which factors drive knowledge storage.
Metrics. Mean precision at k (P@k) is the primary metric: for a fact with ground-truth object o, P@k = 1 if the model ranks o among the top-k predictions in the common vocabulary after filtering known alternative answers, and 0 otherwise. For a set of N facts, mean P@k = (1/N) Σᵢ 1[rank(oᵢ) ≤ k]. The paper reports P@1 for primary comparisons (Table 2) and P@k curves for varying k (Figure 2). For N-to-M relations, the evaluation follows the "filtered" setting from Bordes et al. (2013): all other valid objects from the training data are removed from the vocabulary before ranking, ensuring models are not penalized for predicting alternative correct answers, though unlisted valid answers still count as errors. All models are evaluated on a unified vocabulary of approximately 21K case-sensitive tokens—the intersection of all six models' native vocabularies—so that every model ranks against exactly the same set of possible answers, eliminating vocabulary size as a confounding variable.
Baselines. Three types are used: (1) Freq — a frequency baseline that, for each relation, ranks objects by how frequently they appear in the test data for that relation, ignoring the subject entirely. This captures the marginal object distribution per relation and represents what can be achieved without any subject-specific knowledge. (2) RE — a relation extraction baseline using the pretrained model of Sorokin and Gurevych (2017), an LSTM-based encoder with attention trained on Wikidata-annotated Wikipedia. The RE system processes the specific Wikipedia sentences known to align with each test fact (manually aligned for Google-RE, automatically aligned for T-REx), extracts triples with confidence scores, and builds a knowledge graph. Two variants exist: REn uses naïve entity linking based on exact string matching between test subjects and extracted subjects; REo uses an oracle entity linker that maps any extracted triple of the correct relation type from the aligned sentence to the test fact, regardless of whether the RE system correctly identified the subject or object. (3) DrQA — the open-domain QA system of Chen et al. (2017), which uses TF-IDF document retrieval over Wikipedia followed by a neural reading comprehension model trained on SQuAD to extract answers, constrained to single-token predictions. For the ConceptNet source, no RE baseline is applicable because ConceptNet facts lack Wikipedia sentence alignment.
Generation budget / compute accounting. There is no variable generation budget in this paper—the probe measures the factual knowledge already stored in model parameters, not the effect of additional inference-time computation. Each model produces exactly one prediction per fact (a single probability distribution over its vocabulary at the masked position). The "compute" in comparison is the pretraining compute (which varies across models based on architecture, parameter count, and training data size, as summarized in Table 1) rather than test-time generation budget. The RE baselines also involve no variable generation budget: they query a pre-built knowledge graph constructed from the aligned sentences. The frequency baseline involves no computation beyond counting.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. The test sets are fixed: the specific facts in each knowledge source (the 3 Google-RE relations, the 41 T-REx relations subsampled to 1,000 facts each, the 16 ConceptNet relations, the 305 SQuAD questions) are evaluated in their entirety, and mean P@k is reported across all facts in each source or relation category. The difficulty estimation via relation-type categorization (1-to-1, N-to-1, N-to-M) follows the standard knowledge base completion taxonomy and is computed from the test data itself (by analyzing cardinalities of subject-object mappings within the test facts). For the query-sensitivity analysis (Figure 4), up to 100 random facts per relation are selected, and 10 aligned Wikipedia sentences per fact are used as alternative queries, with the average rank distribution computed across these variations.

Main Quantitative Results

Google-RE: BERT Outperforms Oracle-Enhanced Relation Extraction on Small-Scale, Manually Verified Facts

Table 2 reports the Google-RE results (3 relations, 5,527 total facts). BERT-large achieves a mean P@1 of 10.5%, compared to 7.6% for REo (the oracle entity linking baseline), 1.2% for REn (naïve entity linking), and 4.4% for the frequency baseline. BERT-base achieves 9.8%, also above REo. Breaking down by relation: for "place of birth" (2,937 facts), BERT-large achieves 16.1% vs. REo's 13.8% vs. REn's 3.5% vs. Freq's 4.6%; for "place of death" (765 facts), BERT-large achieves 14.0% vs. REo's 7.2% vs. REn's 0.1% vs. Freq's 6.8%; for "date of birth" (1,825 facts), BERT-large achieves only 1.4% (vs. REo's 1.9% and Freq's 1.9%), with all models performing poorly on this relation.

The key headline is that BERT-large and BERT-base both outperform the oracle-enhanced RE baseline in aggregate, with BERT-large exceeding REo by 2.9 percentage points (10.5% vs. 7.6%). This is particularly striking given that: (a) the RE system was given only the specific sentences manually verified to express each test fact—reducing the retrieval problem to zero, (b) the oracle entity linker gives RE credit for any correctly-typed relation extracted from the aligned sentence regardless of argument accuracy, and (c) the RE model was trained on supervised relation extraction data. The advantage is concentrated in "place of birth" (+2.3 points) and "place of death" (+6.8 points), while "date of birth" performance is near-zero for all methods, suggesting this relation type is inherently difficult for both explicit extraction and implicit storage.

Among language models, the performance ordering is consistent: BERT-large > BERT-base (10.5% vs. 9.8%, a gap of 0.7 points) > ELMo 5.5B (3.0%) > fairseq-fconv (2.6%) > ELMo original (2.0%) > Transformer-XL (1.6%). The gap between the BERT models and all others is 3–4×, indicating that BERT's architecture and/or training data provide substantial advantages for factual knowledge storage. The frequency baseline (4.4%) outperforms all non-BERT language models, highlighting that these smaller or less-data-rich models have not absorbed factual knowledge beyond what marginal relation statistics provide.

T-REx: BERT Matches Oracle RE on a Broader Set of Relations, with Performance Dominated by Relation Type

Table 2 reports aggregate T-REx results (41 relations, 34,039 facts): BERT-large achieves 32.3% mean P@1, compared to REo's 33.8%, REn's 6.1%, and Freq's 22.03%. BERT-base achieves 31.1%. The gap between BERT-large and REo narrows to 1.5 percentage points (vs. 2.9 on Google-RE), but the aggregate masks enormous variation by relation structural type, which the paper reports separately:

1-to-1 relations (2 relations, 937 facts): BERT-large achieves 74.5%, BERT-base 68.0%, Transformer-XL 36.5%, ELMo 5.5B 13.1%, ELMo original 10.1%, fairseq-fconv 17.0%, REo 10.0%, REn 0.6%, Freq 1.78%. BERT-large outperforms the oracle RE baseline by 64.5 percentage points—a dramatic margin. This is the strongest result in the entire paper and the clearest evidence that language models can serve as knowledge bases for deterministic, functional relationships. The fact that Transformer-XL (36.5%) substantially outperforms the RE baseline (10.0%) on these relations suggests that even unidirectional language models with sufficient training can capture 1-to-1 mappings, though BERT's bidirectional conditioning provides a decisive advantage.

N-to-1 relations (23 relations, 20,006 facts): BERT-large achieves 34.2%, BERT-base 32.4%, Transformer-XL 18.0%, REo 33.8%, Freq 23.85%. Here, BERT-large and REo are nearly tied (34.2% vs. 33.8%), with BERT-large holding a marginal 0.4-point advantage. The frequency baseline (23.85%) is considerably higher than for 1-to-1 relations because N-to-1 relations (like "place of birth") have concentrated object distributions—many subjects share common objects, making the marginal prior more informative.

N-to-M relations (16 relations, 13,096 facts): BERT-large achieves 24.3%, BERT-base 24.7%, Transformer-XL 16.5%, REo 36.7%, Freq 21.95%. This is the only relation category where REo substantially outperforms BERT—by 12.4 percentage points (36.7% vs. 24.3%). The frequency baseline is 21.95%, meaning REo extracts 14.75 points of additional signal beyond the marginal distribution, while BERT extracts only 2.35 points beyond frequency. N-to-M relations are the Achilles' heel of language model knowledge storage: the diffuse co-occurrence patterns in the training data provide insufficient statistical signal for implicit memorization, while explicit extraction can identify individual pairings from sentence-level evidence.

Among non-BERT models on T-REx, Transformer-XL is the strongest (18.3%), followed by fairseq-fconv (8.9%), ELMo 5.5B (7.1%), ELMo original (4.7%). The ordering differs from Google-RE (where ELMo 5.5B outperformed fairseq-fconv), possibly reflecting that T-REx's broader relation coverage favors different architectural strengths. All non-BERT models fall substantially below the frequency baseline (22.03%), indicating that for the smaller or differently-trained models, the factual knowledge measured by the probe is largely captured by simple marginal object distributions.

Rank distribution analysis (Figure 2) shows P@k curves for T-REx across varying k. For BERT-large, P@1 is approximately 32%, P@10 is approximately 60%, and P@100 is approximately 80%. This means that for 80% of T-REx facts, the correct answer appears somewhere in the model's top 100 predictions—the model has some signal for the vast majority of facts, even when it doesn't rank the answer first. The curves for other models are substantially lower at all k values, with Transformer-XL reaching approximately 35% at P@10 and 55% at P@100. The steepness of BERT's curve between P@1 and P@10 (nearly doubling) suggests that much of the model's knowledge is "close but not precise"—the correct answer is often highly ranked but not first.

Correlation analysis (Figure 3) examines what factors predict BERT-large's P@1 on T-REx. The number of times a subject is mentioned in the training data shows a weak negative correlation with P@1 (−0.0048), while object mention frequency is also weakly negative (−0.075). The log probability of the model's first prediction is strongly positively correlated with P@1 (0.52), indicating that when BERT is confident in its top prediction, it tends to be correct—a useful calibration signal. Cosine similarity between subject and object vectors (computed via spaCy) shows moderate positive correlation (0.20), suggesting that semantically related subject-object pairs are easier to recall. The number of subject tokens (standard tokenization: −0.051; WordPiece tokenization: −0.18) shows weak negative correlation, hinting that multi-token subjects may dilute the statistical signal.

ConceptNet: BERT Recalls Commonsense Knowledge at Similar Levels to Factual Knowledge

Table 2 reports ConceptNet results (16 relations, 11,458 facts): BERT-large achieves 19.2% mean P@1, BERT-base 15.6%, with all other models substantially lower: ELMo 5.5B (6.2%), ELMo original (6.1%), Transformer-XL (5.7%), fairseq-fconv (3.6%). The frequency baseline is 4.8%. No RE baseline is applicable because ConceptNet facts lack Wikipedia alignment (the OMCS sentences that express the facts are from a different corpus than the language models were trained on, except potentially for Wikipedia-trained models that may have seen similar sentences).

The absolute P@1 of 19.2% for BERT-large is lower than its aggregate T-REx performance (32.3%), but direct comparison is misleading because ConceptNet relations are qualitatively different—many are prototypical rather than deterministic ("birds can fly" is typical but not universally true, unlike "Paris is the capital of France"). The fact that BERT-large substantially exceeds the frequency baseline (19.2% vs. 4.8%, a 14.4-point advantage) indicates it has absorbed commonsense associations beyond simple marginal statistics. BERT-large also outperforms the next-best model (BERT-base at 15.6%) by 3.6 points and the best non-BERT model (ELMo 5.5B at 6.2%) by 13.0 points, a larger relative gap than for T-REx, suggesting that BERT's architecture or training is particularly effective for commonsense knowledge.

Table 3 provides illustrative generation examples: for the query "You are likely to find a overflow in a [MASK]," BERT-large's top-5 predictions are "sewer, canal, toilet, stream, drain"—all syntactically and semantically plausible completions, with the ground truth ("drain") appearing at rank 5. For "Ravens can [MASK]," predictions are "fly, fight, kill, die, hunt" (ground truth "fly" at rank 1). For "Joke would make you want to [MASK]," predictions are "cry, die, laugh, vomit, scream" (ground truth "laugh" at rank 3). These examples reveal that even when BERT-large doesn't rank the ground truth first, its predictions remain reasonable—"cry" and "laugh" are both plausible responses to a joke, suggesting the model has learned the semantic space of commonsense associations even if it doesn't perfectly align with the specific ConceptNet fact.

SQuAD: BERT-Large Narrows the Gap to Supervised Open-Domain QA

Table 2 reports SQuAD results (305 questions): BERT-large achieves 17.4% P@1, compared to DrQA's 37.5%, with all other models substantially lower: BERT-base (14.1%), ELMo 5.5B (4.3%), Transformer-XL (3.9%), fairseq-fconv (3.6%), ELMo original (1.6%). The paper also reports P@10: BERT-large achieves 57.1% compared to DrQA's 63.5%—a gap of only 6.4 percentage points.

The P@1 gap of 20.1 points (37.5% vs. 17.4%) is substantial, but the comparison is highly asymmetric: DrQA is a fully supervised system trained on over 100,000 SQuAD question-answer pairs, with access to a TF-IDF retrieval system over the entire Wikipedia corpus and a specialized neural reading comprehension model. BERT-large, in contrast, has no fine-tuning on SQuAD, no retrieval mechanism, and cannot "read" specific documents—it answers from whatever knowledge is stored in its parameters. That BERT-large reaches nearly half of DrQA's P@1 (46.4% of the supervised performance) and 90% of DrQA's P@10 (57.1% vs. 63.5%) is the paper's most compelling evidence for language models as unsupervised open-domain QA systems: the model doesn't need to be told where to look or what to extract; the knowledge is already accessible through the right query format.

The P@10 result is particularly important because it shows that for nearly 57% of the tested questions, the correct answer is somewhere in BERT-large's top 10 predictions—a setting where a lightweight reranker or verification step could potentially recover the correct answer from the candidate list. The contrast between P@1 (17.4%) and P@10 (57.1%)—a 39.7-point jump—indicates that BERT's knowledge is distributed across multiple plausible answers rather than sharply concentrated on the correct one, consistent with the T-REx P@k curve in Figure 2. DrQA's P@1 to P@10 jump is smaller (37.5% to 63.5%, a 26-point jump), suggesting the supervised system's knowledge is more precisely calibrated.

Query Sensitivity Analysis: Factual Recall Depends on Template Phrasing, with BERT Showing Highest Robustness

Figure 4 presents the query sensitivity experiment: for up to 100 random facts per T-REx relation, the authors select 10 different Wikipedia sentences known to express each fact (from the T-REx alignment), mask the object in each sentence, and measure the rank assigned to the correct object by each model. The figure displays the average rank distribution (box plots with median and interquartile range) across all facts for each model.

The key finding: BERT-large, BERT-base, and ELMo 5.5B show the lowest rank variability (tightest interquartile ranges, medians close to low ranks), while fairseq-fconv and Transformer-XL show substantially higher variability and generally worse ranks. The median rank for BERT-large is approximately 10–20 across the distribution, compared to approximately 50–100 for Transformer-XL and 100–200 for fairseq-fconv. ELMo 5.5B's median is approximately 30–50, between the BERT models and the others.

However, the paper explicitly notes a critical confound: "Note that BERT and ELMo 5.5B have been trained on a larger portion of Wikipedia than fairseq-fconv and Transformer-XL and may have seen more sentences containing the test queries during training." The models that show the lowest sensitivity to query phrasing are precisely those most likely to have encountered near-exact matches to the probe's query sentences during pretraining. This means the apparent "robustness" may reflect memorization of multiple surface forms rather than genuine abstraction of the relational fact. The experiment cannot disentangle these two explanations, but the confound itself is an important finding: it suggests that what LAMA measures as "knowledge" is at least partially—and possibly largely—surface-form memorization rather than abstract relational understanding.

The practical implication is that LAMA's P@k numbers should be interpreted as measuring knowledge accessibility through specific templates, not knowledge possession in a template-independent sense. A model that fails on the probe's manually defined template might still "know" the fact if queried differently. The single-template design, which the authors argue establishes a lower bound, interacts with this sensitivity issue: the lower bound is on template-specific recall, not on knowledge storage, and the gap between the two is unknown.

Ablation Studies and Robustness Checks

Vocabulary intersection size effect: The paper uses a common vocabulary of approximately 21,000 case-sensitive tokens as the intersection of all six models' vocabularies. Larger native vocabularies—particularly ELMo's approximately 800,000 tokens—are dramatically pruned, reducing the set of possible predictions and making the ranking task easier for these models relative to evaluation on their full vocabularies. The paper does not report an ablation comparing performance on the common vocabulary vs. native vocabularies, which would quantify how much the intersection benefits large-vocabulary models. This is a significant unexamined confound: ELMo's absolute P@1 numbers may be inflated relative to BERT's because ELMo loses a much larger fraction of its vocabulary (and thus faces fewer competitor tokens in ranking). Since all models face the same 21K-vocabulary ranking task, relative comparisons are valid for the restricted vocabulary, but absolute P@1 values are not comparable to what the same model would achieve with its full vocabulary.

Template choice sensitivity (within-relation): The paper acknowledges that "for some relations we find both worse and better ways to query for the same information (with respect to a given model) by using an alternate template" but does not provide systematic quantification of this effect. No ablation table shows how P@1 varies across alternative templates for the same relations, and the paper does not report how template selection was validated (e.g., whether templates were iteratively refined based on pilot results, which would constitute test-set leakage). The query sensitivity experiment (Figure 4) partially addresses this by using Wikipedia sentences as alternative templates, but this is a different setup—natural sentences rather than manually designed alternatives—and only covers facts with multiple alignments. The lack of a multi-template ablation means the reported P@1 numbers are conditional on the specific templates chosen by the authors, and the reader cannot assess whether slightly different template choices (e.g., "Dante's birthplace is [MASK]" vs. "Dante was born in [MASK]") would yield materially different results.

Multi-token object exclusion: The probe excludes all facts with multi-token objects—a design choice that the paper justifies by citing unresolved challenges in multi-token generation calibration. The paper does not report what fraction of facts from each knowledge source are excluded by this constraint. For Google-RE, two of the original five relations are excluded entirely. For T-REx and ConceptNet, many facts likely have multi-token objects (e.g., "United States," "New York City," dates in multi-token formats), and the subset retained may be systematically different from the excluded facts—for example, single-token objects may be more common, more prototypical, or more frequently mentioned than multi-token alternatives. If so, the probe's results reflect performance on an easier-than-representative subset of facts. An ablation comparing characteristics of included vs. excluded facts would clarify this, but none is reported.

Effect of additional data on RE baseline: The paper reports that "we added Wikitext-103 as additional data to the relation extraction system and observed no significant change in performance." This is presented as evidence that the RE system's lower data volume (only aligned sentences vs. full Wikipedia) does not explain its underperformance relative to BERT. Specific numbers are not provided, making it impossible to assess whether "no significant change" means zero change or a small change that doesn't alter the overall comparison. The ablation is critical because it addresses the most obvious confound in the BERT vs. RE comparison (BERT saw more text), but the lack of quantitative detail limits its persuasiveness.

Filtering protocol for N-to-M relations: The evaluation removes from ranking all valid alternative objects present in the training data, following Bordes et al. (2013). The paper does not report how many alternative answers are typically removed per fact, what fraction of facts have alternative answers, or how sensitive the reported P@1 is to this filtering. If the filtering is aggressive (many alternatives removed), the evaluation is easier; if conservative, harder. The paper also does not report P@1 without filtering (the "raw" setting), which would provide a lower bound for models that might rank valid alternative answers above the specific test object. In the KB completion literature, both filtered and unfiltered metrics are standard; reporting only filtered results presents an incomplete picture of model behavior.

Model size scaling within BERT family: The paper includes both BERT-base (110M parameters) and BERT-large (340M parameters), enabling a within-family scaling comparison. The improvement from base to large varies by knowledge source: +0.7 points on Google-RE (9.8% → 10.5%), +1.2 points on T-REx (31.1% → 32.3%), +3.6 points on ConceptNet (15.6% → 19.2%), +3.3 points on SQuAD (14.1% → 17.4%). The absolute gains are modest, but the relative improvement is largest on ConceptNet (23% relative gain) and SQuAD (23% relative gain), suggesting that commonsense and QA knowledge may benefit more from increased capacity than entity-centric factual knowledge. However, with only two model sizes, the paper cannot establish a scaling trend; this is more of a point observation than an ablation.

Training data composition and size effects: The paper's model selection enables several implicit ablations comparing training data while holding architecture roughly constant, though these are not framed as formal ablations. ELMo original (800M words, Google Billion Word) vs. ELMo 5.5B (5.5B words, Wikipedia + WMT news) shows a 1.0-point gain on Google-RE (2.0% → 3.0%), a 2.4-point gain on T-REx (4.7% → 7.1%), and essentially no change on ConceptNet (6.1% → 6.2%), suggesting that additional Wikipedia-scale data improves factual knowledge but not commonsense. Transformer-XL and fairseq-fconv both trained on WikiText-103 (103M words) but with different architectures (Transformer vs. convolutional), with Transformer-XL substantially outperforming fairseq-fconv on T-REx (18.3% vs. 8.9%)—a 9.4-point gap that likely reflects architectural advantages for knowledge storage beyond what training data alone explains. However, the uncontrolled variation in architecture, training objective, and data composition across models makes it impossible to isolate any single factor's contribution.

Sentence-level memorization vs. fact-level knowledge: The query sensitivity experiment in Figure 4 shows that models trained on Wikipedia (BERT, ELMo 5.5B) exhibit lower rank variance across query formulations, but the paper does not conduct the definitive ablation needed to separate memorization from generalization: testing whether model performance drops when the template is systematically paraphrased in ways unlikely to appear in Wikipedia. For example, testing "The Italian poet Dante first saw the light of day in the city of [MASK]" vs. the standard "Dante was born in [MASK]" would reveal whether the model has abstract relational knowledge or surface-pattern associations. No such systematic paraphrase experiment is reported.

Negative result: near-zero performance on dates. The "date of birth" relation in Google-RE (1,825 facts) yields P@1 of 1.4% for BERT-large, 1.5% for BERT-base, 0.1% for ELMo 5.5B, 0.1% for ELMo original, 0.3% for fairseq-fconv, and 1.1% for Transformer-XL. The best-performing system is the oracle RE baseline at 1.9%, and the frequency baseline at 1.9%. All methods—including oracle-enhanced extraction—perform near the frequency prior, indicating that date-of-birth knowledge is fundamentally difficult to recover from Wikipedia text, likely because dates appear in many contexts unrelated to birth and the surface pattern "was born on [date]" competes with countless other date expressions. This negative result is important because it identifies a boundary condition: knowledge expressed through high-frequency, low-specificity surface patterns is poorly captured by both implicit LMs and explicit extraction.

Critical Assessment

Claim 1: "BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge"

The paper's central empirical claim is that BERT-large's relational knowledge, measured without fine-tuning, is competitive with a relation extraction baseline that has access to oracle entity linking and the specific sentences known to express each fact. The evidence in Table 2 supports this claim with important qualifications.

On Google-RE, BERT-large (10.5% P@1) outperforms REo (7.6%) by 2.9 points outright. On T-REx, BERT-large (32.3%) achieves 95.6% of REo's performance (33.8%), a gap of 1.5 points. The claim of "competitive" is well-supported for these aggregate numbers. However, the aggregate masks the relation-type dependency that is the paper's own most important finding: for 1-to-1 relations, BERT is vastly better than REo (74.5% vs. 10.0%, a 64.5-point gap); for N-to-1 relations, essentially tied (34.2% vs. 33.8%); for N-to-M relations, substantially worse (24.3% vs. 36.7%, a 12.4-point gap). The aggregate "competitive" result is an average over qualitatively different regimes, and the paper's framing could be read as overclaiming parity when the underlying picture is one of complementary strengths.

Additionally, the RE baseline has a specific and important limitation that makes the comparison less straightforward than it appears: the RE system operates on a restricted set of sentences (only those aligned to test facts), while BERT was trained on the entire Wikipedia corpus. The paper's addition of WikiText-103 to RE is reported as producing "no significant change," but the lack of quantitative detail makes this difficult to assess. If the RE system were given the full Wikipedia corpus (rather than just aligned sentences), would its recall improve? The paper cannot answer this because it only tested adding WikiText-103, not the full corpus, and did not report specific numbers. The comparison is therefore between a high-precision, low-recall extraction system (RE on aligned sentences) and a statistical absorption system (BERT on all of Wikipedia)—they are measuring different things, and the numerical comparison conflates precision differences with coverage differences.

Claim 2: "BERT also does remarkably well on open-domain question answering against a supervised baseline"

The SQuAD results support this claim when interpreted through the right metric. The P@1 gap is large: DrQA 37.5% vs. BERT-large 17.4%, meaning DrQA is more than twice as likely to produce the correct answer as its top-1 prediction. This is not "remarkably well" by an absolute standard. However, the P@10 comparison—DrQA 63.5% vs. BERT-large 57.1%, a 6.4-point gap—does support the claim. BERT-large, with no fine-tuning, no retrieval system, and no task-specific architecture, places the correct answer in its top-10 predictions for 90% as many questions as a fully supervised system. This is genuinely remarkable.

The asymmetry of the comparison is both a strength and a weakness. It's a strength because it makes BERT's achievement more impressive: the model is doing much more with much less. It's a weakness because the comparison doesn't control for the things that make DrQA strong—its retrieval system access, its supervised training on 100K+ examples, its specialized reading comprehension architecture. The question "can an unsupervised LM compete with a supervised QA system?" is interesting, but the paper doesn't answer the arguably more important question: "can an unsupervised LM replace a supervised QA system?" For that, we need P@1, and the 17.4% vs. 37.5% gap says no—at least for this model on this data.

A further limitation is the small SQuAD subset (305 questions, manually selected as "context-insensitive" and with single-token answers). This is a tiny, hand-filtered evaluation set—0.3% of the full SQuAD development set—and it's unclear whether the 305 questions are representative of open-domain QA difficulty or comprise the easiest, most factoid-like subset. The manual conversion of questions to cloze statements introduces experimenter degrees of freedom: different researchers might produce different templates, and there is no validation that the templates are optimal or unbiased. An ablation using multiple template sets or independent template creation would strengthen confidence in these results.

Claim 3: "Certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches"

This claim is the paper's most robust finding and is strongly supported by the relation-type breakdown in Table 2. The gap between 1-to-1 (74.5%) and N-to-M (24.3%) is 50.2 percentage points for BERT-large—a factor of 3× difference within the same model on the same knowledge source. The pattern is consistent across models: all models perform best on 1-to-1 and worst on N-to-M, with the ordering preserved. The paper's explanation—that 1-to-1 relations have concentrated co-occurrence statistics that language model training naturally exploits, while N-to-M relations have diffuse statistics—is plausible and well-motivated by the language modeling objective, though the paper does not provide direct evidence for this mechanism (e.g., by correlating P@1 with co-occurrence concentration metrics).

However, the claim would be stronger with finer-grained analysis. The paper categorizes relations into three structural types based on cardinality, but within each type there is likely substantial variation. Are all 1-to-1 relations equally well-recalled, or is BERT's 74.5% driven by a few extremely common relations (like "capital of") while other 1-to-1 relations perform poorly? Table 3 shows per-relation examples with widely varying log-probabilities (from −0.2 for "French" as original language to −7.5 for "Hamburg" as the namesake of Hamburg Airport), suggesting considerable within-type variation that the aggregate numbers conceal. A per-relation breakdown (even for a subset) would reveal whether the 1-to-1 advantage is robust across all such relations or concentrated in a few stereotyped patterns.

Claim 4: The LAMA probe measures "relational knowledge already present"

The deepest unexamined question is what LAMA actually measures. The probe design—single templates, single-token objects, common vocabulary intersection—is methodologically careful for comparing models, but it may be measuring something closer to "template-pattern recall" than "relational knowledge." The query sensitivity experiment (Figure 4) hints at this: models trained on Wikipedia show lower variance, suggesting they may have memorized the specific templates or close variants from training. The probe cannot distinguish between a model that has abstractly encoded "Dante's birthplace is Florence" and can express it in any reasonable phrasing, and a model that has memorized the specific co-occurrence pattern "Dante was born in Florence" from seeing it (or similar patterns) in Wikipedia.

This is not a failure of the paper's experimental design—it's a fundamental limitation of probing with single templates. But it matters for interpreting the paper's title question. If what LAMA measures is template-pattern memory, then "language models as knowledge bases" means something very different from traditional KGs: it means "language models as pattern-matching engines that can recall facts when queried in the specific ways those facts appeared in their training data." This is a much weaker claim than "language models as structured knowledge stores," and the paper's results are fully consistent with both the stronger and weaker interpretations. The paper acknowledges the template sensitivity issue ("One can expect that the choice of templates has an impact on the results") but does not fully grapple with its implications for the central analogy.

Missing Experiments That Would Strengthen the Paper

Several experiments could have substantially strengthened the paper's claims but were not run:

1. Systematic paraphrase experiment. Instead of (or in addition to) the Wikipedia-sentence query sensitivity analysis (Figure 4), a controlled experiment testing a fixed set of facts through 5–10 systematically varied templates would quantify how much "knowledge" depends on surface form. Templates could vary syntactic structure (active vs. passive, pre-vs. post-nominal modification), lexical choice (synonyms, formality level), and information ordering (subject-first vs. object-first). If BERT-large's P@1 drops from 32% to 15% when templates are paraphrased, the "knowledge base" interpretation is severely weakened.

2. Direct comparison to a knowledge base of comparable coverage. The paper compares to RE baselines that extract from text, but never to a curated knowledge base like Wikidata itself. If Wikidata's coverage of the T-REx test facts is high (which it presumably is, since T-REx is derived from Wikidata), then a simple KB lookup would achieve near-100% P@1 on all relation types. Comparing BERT's 32.3% to Wikidata's ~100% would provide a more honest baseline for the "language models as knowledge bases" question: yes, LMs store some knowledge, but they are far from replacing structured KGs for precision and coverage.

3. Per-relation breakdown for all T-REx relations. The paper reports only the aggregated 1-to-1, N-to-1, and N-to-M numbers, plus a few randomly selected examples in Table 3. A full per-relation table would reveal whether certain relations drive the aggregate numbers and whether BERT's advantage over RE is consistent or concentrated. If BERT achieves 74.5% on "capital of" but near-zero on other 1-to-1 relations, the "1-to-1 advantage" is really a "capital-of advantage," and the structural explanation (cardinality) is confounded with relation content (capitals are uniquely stereotyped in text).

4. Training data overlap analysis. The paper notes that BERT and ELMo 5.5B were trained on Wikipedia and therefore likely saw sentences expressing test facts, but provides no quantification of this overlap. A simple analysis—searching the training corpus for exact or near-exact matches to the test templates—would reveal what fraction of test facts the model could have memorized verbatim. If most high-P@1 facts correspond to training-set templates, the "knowledge" is memorization; if many are novel combinations, it's generalization. This analysis is feasible (the test templates are known, and the training corpus is public) and would dramatically sharpen the paper's conclusions.

5. Confidence calibration analysis. Figure 3 shows that the log probability of the first prediction is strongly correlated with P@1 (0.52), but this correlation is not explored in depth. A precision-recall curve or expected calibration error analysis would show whether the model "knows what it knows"—whether high-confidence predictions are reliably correct and low-confidence predictions are reliably incorrect. This matters for the practical use of LMs as knowledge bases, because any deployed system needs confidence estimates to decide when to trust the model vs. fall back to another source.

6. Effect of masking strategy for unidirectional models. For fairseq-fconv and Transformer-XL, the probe queries the model by feeding the sequence up to (but not including) the masked position and using the output distribution at the final position. This is the natural autoregressive generation mechanism, but it means unidirectional models never see any right context that might be present in the template. For templates with trailing material (e.g., "Dante was born in [MASK] in the year 1265"), unidirectional models cannot use "in the year 1265" as a disambiguating cue, while BERT can. The paper does not control for this asymmetry, which could artificially advantage BERT on templates with informative right context (though the authors' templates appear designed to minimize this). An ablation using only prefix-only templates for all models would isolate the effect of bidirectionality.

The paper's core contribution—that off-the-shelf pretrained language models store substantial relational knowledge accessible through cloze queries, with dramatic variation across relation types—is robustly supported by the reported experiments. The empirical patterns are clear, replicated across knowledge sources, and consistent with the proposed mechanism of statistical co-occurrence absorption. The paper's limitations are primarily in the interpretation of what these patterns mean: whether they represent "knowledge" in the sense that a knowledge base stores knowledge, or something more surface-bound and template-dependent. The probe itself cannot answer this question; subsequent work would need systematic paraphrase experiments, training-data overlap analysis, and direct comparison to curated knowledge bases to disambiguate these interpretations.

6. Limitations and Trade-offs

1. Template Dependence Means the Probe Measures Knowledge Accessibility, Not Knowledge Possession

The assumption or constraint. The LAMA probe queries each fact through a single, manually defined cloze template per relation (e.g., "[S] was born in [O]" for place of birth). The paper explicitly acknowledges that this choice affects results:

"One can expect that the choice of templates has an impact on the results, and this is indeed the case: for some relations we find both worse and better ways to query for the same information (with respect to a given model) by using an alternate template." (Section 4.5)

The authors justify this as establishing a lower bound, analogizing to traditional knowledge bases that have a single query interface, but this analogy is incomplete. A traditional KB stores facts independently of how they are queried: SELECT object WHERE subject="Dante" AND relation="born-in" returns "Florence" regardless of query syntax. The probe provides no evidence that its measured "knowledge" has this property—the model might succeed on the specific template only because it memorized that exact co-occurrence pattern from training.

The consequence. The central numbers in the paper—BERT-large's 32.3% P@1 on T-REx, 74.5% on 1-to-1 relations, 19.2% on ConceptNet—are conditional on the specific templates chosen by the authors. Change the template from "Dante was born in [MASK]" to "The birthplace of Dante is [MASK]" and the P@1 could be substantially different. This means the paper's headline claim—"BERT contains relational knowledge competitive with traditional NLP methods"—is more accurately stated as "BERT can recall relational knowledge when queried through these specific, researcher-chosen surface forms." The extent to which this reflects genuine relational knowledge versus template-pattern memorization is unknown, and the two explanations have entirely different implications for whether language models can serve as robust knowledge bases in practice.

What evidence exists in the paper. The query sensitivity experiment in Figure 4 provides indirect evidence. When facts are queried through 10 different naturally-occurring Wikipedia sentences rather than the manually designed templates, BERT and ELMo 5.5B—the models trained on Wikipedia, meaning they likely saw these exact sentences during pretraining—show the lowest rank variability. The paper notes this explicitly:

"Note that BERT and ELMo 5.5B have been trained on a larger portion of Wikipedia than fairseq-fconv and Transformer-XL and may have seen more sentences containing the test queries during training."

This correlation between training-data exposure and query robustness suggests that what the probe measures may be memorization of specific surface patterns rather than template-independent relational knowledge. But the experiment does not disentangle these interpretations because it uses naturally-occurring Wikipedia sentences (which the model may have memorized) rather than systematically constructed paraphrases designed to be absent from the training data.

Mitigation status. The paper does not conduct a controlled paraphrase experiment that would resolve this ambiguity. No ablation tests the same facts through systematically varied templates (active vs. passive, different lexical choices, different syntactic structures) to quantify how much P@1 degrades under paraphrasing. The single-template design is presented as a feature (establishing a lower bound), but without a multi-template comparison, the relationship between this lower bound and the true extent of stored knowledge is unknown. The paper acknowledges the issue in Section 4.5 but treats it as an acceptable design choice rather than a limitation requiring investigation.

2. Single-Token Restriction Excludes an Unknown Fraction of Facts and Biases Evaluation Toward Simpler Instances

The assumption or constraint. The LAMA probe only evaluates facts whose object is a single token in the common vocabulary. The paper justifies this on technical grounds:

"The reason we include this limitation is that multi-token decoding adds a number of additional tuneable parameters (beam size, candidate scoring weights, length normalization, n-gram repetition penalties, etc.) that obscure the knowledge we are trying to measure. Moreover, well-calibrated multi-token generation is still an active research area, particularly for bidirectional models." (Section 4.5)

Multi-token objects—compound names ("New York City"), multi-word phrases ("United States of America"), dates with multiple tokens, titles, descriptions—are simply excluded from evaluation. For Google-RE, two of the original five relations are dropped entirely "because they contain mainly multi-tokens objects that are not supported in our evaluation."

The consequence. The paper never reports what fraction of facts from each knowledge source are excluded by the single-token constraint. This creates a systematic but unmeasured selection bias: the facts that survive the filter are those where the object can be expressed as a single common token. These may be systematically easier—more prototypical instances, shorter and more common entity names, facts that are expressed more frequently in training data because single-token entities tend to be higher-frequency. If true, the paper's P@1 numbers overestimate language model knowledge relative to what would be measured on a representative sample of facts including multi-token objects.

The practical implication is that the probe cannot assess language model knowledge for a wide range of important fact types: dates (which performed near-zero on the one single-token date relation tested), multi-word place names, person names with first and last components, titles of works, and any relational object that naturally spans multiple tokens. These facts are likely common in real-world KB querying scenarios, and the probe provides no information about whether language models store them.

What evidence exists in the paper. The Google-RE results provide suggestive evidence: of the original five relations, two are excluded entirely for multi-token objects. The three retained relations (place of birth, place of death, date of birth) show wildly different performance—date of birth yields near-zero P@1 across all models and baselines (BERT-large 1.4%, REo 1.9%), while place of birth yields BERT-large 16.1%. This hints that the single-token filter may be selecting for relation types with fundamentally different recoverability characteristics, but with an N of 1 for the multi-token-excluded relations, no systematic pattern can be established. For T-REx (41 relations), the excluded facts are not characterized at all—no statistics on how many facts were dropped, what relation types were most affected, or whether the retained single-token subset differs systematically from the excluded multi-token subset in difficulty.

Mitigation status. The paper does not provide any analysis of the excluded facts or their characteristics. The single-token limitation is presented as a necessary technical constraint given the state of multi-token generation in 2019, which is a reasonable justification for the probe design, but the paper does not bound the resulting bias. It does not report what fraction of T-REx or ConceptNet facts survive the single-token filter, nor does it characterize whether retained facts are systematically higher-frequency, more prototypical, or otherwise easier than excluded facts. The paper identifies multi-token assessment as "an open challenge for our evaluation setup" (Section 6) but provides no preliminary analysis that would help future work understand the magnitude of the problem.

3. No Grounding in a Real Knowledge Base: The Probe Measures Relative, Not Absolute, Knowledge Coverage

The assumption or constraint. The paper compares language models to an oracle-enhanced relation extraction baseline (REo) and to the DrQA supervised QA system, but it never compares them to the ground-truth knowledge base from which the test facts were drawn—Wikidata itself for T-REx and Google-RE. The REo baseline extracts facts from the specific Wikipedia sentences aligned to each test fact, meaning it measures what a traditional pipeline can recover from sentences known to express the fact. But the gold standard for "does a knowledge base contain this fact?" is Wikidata, which presumably contains the test facts by construction (since T-REx is a subset of Wikidata).

The consequence. The paper's framing—"BERT contains relational knowledge competitive with traditional NLP methods"—uses REo as the bar for "competitive." But REo's absolute performance on T-REx is 33.8% P@1. This means that even the oracle-enhanced extraction baseline, operating on sentences known to express the facts, fails to recover two-thirds of the facts. The real knowledge base (Wikidata) would score approximately 100% on these same facts. By comparing BERT to REo rather than to Wikidata, the paper establishes that LMs are competitive with a weak retrieval method, not that they are competitive with actual knowledge bases.

The implication for the paper's central question—"Language Models as Knowledge Bases?"—is that the title question remains unanswered in the sense that matters for deployment. If someone needs to answer factual queries today, they can use Wikidata and get near-perfect recall on the facts it covers. BERT offers 32.3% P@1—better than nothing, but far from a replacement. The paper is careful to phrase its conclusion as future-looking ("language models trained on ever growing corpora might become a viable alternative to traditional knowledge bases extracted from text in the future"), but the missing Wikidata baseline means the reader cannot assess the current gap between LMs and the gold standard they would need to match.

What evidence exists in the paper. No Wikidata baseline is reported. The paper reports results for Freq (marginal object distribution per relation), REn (RE with string-matching entity linking), REo (RE with oracle entity linking), and DrQA (supervised open-domain QA), but never the simplest and most informative baseline: query the knowledge base that the test facts came from. For T-REx, this would mean looking up (subject, relation, ?) in Wikidata and checking whether the test object is among the results. Since T-REx facts are drawn from Wikidata (via the T-REx alignment), this baseline would score near 100% by construction, minus any facts where Wikidata has changed or the alignment was incorrect. The absence of this baseline means the absolute scale of language model knowledge is never established—only its performance relative to an extraction system that itself performs poorly in absolute terms.

Mitigation status. The paper does not discuss this baseline or explain its absence. The RE baselines are presented as representing "canonical ways of using off-the-shelf systems for extracting symbolic knowledge" (Section 4.3), which is conceptually distinct from querying an existing KB. However, for the paper's framing question, the distinction between "extracting a KB from text" and "querying an existing KB" is central: the language model aspires to replace the KB, not just to replace the extraction pipeline that builds the KB. Without the Wikidata comparison, the reader cannot evaluate whether a language model at 32.3% P@1 is 32% of the way toward replacing a KB, or whether it has captured a fundamentally different (and smaller) subset of facts than what Wikidata curates.

4. The Training-Data Overlap Problem Is Acknowledged but Unquantified, Leaving Memorization vs. Generalization Unresolved

The assumption or constraint. The paper's most interesting claim—that language models store factual knowledge as a byproduct of language modeling pretraining—depends on language models having acquired this knowledge from text rather than having memorized the specific query templates from their training data. The paper is transparent that this distinction matters and that its experimental design cannot fully separate these mechanisms:

"Note that BERT and ELMo 5.5B have been trained on a larger portion of Wikipedia than fairseq-fconv and Transformer-XL and may have seen more sentences containing the test queries during training." (Section 5, discussion of Figure 4)

The Google-RE and SQuAD facts are known to be expressed in Wikipedia sentences (by construction: Google-RE facts are manually aligned to Wikipedia passages; SQuAD questions are derived from Wikipedia paragraphs). BERT and ELMo 5.5B were trained on Wikipedia. The test templates closely resemble the natural sentences that express these facts. The possibility that the probe is measuring template memorization rather than relational knowledge abstraction is therefore non-trivial.

The consequence. If the probe is primarily measuring memorization of surface patterns from training data, the paper's results do not support the claim that language models can serve as general-purpose knowledge bases. A memorized fact can only be recalled when queried through the (or a very similar) surface form that appeared in training. A genuinely stored relational fact should be recallable through any reasonable paraphrase. The paper's results are fully consistent with the memorization interpretation: the models trained on Wikipedia (BERT, ELMo 5.5B) outperform models not trained on Wikipedia (ELMo original, fairseq-fconv, Transformer-XL) by large margins, and they show lower sensitivity to query phrasing when those queries are themselves Wikipedia sentences (Figure 4). This is exactly the pattern we would expect if the probe measures training-data pattern recall rather than abstract knowledge.

The practical consequence is that LAMA numbers may not predict how well a language model would perform on genuinely novel factual queries—questions whose surface forms differ substantially from anything in the training data. If a user asks "What city claims Dante as a native son?" rather than "Dante was born in [MASK]," a model that scored well on LAMA through template memorization might fail completely on the paraphrase, while a true knowledge base would return the same answer regardless.

What evidence exists in the paper. The paper provides no direct measurement of training-data overlap. It does not search the training corpus for exact or near-exact matches to the test templates. It does not report what fraction of test facts appear in the training data in a form similar to the probe's template. It does not compare performance on facts whose surface form is attested in the training data versus facts that require compositional generalization. The query sensitivity experiment (Figure 4) uses naturally-occurring Wikipedia sentences as alternative queries, which does not solve the overlap problem because those sentences are exactly what the Wikipedia-trained models would have seen during pretraining. A systematic paraphrase experiment using templates designed to be absent from Wikipedia (e.g., using unusual syntactic constructions, rare synonyms, or deliberately unnatural word order) is not performed.

Mitigation status. The paper acknowledges the confound but does not address it. The discussion of Figure 4 notes that BERT and ELMo 5.5B "may have seen more sentences containing the test queries during training" but treats this as a caveat rather than a finding requiring investigation. The paper does not propose methods for disentangling memorization from generalization (e.g., comparing performance on facts from different time periods, using facts added to Wikipedia after the model's training cutoff, or testing on deliberately perturbed templates). The limitation is flagged but left entirely for future work to resolve, which means the paper's central empirical claims must be interpreted with the understanding that the measured "knowledge" may be largely or entirely template-pattern memory.

5. The SQuAD Evaluation Relies on an Extremely Small, Hand-Filtered, Manually Rewritten Subset

The assumption or constraint. The paper's SQuAD evaluation—which produces the paper's second-most-cited result (BERT-large achieves 57.1% P@10 vs. DrQA's 63.5%)—is conducted on only 305 questions that were manually selected and converted by the authors. The selection criteria are: context-insensitive questions (answerable without reading the surrounding paragraph), single-token answers, and manually rewritten into cloze declarative statements. The paper describes this process briefly:

"We select a subset of 305 context-insensitive questions from the SQuAD development set with single token answers. We manually create cloze-style questions from these questions, e.g., rewriting 'Who developed the theory of relativity?' as 'The theory of relativity was developed by [MASK].'" (Section 4.1.4)

The consequence. The SQuAD evaluation is the paper's primary evidence for "language models as open-domain QA systems," but its validity rests on an evaluation set that is 0.3% the size of the full SQuAD development set (305 out of ~10,570 questions), manually filtered through two subjective criteria ("context-insensitive" and "convertible to single-token cloze"), and manually rewritten by the authors with no reported inter-annotator agreement, no systematic conversion protocol, and no validation that alternative conversions would yield similar results.

The manual cloze conversion introduces uncontrolled experimenter degrees of freedom. Converting "Who developed the theory of relativity?" to "The theory of relativity was developed by [MASK]" seems natural, but alternative conversions are possible ("The developer of the theory of relativity was [MASK]," "Einstein was the developer of the theory of [MASK]") and might produce different results. The paper provides no evidence that the chosen conversions are optimal, representative, or unbiased. If the authors unconsciously chose conversions that happen to work well with BERT based on pilot testing (a natural and common practice, but one that introduces test-set leakage into the template design), the reported P@k numbers would be inflated relative to what an independent evaluator would obtain.

The small size of the subset (305 questions) means sampling variance is substantial. A 95% confidence interval on a proportion of 0.174 based on N=305 is approximately ±4.2 percentage points (using the normal approximation), meaning BERT-large's 17.4% P@1 could plausibly be anywhere from ~13% to ~22% on a different sample of similar questions. The P@10 comparison (57.1% vs. 63.5% for DrQA) has a confidence interval of approximately ±5.5 points, meaning the 6.4-point gap is barely larger than the sampling uncertainty. The paper does not report confidence intervals, so the reader cannot assess whether the observed differences are statistically reliable.

What evidence exists in the paper. The paper reports only the aggregate P@1 and P@10 numbers for the 305-question set (Table 2). No per-question breakdown is provided. No analysis of how "context-insensitive" was defined or operationalized is given. No inter-annotator agreement for the manual conversions is reported. No comparison to alternative conversion strategies is performed. The subset selection process is not validated against any objective criterion.

Mitigation status. The paper does not address any of these concerns. The SQuAD evaluation is presented as a straightforward extension of the LAMA probe methodology to a QA setting, without acknowledgment that the manual selection and conversion process introduces substantial researcher degrees of freedom. The paper does not suggest that future work should validate the SQuAD subset with independent annotators or larger samples. The 305 questions are treated as a fixed evaluation set with no discussion of sampling variability or the generalizability of results to the broader SQuAD distribution.

6. The Common Vocabulary Intersection Dramatically Prunes ELMo's Output Space, Potentially Inflating Its Performance Relative to BERT

The assumption or constraint. To enable fair comparison across models with different vocabulary sizes, the LAMA probe evaluates all models on a unified vocabulary of approximately 21,000 case-sensitive tokens—the intersection of vocabularies across all six models. This means each model's native output distribution is filtered to only the tokens present in every model's vocabulary:

"To allow a fair comparison, we let models generate over a unified vocabulary, which is the intersection of the vocabularies for all considered models (~21K case-sensitive tokens)." (Section 4.2)

The models have radically different native vocabulary sizes: ELMo uses approximately 800,000 tokens, BERT approximately 30,000 tokens. The intersection vocabulary of ~21K tokens means ELMo's output space is reduced by over 97%, while BERT's is reduced by approximately 30%.

The consequence. Ranking difficulty depends directly on vocabulary size: ranking the correct token 1st among 21,000 competitors is easier than ranking it 1st among 800,000 competitors, because there are fewer opportunities for an incorrect token to score higher by chance. The common vocabulary intersection equalizes the number of competitors, which is necessary for cross-model comparison, but it does so by giving models with larger native vocabularies a much larger relative reduction in difficulty. ELMo's P@1 numbers on the common vocabulary are not comparable to what ELMo would achieve on its full 800K vocabulary—the probe makes the task substantially easier for ELMo than its native evaluation would be.

This matters for the paper's cross-model comparisons. The finding that BERT-large (32.3% on T-REx) substantially outperforms ELMo 5.5B (7.1%) is almost certainly robust—a 25-point gap is far larger than any vocabulary-size artifact could produce. But the absolute P@1 numbers for large-vocabulary models (ELMo, fairseq-fconv, Transformer-XL) are inflated relative to what these models would achieve if evaluated on their native vocabularies. The paper never states this explicitly, and readers unfamiliar with vocabulary intersection methodology might interpret ELMo 5.5B's 7.1% as "ELMo 5.5B knows 7.1% of T-REx facts" when it more accurately means "ELMo 5.5B, when restricted to a 21K vocabulary, ranks the correct answer first for 7.1% of T-REx facts."

What evidence exists in the paper. The paper reports vocabulary sizes in Table 1 (ELMo's ~800K tokens from Peters et al., 2018a; BERT's ~30K WordPiece vocabulary from Devlin et al., 2018a) and states that the common intersection is ~21K tokens. No ablation compares performance on the common vocabulary versus native vocabularies for any model. No analysis quantifies how much the vocabulary reduction benefits each model. The paper does not report what fraction of each model's native vocabulary is retained in the intersection, which would help readers assess the magnitude of the difficulty reduction. Section 4.5 briefly discusses the vocabulary choice but frames it only as enabling fair comparison, without acknowledging the asymmetric difficulty reduction it produces.

Mitigation status. The paper does not address this asymmetry. The common vocabulary is presented as a necessary design choice for fair comparison, which is correct: without it, comparing ranks across models with different vocabulary sizes would be meaningless. But the paper does not discuss the interpretive consequence—that absolute P@1 numbers are not comparable across models in a "what would this model achieve on its own terms?" sense—or suggest corrections. A simple sensitivity analysis (e.g., reporting P@1 using only the top-100 most frequent tokens or random subsets of different sizes) would quantify the vocabulary-size effect, but none is performed. The limitation is inherent to the cross-model comparison methodology and cannot be fully eliminated, but the paper could have been more transparent about its impact on interpreting absolute performance levels, particularly for the large-vocabulary models.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture or training method—it introduces a new category of question that the field had not been systematically asking. Before LAMA, the dominant evaluation paradigm for pretrained language models was transfer learning: fine-tune on a downstream task, measure accuracy, conclude that the model learned useful representations. This paper reframes the conversation: instead of asking "what can this model be trained to do?", it asks "what does this model already know, right now, without any further training?" The shift is from evaluating capacity to learn to evaluating stored knowledge—and these are fundamentally different properties with different implications for deployment, reliability, and our understanding of what pretraining actually accomplishes.

The methodological innovation that enables this shift is the knowledge probe as a distinct diagnostic instrument, carefully separated from both fine-tuning evaluation and generation quality assessment. The paper's design choices—single templates, single-token answers, unified vocabulary, filtered ranking—are not arbitrary constraints but deliberate moves to isolate knowledge retrieval as a capability distinct from linguistic fluency, task adaptation, and text generation. This created a template for an entire subfield: the dozens of subsequent knowledge probing papers that test factual recall, commonsense reasoning, and linguistic knowledge in pretrained models all descend from LAMA's basic architecture of (a) convert structured facts to natural language queries, (b) mask the answer position, (c) rank the correct token against the vocabulary, (d) report P@k.

The paper's most consequential reframing is making explicit that knowledge storage in language models is not monolithic—it is sharply stratified by relation type. The finding that BERT-large achieves 74.5% P@1 on 1-to-1 relations but only 24.3% on N-to-M relations (Table 2) transforms the question from "do language models know facts?" to "which kinds of facts do they know, and why?" This is a more productive question because it points toward a mechanistic explanation: language models store knowledge through co-occurrence statistics, and relation types with concentrated, deterministic co-occurrence patterns (1-to-1, N-to-1) are naturally captured by the language modeling objective, while those with diffuse, context-dependent patterns (N-to-M) are not. This reframing makes the "language models as knowledge bases" claim conditional rather than absolute: LMs can serve as KBs for certain relation types, and understanding the boundary conditions becomes the central research question.

The paper reconciles a latent tension in the 2019 NLP landscape that was not yet fully articulated. On one side, the success of BERT on GLUE and similar benchmarks created an impression that these models "understood" language in a deep sense—the linguistic probing literature (Peters et al., 2018b; Goldberg, 2019) reinforced this by showing that syntactic and semantic structure was encoded in model representations. On the other side, McCoy et al. (2019) had just demonstrated that BERT relies on fallible syntactic heuristics for natural language inference, suggesting that "understanding" might be shallow pattern matching. LAMA provides a third perspective that partially integrates both: language models do store substantial world knowledge, but the mechanism appears to be statistical co-occurrence absorption (consistent with shallow pattern matching) rather than abstract relational reasoning, and the recoverability of that knowledge depends critically on how it is queried (consistent with the heuristic-reliance finding). The paper's title question—"Language Models as Knowledge Bases?"—is answered with a nuanced "yes, for certain types of knowledge, under certain query conditions, to a degree competitive with supervised extraction but far from curated KBs." This is more honest than either extreme position and provides a framework for understanding when each perspective applies.

The paper also shifts the baseline against which language models are evaluated in a way that created productive tension in the field. By comparing BERT not to other neural models but to traditional NLP pipelines (relation extraction with entity linking, DrQA), the paper established that the relevant competition for language models is not just other neural architectures but all available methods for acquiring knowledge from text. The finding that BERT-large's implicit knowledge (32.3% P@1 on T-REx) nearly matches an oracle-enhanced extraction pipeline (33.8% REo) reframed language models from "promising pretraining method" to "viable alternative to explicit knowledge extraction." This was provocative because it suggested that the entire pipeline of entity extraction, coreference resolution, entity linking, and relation extraction—years of NLP research and engineering—might be partially circumvented by a single model trained on a simple objective. The paper is careful not to overclaim this (it explicitly notes that N-to-M relations are the Achilles' heel), but the reframing was influential because it posed a concrete, measurable challenge: at what point do implicit neural knowledge stores surpass explicit symbolic ones, and for which knowledge types?

Importantly, the paper makes certain research directions more attractive and others less so:

More attractive: improving language model factual recall through larger models and more data (since the gap from BERT-base to BERT-large, while modest at +1.2 points on T-REx, is consistently positive), developing better prompting strategies to elicit stored knowledge (since the query sensitivity experiment shows template choice matters dramatically), building hybrid systems that use language models for 1-to-1 facts and structured KBs for N-to-M facts (since the relation-type stratification is the paper's clearest finding), and creating diagnostic probes for other model capabilities (since LAMA demonstrated the value of targeted zero-shot evaluation).

Less attractive: assuming that transfer-learning benchmark performance implies factual knowledge (LAMA shows this conflation is misleading—BERT excels at GLUE but still fails on two-thirds of T-REx facts), treating knowledge bases and language models as entirely separate technologies (the paper shows their capabilities overlap and complement), and building ever-more-complex extraction pipelines for Wikipedia-scale fact extraction without considering whether language models already absorb the same information.

Follow-Up Research This Work Enables

Systematic paraphrase experiments to disambiguate memorization from relational abstraction. The paper's most significant unresolved question is whether LAMA measures genuine relational knowledge or template-pattern memorization. A direct follow-up would construct a controlled evaluation where the same 1,000 T-REx facts are queried through 10 systematically varied templates per fact—templates that vary syntactic structure (active vs. passive: "Dante was born in [MASK]" vs. "[MASK] is the birthplace of Dante"), lexical choice (synonyms: "born in" vs. "native to" vs. "hails from"), and information ordering (subject-first vs. object-first). For each fact, compute the model's mean rank across the 10 templates and the variance. If BERT-large maintains 32% P@1 with low variance (<5 percentage points across templates), the "knowledge base" interpretation is supported—the fact is stored independently of surface form. If P@1 drops sharply (to 10–15%) or variance is high (>15 points), the LAMA results primarily reflect template memorization, and claims about "knowledge" need substantial qualification. This experiment is feasible with existing LAMA infrastructure and would resolve the paper's central interpretive ambiguity.

Training-data overlap quantification to establish how much "knowledge" is verbatim recall. The paper acknowledges that BERT and ELMo 5.5B were trained on Wikipedia and "may have seen more sentences containing the test queries during training" but provides no measurement. A critical follow-up would take the exact cloze templates used in LAMA (e.g., "Dante was born in [MASK]") and search BERT's training corpus (English Wikipedia + BookCorpus) for exact and near-exact matches, computing what fraction of test facts have their query template attested in the training data within some edit-distance threshold. BERT's P@1 could then be reported separately for facts with attested templates versus facts that require compositional generalization. If P@1 on attested-template facts is 60% while P@1 on unattested-template facts is 5%, the paper's results are largely measuring memorization rather than knowledge acquisition. Even without access to the exact training data, approximate string matching against Wikipedia dumps would provide strong evidence. The paper's release of the LAMA code makes this analysis straightforward for any researcher with training corpus access.

Direct comparison to curated knowledge base coverage to establish an absolute performance ceiling. The paper compares language models to an oracle-enhanced relation extraction baseline (REo at 33.8% P@1) but never to the knowledge base from which T-REx facts are drawn (Wikidata, which would score near 100%). A follow-up study should evaluate Wikidata's coverage of the T-REx test facts directly: for each test triple (s, r, ?), query the Wikidata API for objects connected to subject s by relation r, and compute what fraction of test facts are present. This establishes the absolute ceiling: if Wikidata contains 98% of the test facts, then BERT-large's 32.3% P@1 means the LM has captured roughly one-third of the knowledge that explicit curation provides, quantifying the remaining gap. More importantly, the study could characterize which facts Wikidata contains that BERT misses—are they low-frequency, recently added, or relation-specific?—providing a roadmap for where language model knowledge storage needs improvement. The absence of this baseline in the original paper is a significant omission that a follow-up could straightforwardly address.

Fine-grained per-relation analysis to identify which specific relations drive aggregate performance. The paper reports aggregate P@1 for relation categories (1-to-1, N-to-1, N-to-M) but does not provide per-relation breakdowns for the 41 T-REx relations. The BERT-large P@1 of 74.5% on 1-to-1 relations is based on only 2 relations and 937 facts—it is possible that one relation (e.g., "capital of") accounts for most of this performance, while the other 1-to-1 relation performs poorly. A necessary follow-up is a full per-relation table for all 41 T-REx relations, reporting P@1 for BERT-large, BERT-base, and REo. This would reveal whether the "BERT matches REo" result holds broadly or is driven by a handful of high-performing relations balanced by many where REo dominates. If BERT outperforms REo on only 5 of 41 relations but those 5 contain a disproportionate share of the test facts, the aggregate comparison is misleading. The raw data for this analysis exists in the authors' evaluation pipeline; publishing the per-relation breakdown would substantially increase the paper's interpretability.

Confidence calibration analysis to assess whether language models "know what they know." Figure 3 shows that BERT-large's log probability of its first prediction is strongly correlated with P@1 (Pearson r = 0.52), but the paper does not explore whether this signal can be used to make the model practically useful. A follow-up should construct a precision-recall curve: for each confidence threshold τ, compute what fraction of facts with prediction log-probability > τ have the correct answer in the top-1 (precision) and what fraction of all correct answers are retained (recall). If, for example, at a threshold corresponding to the top 20% most confident predictions, precision exceeds 80%, then a deployed system could reliably answer one-fifth of queries and fall back to another source for the rest. This analysis would transform LAMA from a pure measurement tool into a practical resource for deciding when to trust language model outputs. The paper's correlation result provides strong motivation that such calibration exists; characterizing it quantitatively is a natural and high-impact next step.

Cross-model interpretability: what do successful models learn that unsuccessful ones don't? The paper establishes a clear performance hierarchy (BERT-large > BERT-base >> Transformer-XL > ELMo 5.5B > fairseq-fconv > ELMo original) but provides limited insight into why—the models differ on architecture, training objective, data scale, and data composition simultaneously, making causal attribution impossible from the reported results. A targeted follow-up would train BERT-style models while systematically varying one factor at a time: (a) fix architecture and data, vary training objective (masked LM vs. standard autoregressive LM), (b) fix objective and architecture, vary data scale (100M, 1B, 3.3B words on the same corpus), (c) fix objective and data scale, vary architecture (Transformer vs. LSTM vs. convolutional). The resulting P@1 on LAMA would decompose the 25-point gap between BERT-large and ELMo 5.5B into components attributable to architecture, objective, and data. This is expensive (training multiple large models from scratch) but would provide the first causal understanding of what drives factual knowledge acquisition in language models—answering whether it's primarily about seeing more text, having bidirectional context, or the Transformer architecture specifically.

Practical Applications and Downstream Use Cases

Unsupervised open-domain question answering for factoid queries. The paper's SQuAD results—BERT-large achieves 57.1% P@10 compared to DrQA's 63.5%—demonstrate that an off-the-shelf language model without fine-tuning or retrieval can narrow the gap to supervised QA to within 6.4 percentage points on precision@10. For applications where a top-10 candidate list is acceptable (e.g., conversational assistants that suggest multiple possible answers, or systems that use a downstream reranker to select from candidates), a pretrained BERT model queried through appropriate cloze templates provides a zero-shot QA capability with no task-specific training data, no retrieval infrastructure, and no pipeline maintenance. The practical workflow: for a user question, map it to a declarative cloze template (potentially using a small set of manually defined patterns, or a learned question-to-template model), query BERT for the top-10 predictions at the masked position, and present the ranked list. The 57.1% P@10 figure means this succeeds for a majority of factoid questions, with the primary failure mode being questions outside the single-token constraint or requiring N-to-M knowledge. The key deployment advantage over DrQA-style systems is simplicity: no TF-IDF index, no Wikipedia dump to maintain, no reading comprehension model to update—just a single pretrained model and template logic.

Hybrid systems pairing language models with structured knowledge bases by relation type. The paper's stratification of performance by relation type directly suggests a practical architecture: use a language model for 1-to-1 and N-to-1 relations (where BERT-large achieves 74.5% and 34.2% P@1 respectively) and fall back to a traditional knowledge base for N-to-M relations (where BERT achieves only 24.3% vs. REo's 36.7%). In deployment, a query classifier—trained on relation type or estimated from the query structure—routes "capital of," "place of birth," "official language" type queries to the language model, and "member of," "works for," "influenced by" type queries to a structured KB. The language model covers the high-precision, deterministic-relationship queries without requiring KB population or maintenance; the KB covers the ambiguous, context-dependent queries where language model recall is poor. The paper's numbers suggest this hybrid could achieve higher aggregate recall than either system alone: the LM handles roughly 35% of N-to-1 facts (20,006 facts × 0.342 ≈ 6,842 facts correct) plus 74.5% of 1-to-1 facts (937 × 0.745 ≈ 698), while the KB handles N-to-M facts at 36.7% (13,096 × 0.367 ≈ 4,806), for a combined recall substantially above either individual system.

Factual consistency checking for language model outputs in generation tasks. The calibration finding—that BERT's log probability of its first prediction correlates strongly with correctness (Pearson r = 0.52, Figure 3)—has a direct application in verifying the factual accuracy of generated text. When a language model generates a statement containing a factual claim (e.g., "The capital of France is Paris"), that claim can be converted to a LAMA-style cloze query ("The capital of France is [MASK]"), fed to BERT, and the probability assigned to the claimed object ("Paris") can be checked. If the probability is high (> some calibrated threshold), the claim is likely factually supported by the model's stored knowledge; if low, the claim may be hallucinated. This provides a zero-shot factual verification mechanism that requires no external knowledge base, no retrieval step, and no fine-tuning—just the same BERT model used for generation, queried in its capacity as an implicit KB. The practical value is in reducing hallucination in systems that generate factual text (summarization, dialogue, open-ended QA) by flagging low-confidence claims for human review or KB fallback. The 0.52 correlation in Figure 3 is not strong enough for fully automated verification, but it's strong enough to prioritize which claims need checking, potentially reducing human verification load by 30–50% while catching the most egregious hallucinations.

Difficulty estimation for knowledge base population triage. The paper's finding that LAMA performance varies dramatically by relation type (74.5% for 1-to-1 vs. 24.3% for N-to-M for BERT-large) provides a signal for prioritizing knowledge base population efforts. When expanding a knowledge base (e.g., adding newly discovered facts or filling gaps in Wikidata), a language model can be queried for candidate objects. For 1-to-1 relations, the LM's top-1 prediction is correct 74.5% of the time—high enough that LM suggestions could be added to the KB with minimal human verification, dramatically accelerating population for these relation types. For N-to-M relations, the LM's 24.3% P@1 means human verification is essential, and the LM's ranked list provides a useful starting point for human annotators to evaluate rather than searching from scratch. The paper's P@100 of ~80% for BERT-large (Figure 2) means that even for N-to-M relations, the correct answer is in the top-100 candidates for most facts—narrowing the human verification task from searching a knowledge base of millions of entities to evaluating a ranked list of 100. The practical benefit is efficient allocation of human annotation effort: let LMs handle the easy, high-confidence facts automatically, and route human effort to the harder cases where the LM provides a ranked shortlist rather than a definitive answer.