Google Research Blog Post: Empty Shelves or Lost Keys?
In a research post published by Nitay Calderon and Gal Yona, research scientists at Google Research, on August 12, 2026, a new approach is presented for understanding factual errors in large language models (LLMs). In their paper "Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality", the researchers introduce the "knowledge profiling" framework and a new evaluation benchmark called WikiProfile, aiming to determine whether advanced language models make factual errors because the facts were never stored in the model (an encoding failure, referred to as "empty shelves") or because they are encoded but the model fails to access and retrieve them without external cues (a recall failure, referred to as "lost keys").
According to the authors, understanding the difference between an encoding failure and a recall failure is crucial, as the two types of failures require entirely different solutions. An encoding failure requires scaling the model size or expanding the training dataset, while a recall failure may indicate a need for post-training or inference-time methods to help models better utilize the information already encoded in them.
The Knowledge Profiling Framework: Definitions and Conceptualization
The knowledge profiling framework shifts the unit of analysis from individual questions to the facts themselves. Instead of merely examining whether a specific question was answered correctly, it classifies each fact into one of five defined profiles:
- Encoding failure (encoding failure): The fact is not represented in the model's parameters.
- Recall failure (recall failure): The fact is encoded in the model, but it fails to recall it without external cues.
- Direct recall (direct recall): The model knows and provides the fact directly without needing to invoke thinking.
- Recall with thinking (recall with thinking): The model recalls the encoded fact only when allowed to perform intermediate calculations (including chain-of-thought or in thinking-optimized models).
- Inference without encoding (inference without encoding): The model correctly answers a question about a fact that is not encoded in it, by relying on other encoded facts, performing multi-hop reasoning, or making educated guesses under the activation of thinking mechanisms.
The researchers define three key behaviors to operationalize the framework:
- Encoding (Encoding): A model encodes a fact if it is capable of reproducing it in a context similar to that of the pre-training stage. The study measures this using proposition completion and contextual questioning that places the model in a context where the fact naturally appears, without revealing the answer.
- Knowledge (Knowledge): A model knows a fact if it can correctly answer semantically equivalent questions in different phrasings, including both direct and reverse questions (for example, for the fact that A is B, a direct question would ask "What is B?", while a reverse question would ask "What is A?").
- Recall (Recall): A model recalls a fact if it knows an encoded fact. Recall without thinking is defined as direct recall.
- Recognition (Recognition): Recognizing the correct fact when presented alongside alternative options (in multiple-choice tasks).
The WikiProfile Benchmark: Data Construction and Methodology
To operationalize the knowledge profiling framework, the researchers developed WikiProfile, an evaluation benchmark consisting of 2,150 facts sourced from Wikipedia, which is a major source of pre-training data.
WikiProfile was built using a fully automated process based on the Gemini-2.5-Pro language model in thinking mode. System prompts were manually optimized on a small, pre-separated held-out dataset. The fact extraction process consists of extracting facts from Wikipedia pages, defined as propositions containing an ordered pair of entities (subject and object), with the subject appearing first in the document.
Each fact is paired with 10 tasks: two tasks for measuring encoding, four tasks for evaluating knowledge (direct and reverse questions in different phrasings), and four multiple-choice tasks for evaluating recognition. The generation of direct and reverse questions is carried out through a three-step process of generation, refinement, and filtering, to ensure that each question is unambiguous, specific, minimal, and has a unique answer. All questions undergo filtering grounded in a search engine, and cases where multiple answers were returned or clarification was needed were disqualified. After automated filtering and a final manual verification step, 2,150 facts remained in the benchmark.
Model Evaluation and Key Results
The study evaluated 13 different language models (including Gemini-2.5-Pro, Gemini-3-Pro, Gemini-3-Flash, GPT-5, and the Gemma 3 model family), with each model tested both with and without the activation of a thinking mechanism. For each model, fact, and task, eight responses were sampled, which were graded automatically by prompt-instructed LLM autoraters, analyzing a total of approximately 4.5 million responses.
The main result emerging from the study is that in leading frontier models (Gemini-3-Pro, GPT-5, and Gemini-2.5-Pro), factual encoding is close to full saturation, but recall is far from it:
- In the Gemini-3-Pro and GPT-5 models, between 95% and 98% of the facts are encoded in the model's parameters.
- Despite this, the models fail to directly recall 26% to 34% of these facts.
- Even when the thinking mechanism is active, the models still fail to recall 11% to 12% of the encoded facts.
This means that factual errors in advanced models stem from an inability to reliably access knowledge already stored in them, rather than its absence. The bottleneck has shifted from knowledge acquisition to knowledge utilization.
This phenomenon is also reinforced under the analysis of the scaling effect. In the Gemma 3 model family, larger models display a sharp decline in encoding failures, but recall failures remain significant and become the lion's share of the remaining errors. In other words, scaling improves what the model stores far more effectively than it improves its ability to access this information.
Why Does Recall Fail? Rare Facts and the Reversal Curse
The results indicate that recalling facts is tightly coupled to the conditions under which the fact was learned during training. When the query is phrased differently from the training context, phrasing, or original word order in which the model encountered the fact, recall becomes harder. The researchers demonstrate this in two distinct cases:
- Rare facts (rare facts): While previous studies attributed the difficulty of models in handling facts in the "long tail" to model capacity limitations, the current study suggests a complementary picture. Comparing facts with low popularity (bottom 20%) to facts with high popularity (top 20%), it was found that rare facts are encoded at rates very close to popular facts. The gap in encoding is relatively small, but the gap in recall ability is significantly wider. Therefore, information about many rare facts is not missing from the model's parameters, but is simply difficult to access.
- The reversal curse (reversal curse): A phenomenon where models know that "A is B" but struggle to answer "What is B?". The study shows that in open-ended generation tasks (recall), reverse questions are indeed significantly harder than direct questions. However, in multiple-choice verification tasks (recognition), reverse questions are no harder than direct questions, and in many cases, they are even easier. This dissociation indicates that the bidirectional information exists and is encoded in the model, and it is even capable of recognizing it, but it struggles to recall it independently when the direction of the query differs from the original direction of learning. The reversal curse is a clear recall problem.
Thinking as a Knowledge Recall Mechanism
The researchers examined the role of the thinking mechanism (thinking) in overcoming recall failures. The findings show that thinking improves recall most significantly precisely where direct recall is weakest: for rare facts and for reverse questions, thereby narrowing both the popularity gap and the directionality gap.
In thinking-optimized models, the thinking mechanism makes it possible to recall about 40% to 65% of facts that were encoded but could not be directly recalled. On the other hand, thinking helps at a much lower rate (only 5% to 15%) for facts that are not encoded in the model at all. This pattern proves that thinking primarily serves as a mechanism that facilitates recall and access to existing information, rather than as a tool for generating answers from scratch through complex logical inference of missing facts.
However, the researchers note that invoking the thinking mechanism is not free of costs, as it involves a significant computational overhead, and it is still not entirely clear how to determine exactly when the model should activate this mechanism.
Conclusions
The study by Calderon and Yona suggests a conceptual shift regarding factual errors in advanced language models. Since the encoding stage of facts is already reaching saturation in newer models, further improvements in factuality will not necessarily be achieved by further scaling up the models or expanding the database. The key to improving factuality in future generations lies in better utilization of encoded knowledge, with thinking mechanisms emerging as an effective tool for reducing these recall failures.