This article originally appeared in "The Algorithm," the weekly artificial intelligence newsletter from MIT Technology Review.
Anthropic, currently considered the world’s most valuable artificial intelligence company with a valuation of nearly $1 trillion, holds a reputation for publishing particularly strange and complex research. For instance, the company examines whether AI models can feel pain and sometimes even cuts off conversations with chatbots if it suspects users are "abusing" the model.
One niche where Anthropic spends more time and money than other AI companies is called "mechanistic interpretability." This means looking inside the complex math of an AI model to learn why it arrives at one specific output and not another. It is an incredibly complicated subject; there are millions of data points that might contribute to any single outcome, and trying to find a path through them can often look more like a word salad than anything useful. This field is also controversial. Describing AI models using terms borrowed from psychology and neuroscience can make their behavior seem far more sophisticated than we might otherwise judge it to be.
That is why when Anthropic announced last week that it had found a new "window" into its models’ "internal thoughts" as they analyze and reason through their answers, there was a need to speak with senior editor Will Douglas Heaven. Beyond having a PhD in computer science, Heaven has spent a lot of time investigating what can actually be said about how AI models function in practice. The conversation with him focuses on what we should understand and learn from Anthropic's new and, predictably, quirky research.
Discovering the J-space: What did Anthropic find inside the Claude model?
For several years, Anthropic has been trying to understand how large language models (LLMs) work. Anthropic is not the only one doing this, but it seems the company has made this topic a central part of its core mission more than most other firms. Anthropic's CEO, Dario Amodei, has even declared in the past that we will not be able to fully control large language models unless we learn more about how they operate.
This new research fits right into that context, diving deeper than ever before into the strange mechanisms inside large language models. What Anthropic discovered is that these models have an internal space within them—which the company calls the "J-space" (or J-space)—filled with words that do not appear in their final output, but seem to influence the way they solve problems and crack tasks. This entire space remained completely hidden until Anthropic developed a new technique that allows probing and investigating its model, Claude, making this a genuine and proven discovery.
Sometimes, these words in the J-space track the stage the large language model has reached in performing a specific task; sometimes they look more like "flashes of recognition" (for example, the word "protein" might suddenly pop up when the language model is fed only the letters that make up a certain protein sequence); and in other cases, they represent a kind of internal commentary on the model's decision-making process. In Heaven's favorite example, the Claude model decided to cheat on a programming test exactly when the word "panic" appeared in its J-space.
In addition, Anthropic found that large language models are capable of describing and manipulating the words within this space. This suggests that they use this space and derive some benefit from it during their operation.
Why is it so hard to peer inside a large language model?
If we look at the big picture, large language models are not simple, but they are also not magic. It is a system of mathematical calculations that learns the relationships and connections between words. If so, why is it so difficult to peer inside a large language model and know what is happening within it?
Indeed, models are not magic, and the fact that we do not fully understand them feeds the creation of myths around them. It is worth noting that the entire narrative Anthropic is aiming for here—that it built a highly mysterious and complex technology, but don't worry, it is also the one that will crack and understand it—very much fits the company's character and image. This is similar to how Anthropic previously warned that its new models were so good at writing code that they posed a global information security risk, only for the United States government to shut them down shortly thereafter.
Large language models are indeed just math, but it is vastly complex math on an immense scale. Not only are today's large language models made of hundreds of billions of numbers, but running them triggers a massive cascade of millions and millions of calculations simultaneously. Heaven pointed out last year that if we were to print out even a medium-sized large language model on paper, it would cover an entire city the size of San Francisco.
It is impossible to understand or give meaning to this math without dedicated, specialized tools that highlight specific parts inside the large language model at defined times. You need to know exactly where to look and how to look, and building tools of this kind requires a deep understanding of that complex math to begin with.
Anthropomorphizing AI models and comparing them to the human brain
In Heaven's writing elsewhere, he has referred to the concept of investigating large language models in a manner similar to studying the brain of a living organism. Is it fair to use brain-like terms when talking about how a large language model works?
The senior editor explains that he does not like using these kinds of terms. Large language models are not human brains. Using such terms is misleading because it can suggest that large language models are capable of performing more human-like actions than they actually are, or that we can make assumptions about how they might behave—assumptions we should not make. The entire issue of anthropomorphization is also tied to a variety of strong ideological positions regarding what this technology is and what it is destined to be in the future.
However, at the same time, we lack a good alternative vocabulary to describe what these models actually do. It is understandable why people turn to and use words like "think," "understand," and "brain-like"—they simply serve as a convenient shorthand for describing these processes.
Anthropic compares this new space it discovered inside large language models to the space that some neuroscientists believe our brains use to track conscious thoughts. When the company was asked how seriously we should take this comparison, it stated in an official response: "Drawing these analogies was helpful to us in designing our experiments, as they allowed us to make many non-obvious experimental predictions about the J-space that turned out to be true. At the same time, it’s important to note that there are some important differences between the J-space (and language models in general) and the human brain, so we don’t mean to claim there’s a perfect correspondence."
Practical applications: Can J-space solve problems in AI?
Regarding the question of what problems in the field of artificial intelligence the new concept of J-space might solve, Anthropic claims that monitoring the J-space could be a way to detect and catch models when they perform actions they are not supposed to do. Because words pop up inside this space without appearing in the model's visible output, they can reveal and tell us things about its behavior that we might not have noticed otherwise—such as cases where the model provides biased answers or when it is weighing the pros and cons of cheating on a task.
However, Heaven emphasizes that this is only the theory. In his view, it is more correct and better to treat this research result as just another step on the long path to understanding this technology in general, rather than as a tool that will be useful on its own at this stage.