According to a comprehensive review by MIT Technology Review as part of its "What's Next" series, the entire artificial intelligence industry currently relies on the transformer technology introduced by Google researchers in 2017 in their seminal paper "Attention Is All You Need." Nine years later, transformer engines power all the leading large language models (LLMs) on the market. Justin Dangel, co-founder and CEO of the startup Subquadratic, notes that this represents one of the most important innovations in the history of computer science, profoundly changing the world.
However, transformers are beginning to show their age. Recent advances in the field, such as reasoning models and the ability to process massive amounts of input data simultaneously, are not natural and smooth extensions of this core technology, but rather workarounds and patches designed to cover up its fundamental flaws. Many scientists and engineers are now asking what will come next. While LLMs themselves are not going away, the way they are built is completely open to change. MIT Technology Review dubbed this next generation of models "LLMs+" in this year's list of the ten most important things in AI. Now, a wave of new startups is attempting to push the boundaries of the field, knowing they have less to lose than the leading companies currently at the head of the industry.
The Bottleneck and Computational Costs of Transformers
To understand the need for change, one must first examine the central problem with transformers. Their strength is based on a mechanism known as "dense attention," which encodes the meaning of text by comparing every word (or part of a word, known as a token) to every other word in the text through numerous multiplication operations. This mechanism captures the meaning of the text with high accuracy, but as the text grows longer, the number of computations increases rapidly: processing a document 10,000 words long requires a transformer to perform approximately 50 million multiplications, which is the primary driver of the massive energy consumption of these models.
The associated costs are immense. OpenAI President Greg Brockman stated that the company is expected to spend approximately $50 billion on computing resources during this year alone. Meanwhile, the International Energy Agency (IEA) predicts that the total amount of electricity consumed by data centers worldwide will double by 2030.
Furthermore, transformer mechanisms struggle to perform the tasks for which the most modern models are designed. Because they process text word by word, they struggle to manage a lot of information simultaneously, meaning their "context window" cannot be too large. However, for LLMs to carry out complex tasks, they must ingest much larger amounts of information: document libraries, entire codebases, or outputs from other models for AI agents. Reasoning models add to this burden because they write notes to themselves in a "chain of thought" and read them back, further increasing the amount of information that must be managed.
Below are four innovative and groundbreaking directions designed to solve the transformer problem and make language models faster, more efficient, and much smarter.
01: Rethinking the Attention Mechanism
One direct way to make language models faster and cheaper is replacing dense attention with a "sparse attention" mechanism, which performs computations on only some of the word pairings in a block of text instead of all of them, thereby significantly reducing the required computation. In the past, proposed sparse attention mechanisms were not as good as dense attention at capturing text meaning.
This situation may be changing. The Miami-based startup Subquadratic claims to have developed the first sparse attention mechanism, called SubQ, which delivers performance that rivals leading models on the market across several tasks, including search and coding. The model identifies in real time, for each piece of text, which words are relevant and which are unimportant. The company plans to make the model publicly available soon and says thousands have already signed up for its waitlist.
At the same time, San Francisco-based Manifest AI has developed an alternative mechanism called "power retention." Instead of keeping all information within the context window, the mechanism provides a rolling summary of the most relevant information, omitting less important details as new data is received. The company claims that an existing transformer model can be converted into a power retention-based model with minimal retraining. It demonstrated this by converting the open-source coding model StarCoder into a version called PowerCoder, and has also released a model named Brumby, which it claims rivals versions of Alibaba's open-source Qwen model. Manifest AI co-founder and CTO Carles Gelada emphasized that the technology is particularly suited for processing massive amounts of data, ranging from analyzing hours-long videos to AI agents operating for consecutive weeks.
02: Smaller and More Flexible Models
Liquid AI, an MIT spinout based in Cambridge, Massachusetts, combines transformers with its own unique "liquid neural networks" technology to create liquid foundation models, or LFMs. CEO and co-founder Ramin Hasani explains that these models are significantly smaller and more efficient. They are designed for systems like small chips in Mercedes vehicles and can run even on a low-powered, $50 Raspberry Pi computer. The models are offered for free to organizations with under $10 million in annual revenue and have recorded nearly 34 million downloads, according to Hasani.
Liquid networks, which were inspired by worm brains, represent an extension of convolutional networks and allow the model to adapt its behavior to new information in real time, unlike transformers whose behavior is completely fixed after the training phase. The company's latest LFMs are hybrid models composed of 20% transformers and 80% liquid neural networks, a ratio determined by a specialized AI system developed by the company to design its models. The models deliver performance comparable to competitors four times their size, such as versions of Qwen or Google's Gemma. Hasani notes that the human brain operates on only 20 watts of power, and we can be far more innovative in building AI systems.
03: Generating Whole Text All at Once
Most language models generate output word by word, a process that is highly inefficient for computers. Inception, a Palo Alto-based startup, proposes generating entire blocks of text all at once—complete sentences or paragraphs—using a "diffusion" technique, which typically drives image and video generation models.
Diffusion works on all pixels simultaneously to turn static noise into a clear image. Inception applies this principle to text, training models to turn a random string of words into logical sentences. Co-founder and CEO Stefano Ermon, who is also a researcher at Stanford University, explained that these models are much faster and cheaper because they allow predicting multiple tokens in parallel despite using a large transformer. In 2024, Ermon and his Stanford colleagues cracked the mathematics of this method and trained a diffusion model that matched OpenAI's GPT-2 but was 10 times faster.
The company's current model, Mercury 2, matches the performance of OpenAI's GPT-4 models from 2023 but is 10 times faster. Google is also experimenting with this approach with a prototype called Diffusion Gemma. Ermon emphasizes that speed and cost are the decisive variables, and the primary currency of the future will be the level of intelligence obtained per dollar.
04: Moving Beyond Words
Another Palo Alto-based startup, Pathway, aims to free language models from the constraints of words, arguing that language is not the most suitable tool for all types of reasoning (such as chess or mathematics). The company developed a model called Dragon Hatchling, which successfully solved over 97% of a benchmark consisting of 250,000 extremely difficult sudoku puzzles, a task on which other leading models failed.
Co-founder and CEO Zuzanna Stamirowska explains that the company replaces the attention mechanism with a mathematical structure called a "state space," which compresses information into an abstract representation instead of encoding it word by word. This allows the model to perform reasoning tasks that are not based on word sequences. According to her, transformers were a convenient engineering compromise that became a "religion," but there is no reason to assume that another breakthrough won't occur to change the field and help solve complex problems like curing cancer.