a16z Dialogues with AI Star Unicorn: How Will Large Models Evolve in the Future?
TechFlow Selected TechFlow Selected
a16z Dialogues with AI Star Unicorn: How Will Large Models Evolve in the Future?
a16z investors discussed the future direction of LLMs with the CEOs of four emerging AI startups.
Navigating Web3 tides with focused insightsInvestors from a16z discussed the future direction of LLMs with the CEOs of four emerging AI startups.
Participants included: Dario Amodei, CEO of Anthropic; Aidan Gomez, CEO of Cohere; Noam Shazeer, CEO of Character.AI; and Yoav Shoham from AI21 Labs.
They summarized four core directions:
-
Addressing the hallucination problem—steering the model effectively
-
Solving personalization through larger and more precise memory solutions
-
Moving from knowledge to reasoning to action—teaching models to use tools
-
Multimodality, enabling models to achieve truly general-purpose capabilities
The Hallucination Challenge: Holding the Steering Wheel
Because large models still suffer from "hallucinations," many entrepreneurs remain cautious about integrating LLMs into products and workflows.
To address this, leading model companies are focusing on improving how they control LLM outputs—a process known as steering. The goal is to better focus model output and help models understand and execute complex user requests more accurately.
Noam Shazeer, CEO of Character.AI, compares LLMs to children: "It's about guiding the model properly. We need the right methods to tell models exactly what we want them to do. Children are similar—they sometimes make things up, lacking a firm grasp on fantasy versus reality."
Tools like Guardrails and LMQL have already emerged as research advances, but work continues. a16z believes progress in steering will be key to unlocking broader LLM productization for developers.
For enterprises, improving steering is critical. Amodei, founder and CEO of Anthropic, notes that unpredictability makes users uneasy. As an API provider, he wants to confidently say to customers: “No, the model won’t do that,” or at least, “It rarely does.”
By refining LLM outputs, developers can gain greater confidence in aligning model performance with customer needs.
Improved steering also benefits industries requiring high accuracy and reliability, such as advertising.
“In legal, medical, financial data storage, risk management, and brand protection scenarios, you absolutely don’t want to rely on technology that’s unpredictable or hard to describe.”
Better steering allows LLMs to handle more complex tasks with fewer prompt engineering efforts, since they’ll better grasp user intent.
More precise control over LLM outputs could also unlock new possibilities for sensitive consumer-facing applications. Users expect personalized, accurate responses.
While users may tolerate inaccuracies when generating creative content or casual conversations, they demand higher accuracy when relying on LLMs for important decisions—or when using them as life coaches, therapists, or doctors.
Whether LLMs can truly replace entrenched internet-era tools like search may ultimately depend on their ability to steer outputs effectively, improve reliability, and build user trust.
The Memory Challenge: The Goal Is Personalization
Contextual capability remains a major bottleneck limiting personalization.
Although prompts and fine-tuning offer some level of customization, prompts are hard to scale, while fine-tuning is costly, requires retraining, and often demands close collaboration with closed-source LLM providers—making it nearly impossible for small teams or individual users.
The holy grail is contextual learning—the ability for LLMs to learn from enterprise-specific content, terminology, or unique contexts—to generate more refined, scenario-specific outputs.
Unlocking this capability requires stronger memory functions in LLMs.
LLM memory consists of two main components: context windows and retrieval.
A context window refers to the text fed into the model alongside its training data, which it uses to process and generate responses.
Retrieval involves pulling relevant information and documents (contextual data) from sources outside the model’s original training corpus.
Currently, most LLMs have limited context windows and lack local retrieval capabilities, resulting in insufficient personalization. However, with larger context windows and improved retrieval mechanisms, LLMs can deliver highly tailored, precise outputs directly suited to individual needs.
Expanding context windows enables models to process longer texts and maintain continuity across extended dialogues.
This significantly enhances the model’s ability to perform tasks requiring deep understanding and long inputs—such as summarizing lengthy documents or generating coherent, context-aware responses in prolonged conversations.
Progress is underway: GPT-4 supports 8k and 32k context windows, while GPT-3.5 and ChatGPT support only 4k and 16k tokens.
Claude recently expanded its context capacity to 100k tokens.
However, simply increasing context length isn’t enough—inference cost and latency grow almost linearly, or even quadratically, with length.
Retrieval mechanisms enhance the model’s base training data by providing the most relevant contextual information. Since LLM knowledge is typically static, retrieval offers two advantages, according to AI21 Labs’ Shoham: “First, it gives access to information not available during training; second, it focuses the model on task-relevant data.”
Vector databases have become the de facto standard for efficient retrieval and serve as the memory layer for LLMs, enabling faster, more accurate searches and correct referencing of vast datasets.
Expanded context windows and robust retrieval systems will prove invaluable in enterprise settings—such as navigating large knowledge bases or complex databases. Companies can leverage proprietary data (internal knowledge, historical customer records, financial results) as direct LLM input without costly fine-tuning.
Enhanced memory capabilities will drive improvements in training, reporting, internal search, data analytics, business intelligence, and customer support—with deeper personalization.
In consumer applications, improved context handling and retrieval will enable powerful personalization, transforming user experiences.
Noam Shazeer believes: “One major breakthrough will be developing high-memory-capacity models capable of personalizing for each user and delivering services at scale cost-effectively. You’d want your therapist to know every aspect of your life; your teacher to understand what you’ve already learned; your life coach to give advice grounded in your reality. All of these require context.”
Aidan Gomez shares this excitement: “By giving models access to personally relevant data—like emails, calendars, or messages—they can understand your relationships and communication patterns, helping you far more effectively within that context.”
From Knowledge to Action: Teaching Models to Use Tools
The true power of large models lies in using natural language as a medium for action.
LLMs possess sophisticated understanding of common, well-documented systems—but cannot execute actions based on that knowledge.
For example, OpenAI’s ChatGPT, Anthropic’s Claude, and Character AI’s Lily can describe flight booking procedures in detail—but cannot book flights themselves (though plugins like those in ChatGPT are beginning to solve this).
Amodei explains: “In theory, the model has a brain full of knowledge, but lacks the mapping from conceptual instructions (names) to actual execution steps (pressing buttons). Connecting different components doesn’t require much training. The LLM is like a brain without a body—it understands operations theoretically, but lacks hands and feet to act.”
We’re already seeing rapid progress in tool usage. Established players like Bing and Google, along with startups like Perplexity and You.com, have integrated search APIs. AI21 Labs launched Jurassic-X, combining models with predefined tools (calculator, weather API, Wikipedia API, database) to overcome standalone LLM limitations.
OpenAI introduced plugins allowing ChatGPT to interact with tools like Expedia, OpenTable, Wolfram, Instacart, Speak, web browsers, and code interpreters—an innovation likened to Apple’s “App Store moment.” Recently, OpenAI added function calling to GPT-3.5 and GPT-4, enabling developers to link GPT capabilities with any external tool.
By shifting focus from knowledge extraction to action-oriented behavior, LLMs can gain “hands” and “feet,” opening up countless new application scenarios across businesses and user types.
For consumers, LLMs may soon suggest recipes and then order ingredients, or recommend brunch spots and reserve tables.
For enterprises, founders can make their apps dramatically easier to use via LLM integration.
As Amodei notes: “Functions that are extremely difficult to use via UI could become accessible just by describing them in natural language.”
For instance, integrating LLMs into platforms like Salesforce could let users update CRM records using natural language, with the model automatically making backend changes—drastically reducing maintenance time. Startups like Cohere and Adept are working on embedding LLMs into such complex tools.
Gomez believes that within two years, LLMs might be able to use applications like Excel—but significant improvements are still needed.
“We’ll get first-generation models that can use tools—exciting yet fragile. Eventually, we’ll reach ideal systems where you can hand any software to the model, describe its functions and usage, and the model will operate it. Once we equip LLMs with both specific and general-purpose tools, the automation they unlock will become star products in their domains.”
Multimodality: Language Models Aren’t Truly General-Purpose
While chat interfaces feel intuitive to many users, people speak and listen more frequently than they read and write.
As Amodei points out: “AI systems are limited because not everything is text.”
Models with multimodal capabilities—able to seamlessly process and generate audio, visual, and other formats—can elevate interaction beyond text.
Models like GPT-4, Character.AI, and Meta’s ImageBind can already handle images, audio, and other modalities—but quality remains basic (though steadily improving).
As Gomez says: “Our models still fall short in directly processing visual information—this needs improvement. We’ve built many GUIs assuming users can see them.”
As LLMs evolve, multimodal capabilities will deepen—both in understanding and interaction. They’ll be able to navigate GUI-dependent apps like browsers. They’ll offer richer, more connected, comprehensive experiences, enabling interactions beyond chat boxes.
Shazeer notes: “Integrating multimodal models makes experiences more engaging and closely tied to users.” He adds: “I believe core intelligence still comes mainly from text, but audio and video make experiences more compelling.” From video chats with AI tutors to co-writing screenplays with AI, multimodal tech holds transformative potential across entertainment, education, development, and content creation—for both consumers and enterprises.
Multimodality also ties closely to tool use. While LLMs initially connect to external software via APIs, multimodal capabilities will allow them to use human-designed tools without custom integrations—such as traditional ERP systems, desktop apps, medical devices, or manufacturing machinery.
Here, promising progress is already visible: Google’s Med-PaLM-2 can analyze mammograms and X-rays. And in the long run, especially when combined with computer vision, multimodality could extend LLMs into the physical world via robotics, autonomous vehicles, and other real-time physical interaction applications.
Despite current limitations, researchers have made astonishing improvements in a short time—so much so that this article required multiple updates during writing, reflecting the rapid pace of advancement.
Gomez agrees: “A 1-in-20 chance of hallucination is clearly too high. But I’m very confident—this is the first time we’ve built such systems. Expectations are sky-high, shifting from ‘computers only doing math’ to ‘outperforming humans.’ We’ve narrowed the human-machine gap so much that criticism now centers on whether computers can match human-level performance.”
We’re especially excited about these four innovations, which are poised to transform how entrepreneurs build products and run companies. In the long term, the potential is even greater.
Amodei predicts: “At some point, we might have a model that finds a cancer cure by analyzing all biological data.”
In reality, the best new applications remain unknown.
At Character.AI, Shazeer empowers users to discover these use cases: “We’ll see many new applications emerge. I can’t predict exactly what they’ll be. There will be thousands, and most users—not just a few engineers—will be better at discovering how to harness this technology.”
We eagerly anticipate the profound impact these advancements will have on how we live and work, empowering entrepreneurs and companies with powerful new tools and capabilities.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
