OpenAI also lacks GPUs, and reducing costs is the top priority
TechFlow Selected TechFlow Selected
OpenAI also lacks GPUs, and reducing costs is the top priority
$1.2 billion + nearly all its computing power: Microsoft puts half its future in OpenAI's hands
Navigating Web3 tides with focused insightsAuthor | Ling Zijun
Editor | Wei Shijie
Sam Altman’s European tour is still ongoing. Recently in London, he held a private discussion with Raza Habib, CEO of AI company HumanLoop. HumanLoop is a company that helps developers build applications on large language models.
Raza Habib documented key points from the conversation and published them on the company's official website. However, under OpenAI’s request, the summary was later taken down. This only heightened external curiosity about the discussion. Some speculate that certain ideas mentioned regarding OpenAI may have since changed.
After reviewing the deleted transcript, GeekPark found it not only reveals Sam’s short-term vision for OpenAI but also uncovers the pressures OpenAI faces despite strong cloud computing support from Microsoft.
After all, model fine-tuning and inference continue to consume massive computational resources.
According to The Information, OpenAI’s models have already consumed $1.2 billion worth of Microsoft Azure compute capacity, concentrating computing resources on OpenAI has limited server availability for other Microsoft divisions.
On this point, Sam stated that reducing costs is currently the top priority.
Additionally, Sam revealed that services such as offering longer context windows and providing fine-tuning APIs are currently constrained by GPU resource limitations;
In this dialogue, Sam Altman addressed many externally raised questions, including competition and commercialization:
-
Despite recently hiring world-class product manager Peter Deng, OpenAI does not plan to launch more products;
-
The future trend will be embedding large model capabilities into more apps, rather than growing numerous plugins within ChatGPT, because most existing plugins have not demonstrated PMF (Product/Market Fit);
-
Over the past few years, OpenAI scaled its models at millions-of-times speed, but this pace isn’t sustainable. Moving forward, OpenAI will continue to increase model scale by 1x to 3x annually to improve performance.
The conversation transcript was originally published on May 29 and deleted around June 3, according to online records. Below is the content obtained via backup:
OpenAI Is Currently Severely Constrained by GPU Availability
As conversations grow longer, required computational resources increase exponentially
Currently, OpenAI faces significant GPU shortages, delaying many of its short-term plans. The biggest customer complaints concern API reliability and speed. Sam acknowledged these concerns and explained that most issues stem from GPU scarcity.
The longer 32k context cannot yet be rolled out to more people. OpenAI hasn't overcome the O(n^2) scaling of attention mechanisms, so while 100k–1M token context windows seem plausible soon (this year), anything larger would require a research breakthrough.
Longer 32K contexts still can't be offered to more users. OpenAI has not yet overcome the O(n^2) scaling issue in attention mechanisms; although achieving 100k–1M token context windows appears feasible in the near term (this year), any further expansion would require a research breakthrough.
Note: O(n^2) means that as sequence length increases, computational resources required for Attention operations grow quadratically. "O" describes the upper bound or worst-case growth rate of algorithmic time or space complexity; (n^2) indicates complexity proportional to the square of input size.
Fine-tuning APIs are also currently limited by GPU availability. They haven’t adopted efficient fine-tuning methods like Adapters or LoRa, making running and managing fine-tuned models highly computationally intensive. Better fine-tuning support is planned for the future. They might even host a community-driven model contribution marketplace.
Dedicated capacity offerings are constrained by GPU availability. OpenAI offers dedicated capacity—private copies of models for customers. Accessing this service requires customers to commit upfront payments of $100,000.
OpenAI’s Near-Term Roadmap
2023: Reduce the cost of intelligence;2024: Limited multimodal demonstrations.
Sam also shared his view of OpenAI’s temporary near-term API roadmap.
2023:
-
Cheaper and faster GPT-4 — This is their top priority. Overall, OpenAI aims to minimize the “cost of intelligence,” working continuously to reduce API costs over time.
-
Longer context windows — Context lengths could reach up to 1 million tokens in the near future.
-
Fine-tuning API — Fine-tuning APIs will extend to newer models, though exact formats depend on what developers express they truly want.
-
A stateful API — Currently, calling the chat API requires repeatedly sending the same conversation history and paying for the same tokens each time. A future version will remember session history.
2024:
Multimodal capabilities — Demonstrated as part of a GPT-4 release, but won’t scale to everyone until more GPUs come online.
Commercial Predictions and Reflections:
Plugins lack PMF and likely won’t appear in the API soon.
Many developers are interested in accessing ChatGPT plugins via API, but Sam said he doesn’t expect these to be released anytime soon. Aside from browsing plugins, usage data shows no clear PMF (Product/Market Fit). He pointed out that while many believe they want their apps inside ChatGPT, what they really desire is having ChatGPT embedded within their own applications.
Beyond ChatGPT, OpenAI Will Avoid Competing With Its Customers
Great companies have one killer app.
Many developers expressed nervousness about building with OpenAI APIs, fearing OpenAI might eventually release competing products. Sam said OpenAI won’t launch additional products beyond ChatGPT. He noted historically great platform companies had one flagship application. ChatGPT will allow developers to become direct users of their own products and help refine the API. While ChatGPT’s vision is to become a super-intelligent work assistant, OpenAI won’t pursue many other GPT use cases.
Regulation Is Needed, But Not Now
“I’m skeptical about how many individuals and companiesare capable of hosting large models”
While Sam calls for regulation of future models, he believes current models aren’t dangerous and thinks regulating or banning them would be a major mistake. He reiterated the importance of open source and said OpenAI is considering open-sourcing GPT-3. One reason they haven’t done so yet is skepticism about how many individuals and organizations are capable of hosting and serving large language models (LLMs).
Scaling Laws Still Hold
Years of scaling up by millions of timescan’t continue indefinitely.
Recently, many articles have claimed “the era of giant AI models is over.” That’s inaccurate. (Note: At an MIT event in April, Sam Altman said: “We’re now approaching the end of the era of giant models.”)
Internal data at OpenAI shows scaling laws for model performance still apply—increasing model size continues to boost performance.
Given that OpenAI scaled its models by millions of times within just a few years, such rapid expansion isn’t sustainable. This doesn’t mean OpenAI will stop trying to build larger models—it means annual scale increases may be limited to 1x–3x, rather than orders of magnitude.
The validity of scaling laws has important implications for AGI development timelines. The assumption behind scaling laws is that we may already possess most components needed to build AGI, and the remaining task mainly involves scaling existing methods to larger models and datasets. If the age of scaling were truly over, AGI might be much further away. The continued applicability of scaling laws strongly suggests shorter timelines ahead.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
极客公园
