Interview with SemiAnalysis Founder Dylan Patel: Memory Shortage Will Continue, CPO Two Years Behind Market Expectations
TechFlow Selected TechFlow Selected
Interview with SemiAnalysis Founder Dylan Patel: Memory Shortage Will Continue, CPO Two Years Behind Market Expectations
A timeline for AI hardware investors from a semiconductor analysis expert.
Navigating Web3 tides with focused insightsOrganized & Compiled: TechFlow
Program: The Next Big Thing (Produced by WisdomTree)
Guest: Dylan Patel, Founder of SemiAnalysis
Host: WisdomTree Host + Klay Hyman
Release Date: July 9, 2026
Summary
SemiAnalysis founder Dylan Patel joins WisdomTree's The Next Big Thing podcast, sorting through the latest status of the AI infrastructure supply chain with the host and Klay Hyman. Topics cover SemiAnalysis's journey from a motel startup to a 90-person team, the behind-the-scenes of Jensen Huang naming "sandbagging" at GTC, AI ROI, AI cost optimization strategies (switching to the latest models is actually cheaper), the memory supercycle (KV cache explosion, capacity bottlenecks, smartphones being squeezed out), CPU demand inflection point, networking and CPO timeline (delayed to 2028-2029, copper still dominates), power and energy infrastructure, and the conversion supply chain.
Key Quotes
- "Jensen, I was wrong, you were hiding your strength, it's actually 30x." (Dylan emailing Jensen Huang admitting Blackwell performance exceeded expectations)
- "We spend more than one-third of employee compensation on AI, by the end of the year it might be half." (Dylan on SemiAnalysis's AI ROI)
- "Memory capacity will only grow 20-30% annually over the next three years, but demand is doubling, doubling." (Dylan on the memory supercycle)
- "Take a diesel truck engine, convert it to natural gas, reverse-connect an electric motor, put it behind the data center, then hire a bunch of people from auto repair shops to maintain it." (Dylan on behind-the-meter power generation innovation)
- "You can retrofit truck engines, hire a bunch of mechanics, and operate a site like that. All the way to 'I'm going to launch it into space.' There are solutions to data center problems, whether you go the full dirty route or the space route." (Dylan on the crazy spectrum of power solutions)
- "Cost optimization is actually switching to the latest models. Because the latest models might only need a quarter of the tokens to complete the same task." (Dylan on the AI cost paradox)
Chapter 1: Opening Introduction
Host: Okay, hello everyone, welcome back to the next episode of The Next Big Thing podcast. Today along with my colleague Klay Hyman, we have Dylan Patel, our new partner, founder of SemiAnalysis Research Group. Very excited to sort through the latest status in the AI infrastructure field with Dylan today. You may have heard Dylan's shares on many different podcasts, I've been following his content, and the newsletter on the SemiAnalysis website. Recently I saw an article about space data centers, if anyone is interested in this topic, they have a very detailed long article. But Dylan, I'd love to hear how the idea for SemiAnalysis originally came about. I know in the Substack community, everyone has been discussing your company's revenue and success recently, but often people only see the current success, forgetting the journey and the starting point, forgetting how much effort was invested.
Chapter 2: SemiAnalysis Startup Story
Dylan: Okay. I think the origin of SemiAnalysis actually comes from "posting online". Posting online in a not-so-serious way. I recall my earliest posts about semiconductors were when I was in my early teens. At that time I was posting online about chips, smartphones, phone screens, phone SoCs, and things like that. Before I owned a smartphone myself, I was obsessed with them. Game hardware too, PC hardware, console hardware, I kept posting and discussing these things on various forums. By the time I was 12, I was already managing and creating many forums, covering areas like Android, Apple, Google, Intel, Nvidia, AMD these hardware topics, and various related forums on Reddit.
Dylan: That's where everything started. I've always been a poster, always posting my views, always replying, always thinking, always accepting comments. Now we are a 90-person organization, people on my team doing marketing will tell me: "Dylan, stop replying to those boring people online, you make us look bad." But I have this impulse, whoever criticizes me online, I want to respond. Maybe this is a bad thing, but basically throughout my entire teenage years, I was managing these forums.
Dylan: I started investing after I began making money in my late teens. I did quantitative trading for two years, then started my own company. But throughout the whole process I kept posting posting posting posting. I had anonymous blogs, anonymous posts. By 2020, I was a bit tired of work. The disillusionment of doing quant, you know, not as glamorous as it looks. Yes, you can make money, but not that amazing. I pretty much quit and started a business. Wasn't sure what would happen at the time, but I posted under my real name on a WordPress site, content was a mix of technology, business, finance, supply chain, these were the areas I was most interested in.
Dylan: I grew up in a small business environment. I grew up in a motel, my parents had a motel in rural Georgia, we lived inside it. I understood business from a young age. Later we opened a gas station, so always grew up immersed in business. I've always liked business. Supply chain has always been interesting from an investment perspective and from a product manufacturing perspective. I've always had an intuition about how things are made. Technology方面 of course super exciting, finance方面也 super exciting.
Dylan: Combining these, my earliest posts, at that time the US was banning Huawei from accessing TSMC's foundry services. My first post was actually about how MediaTek became the biggest winner. Huawei was number one in Chinese smartphone chips and smartphone market share at the time. Obviously this would decline significantly, because they could no longer use TSMC. The US market thought Qualcomm would win. But MediaTek, this Taiwanese company, I thought they would win more share, because from a geopolitical perspective, China would rather buy from a Taiwanese company than a US company,毕竟 we just banned Huawei. Both companies benefited, but MediaTek benefited more. This is an example of technology, supply chain, finance, geopolitics these elements mixed together.
Dylan: In the following years, I turned WordPress into Substack, at some point started charging, writing topics covering the entire semiconductor and AI supply chain. I followed AI during the four years I did quant, following semiconductors out of passion. Then just kept growing, growing, growing.
Dylan: Over four years I ran around the world, attended every conference in the world. 40 conferences a year. I didn't have a fixed residence, just went to every conference I could, whether AI conferences like NeurIPS, ICML, ICLR these mainly researcher conferences, or downstream some very niche area conferences, like semiconductor supply chain chemical raw material conferences. I ran the entire technology stack from top to bottom, servers, networking, wafer fabrication, AI, looked at everything. 40 conferences a year. Some conferences very niche, only 300 people attending, except for about 5 people everyone spoke Japanese, I just thought "okay, so be it". There were also conferences with 10,000 to 20,000 people attending, huge scale. The entire spectrum and continuum.
Dylan: This way I could cover the entire ecosystem. You go to a conference three times, you truly understand the language of this field. You know the people there, can ask them questions, build up these networks. I developed an entire ecosystem and cognitive system, covering inflection points in every link. I was very curious technically, but once something in technology or supply chain stood out from conferences, I knew what consequences it would cause at the supply chain or finance level. Sometimes reports were technology-centric, finance circles didn't care. But sometimes everyone suddenly realized this is a bottleneck, or this is an inflection point, or this company will gain a lot of market share because of next-generation technology. I would point it out before anyone on Wall Street, before any hedge fund, before anyone.
Dylan: That was the starting point. Then as Substack got bigger, in 2022 I started hiring people. The first two employees were people I knew on Discord for several years. The third employee was Myin, he previously worked at a hedge fund, preparing to move to Japan to live with his wife, so basically free. I posted an article at the time, interesting article, early 2023, saying memory was the biggest loser in AI. The reason was AI chips and AI servers use much less memory proportionally than ordinary servers. Ordinary servers about half the BOM is memory, but in AI servers memory proportion is much less. Partly because Nvidia's profit margins are much higher, and other factors. Of course Nvidia's next-generation chips significantly increased memory capacity, now much more. But at that time I said memory was the biggest loser.
Dylan: In the paid section I said "I'm hiring". Myin contacted me, he was the first person from a hedge fund background, the other two were technology background. Once he joined the company, we started building various models, truly transforming the business from a newsletter model to selling information services, selling reports and datasets model. As these started happening, the snowball started rolling down the hill. 2023 to 2024, grew from 2 people to 7 people. End of 2024 to early 2025, grew from 7 to 20. 2025 to 2026, grew from 20 to 60. Now this year we reached 90 people. Added 30 people this year already. Just a snowball rolling down the hill rhythm.
Dylan: We just kept adding new areas. I've always been interested in everything, but now I can hire real experts. I think the most exciting thing about SemiAnalysis is, I don't know any other company with our level of expertise and concentration. I have people who worked at ASML, Applied Materials, Lam Research, these are equipment companies manufacturing wafers. Upstream there are people who worked at Intel, TSMC, Nvidia, Microsoft, Amazon. Also people who worked on models at OpenAI, worked on FSD at Tesla, worked at Coherent. We have people doing model layer, also people doing data centers. Someone in my company actually built a power plant in Kazakhstan. We just have this crazy talent density.
Dylan: Half the company are people from engineering backgrounds, the other half are either ex-hedge fund, or super passionate netizens I discovered on Twitter or Discord. I think you're smart, come work for me. This really works. Now SemiAnalysis has many business lines: data services, consulting, information services, newsletter, we are also doing various media content, soon holding a large conference. Various different businesses. This journey has been too exciting.
Chapter 3: GTC Sandbagging Moment
Host: Speaking of this journey, Dylan, WisdomTree and SemiAnalysis have been cooperating for several months. Nvidia GTC was held in March, held every year. I was watching the live stream in Charlotte, North Carolina at the time. About 55,000 people watching the live stream simultaneously.
Dylan: There were another 20,000 people in the stadium. 20,000 people in the stadium, man.
Host: Then he directly mentioned you, saying you were like "sandbagging" a number, saying he was hiding his strength, your chart appeared directly on the stage big screen. Honestly I had a moment at the time, watching the CEO of the world's largest company basically quoting your research, saying you criticized some of his numbers. I'd love to hear, sounds like you were in the stadium at the time.
Dylan: Yes, that moment was very surreal. One thing SemiAnalysis does is we have a batch of engineers, we do open source benchmarking on all open source AI models and all hardware. This is a great project. We have a batch of engineers on this side, also collaborate extensively with the industry. We received hardware donations worth over 50 million USD, from OpenAI, Microsoft, Amazon, Google, CoreWeave, Nebius, Crusoe, Oracle etc all the major cloud vendors you can think of. We run benchmarks on this hardware.
Dylan: We have 8 different GPUs, H100, H200, Blackwell, AMD's various GPUs. Also Google's TPU and Amazon's Trainium. What we do is run benchmarks on the latest version of software every day. Why run every day? Because every night maybe new CUDA version releases, PyTorch version, driver updates, inference engine vLLM or SGLang version etc. We run these benchmarks across the entire curve every night, testing how fast token generation speed you want versus how high cost efficiency you want, and optimal scenarios. All automated running.
Dylan: When Jensen initially released Blackwell he claimed there would be a 25x improvement. No one believed him at the time, right? This is Jensen嘛, he's doing marketing. Even we at the time thought, okay. We were more optimistic than others, we thought based on simulation maybe 15 to 20x improvement, because we have performance simulators. But as we built this inference benchmarking platform called Inference-X, we got actual results: on DeepSeek V3, Blackwell was 30x faster than Hopper at some point.
Dylan: Then I had this result and emailed him. These results automatically publish to open source GitHub, this is an open source collaboration project, Nvidia people also participated, they know. But I specifically told Jensen, I emailed him saying: "Hey Jensen, when you released Blackwell in 2024 you said 25x, everyone roasted you. Even I roasted you. I said impossible 25x, at most 15 to 20x. Many people said no no no, only 3x. We were considered very optimistic. But Jensen, I was wrong, you were hiding your strength, actually it's 30x."
Dylan: He made an article out of this. I didn't know what he would do with this. I heard from several clients, someone at Meta told me they had a meeting, Jensen used this as evidence saying he doesn't hide numbers. He used this when talking about next-generation chips. Then all this happened, I didn't expect it to happen on stage.
Dylan: Additionally in the Inference-X project, we made a belt. Looks like a WWE championship belt, written on it "Inference King". We mailed it to all partners, mailed to Nvidia, AMD, also SG Lang, vLLM these people who helped us do benchmarking, those who donated hardware. Because this is an open source project, I spend several million USD annually on engineer salaries, others spend millions on hardware donations, or spend millions on engineer salaries donations.
Dylan: I mailed this belt to them, then this belt appeared on his slide. He held it up to show, then our chart was also on it, he spent five minutes on the slide talking "Dylan said I was sandbagging, but I wasn't", talking our performance is the best. That was truly a surreal moment.
Host: He talked about us for longer than anyone else during the entire speech. The only one talked about for差不多 time was OpenClaw, that obviously is sweeping the entire world.
Host: This was an incredible moment.
Chapter 4: AI ROI and Corporate Spending
Klay: Dylan, you mentioned a few things. You mentioned open source, now maybe can shift to some recent developments and market topics. There has always been discussion about open source models versus closed source models actual inference efficiency. And until today many investors are still questioning AI ROI. Just in the last week or two, there were Bloomberg economists discussing many AI projects might be failing at some companies. I know you mentioned your company is using AI heavily, giving employees large token access permissions. You are also hiring. So I'm curious about your view on terminal demand, and whether terminal demand is really driving the large-scale construction we see. This construction is full of various constraints, at least in the past month, except for recent days market volatility, has been pushing up various stocks related to these themes.
Dylan: Okay, I'll say a few points. When you look at this big question, about ROI, about whether companies are making enough money from AI, whether this will continue, whether people using AI are truly gaining value from it, this is a big question many people are asking.
Dylan: When I look at this question, there are several ways to break it down. First, Anthropic is already free cash flow positive, and was profitable in Q2. Even in April, April's accounts are closed, they were profitable, free cash flow positive. May was also free cash flow positive, profitable. June looks like it will be the same, although not fully closed yet, but at least two out of three months were free cash flow positive and profitable. Their recurring revenue has skyrocketed to over 50 billion USD ARR. They are doing very well.
Dylan: This is one side. Anthropic is printing money. Of course many companies haven't printed money yet, but are heading in that direction. OpenAI's revenue also started showing an inflection point with Codex adoption growth. These companies are all becoming more profitable. Anthropic's gross margin is very high, over 70%.
Dylan: The other side is company spending on AI. At least at SemiAnalysis, we look at annual recurring spend, I like to call it ARS, Annual Recurring Spend, not ARR. Last November, December, before Claude Code really started taking off, our annual recurring spend was less than 100,000 USD. At that time we subscribed every employee to ChatGPT 200 USD package, if someone wanted xAI or Claude, we gave that too. But the standard was giving everyone OpenAI 200 USD subscription. November was this state, at that time I thought we were already on the frontier.
Dylan: But Claude Code started reaching an inflection point with Claude Opus 4.5 and 4.6 etc versions. By end of January, our ARS reached 4 million USD. Because everyone was using Claude Code. Now it's about 11 million USD. At the highest, if we take one week's spend multiplied by 52, reached 11 million, highest once reached 14 million. It fluctuates significantly depending on what work everyone is doing. But the current average looks like about 1 million USD annual spend, for a 90-person company. This is too crazy.
Dylan: Our spending on AI exceeds one-third of employee compensation, by year end might reach half, depending on Methos and other models becoming better and better. This is a huge expenditure. The question is what is the ROI? I think the ROI is very large, because we can develop products, can sell more, can improve everyone's efficiency. I see the ROI, but many companies are questioning: if I have a good developer with annual salary 300,000 USD or more, their spending on AI starts approaching one-to-one. This is true for good developers, for non-developers spending range will be lower, but at SemiAnalysis at least, many of our biggest AI spenders are people who can't write code, they just tell the model what they want, then iterate repeatedly, until they get the desired result.
Dylan: You see AI spending per employee skyrocketing. Many companies now reasonably asking: our full year AI budget was used up in Q1, Q2, now what? Cut spending or cut elsewhere? Many companies saying maybe need to slow down AI spending. But I see many companies starting to cut elsewhere. They are cutting SaaS products used before. They are saying "we can grow faster, so just like this". They are saying "spending money on AI is okay, we temporarily bear this cost. AI will become cheaper". As adoption rates rise, what I did with AI six months ago, today AI does much cheaper. Of course, what I do with AI today is much broader than six months ago.
Dylan: Some people even cut employees without cutting AI. Some tightened AI spending, but these companies will be left behind on productivity improvements and product development capabilities.
Chapter 5: AI Cost Optimization Strategies
Klay: Then one way to reduce incremental costs is choosing cheaper, maybe less intelligent models, not always on the frontier. Now I feel it's still early, about these discussions. But I'm curious, whether companies like yours have a node, deciding certain use cases are more suitable for models like DeepSeek V4, while work needing more intelligence uses more expensive models. Is this part of the calculation?
Dylan: I think for some people this is definitely part of the calculation. You need to divide AI workloads into two categories. First category is AI integrated into a process. For example customer sends me a document, I check XYZ, put document into model, model checks, done. In this case I only need to reach a certain quality level, then can stop improving models, start reducing costs by waiting for updated models, cheaper models or higher cost-performance solutions. We see AI models improving in cost at about 60x speed annually. You take a quality level, one year later it's 60x cheaper.
Dylan: People made a big fuss about DeepSeek at the time, because it was 600x cheaper than GPT-4. Actually that was two years after GPT-4 release, so 60x multiplied by 60x is 360x, actual result was 600x. So at some point on the curve, whether annually 60x cheaper or 90x, roughly in this range. If you have a workflow, integrate AI into it, reach quality level then go use cheaper ones.
Dylan: Second category is AI assistants. Here actually there is a misconception. If you let the model help you do this, help you find that, help you figure out this in daily work, cost optimization is actually not switching to cheaper models. Cost optimization is often switching to the latest models. Because the latest models, for example Claude 4.6 Opus might need 100,000 tokens to complete a task, might need several rounds of conversation, 100,000 tokens, 10 minutes time. But Claude 4.8 Opus can complete with a quarter of the tokens, 25,000 tokens, and might only need one round. Cost is actually lower, because generated token count is less, time you spend is also less.
Dylan: So when I look at a developer or someone doing intellectual work, how do I reduce costs? Actually should use newer models, take a task that previously needed back-and-forth entanglement with the model to complete, use newer and newer models, now either complete in one iteration, or directly handle the entire workflow in one go. Fewer tokens.
Dylan: We saw when upgrading from 4.6 Opus to 4.7 Opus, my costs actually dropped for a week first, then skyrocketed back, because everyone used more and more. Why go back up? Because everyone adjusted to new workflows: work I did before is completed, let me do more. Same from 4.7 to 4.8, costs dropped for a week to a week and a half first, then skyrocketed, because everyone discovered "oh, I can do more work now".
Dylan: You must measure productivity and costs together. When it's an AI assistant, token efficiency is very important. This is why Anthropic has been beating OpenAI, because their models are higher in token efficiency than OpenAI. Actually OpenAI's models in extreme cases, in frontier science, frontier mathematics, frontier code aspects, sometimes can complete tasks Anthropic models can't. But they need 3x time and 4x tokens, therefore costs are higher, and human and AI feedback loop isn't that fast.
Dylan: Ultimately perceived worse by customers. Because there is a situation where you say "hey model do this task", then you come back to see if task is completed. Another situation is you say "I have four hours to do this task", whether one call letting model work four hours, or four calls back-and-forth interaction. Turns out Anthropic is much faster, much better in human-in-the-loop feedback loops, because token efficiency is higher. This is why we are still a team mainly using Anthropic. Some tasks everyone uses OpenAI, usually those letting it run overnight, hand over to OpenAI's Codex. But most tasks still use Claude Code.
Dylan: This is an interesting factor about models and token efficiency. Costs are a bit hard to completely isolate to look at, some tasks you need to freeze model quality then wait for models to become cheaper, some tasks you just want the smartest model because that's actually cheaper.
Chapter 6: Memory Supercycle
Klay: Dylan, I want to hear your views on hardware aspects. Earlier this year there was an article about memory in the newsletter. Memory is usually a cyclical product, maybe 18 to 24 months up, 18 to 24 months down. Now feels like almost everything is in shortage. If you are a supplier of some data center component, feels like the problem has changed from "can you get goods" to "how long do you have to wait". Because in today's world you almost can't get any components. With your experience in hardware, what do you think will happen to memory, this product that has been commoditized for the past 40 years? Before it was just ride the up cycle, endure the down cycle, cycle repeats.
Dylan: Okay. I'm not saying there won't be cycles in the future. Cycles will still happen. Obviously we are in a supercycle, upside is very crazy, there will also be downside, downside will be very cruel. But from trough to trough, there is still huge growth. About memory and other components, important is the phase change happening.
Dylan: Historically, up cycle terminal markets might rise 50%, for memory this kind of commodity market, pricing elasticity is larger, stocks might rise 2 to 3 times. But now rising is not 50% anymore, total spending has doubled in past few years, and will double again. Total spending doubled, when you look at different terminal market elasticity, memory pricing has already risen 4 times, will rise another 2 to 3 times, plus capacity growth. So stocks skyrocket then will fall back.
Dylan: What's truly exciting about memory, not just terminal markets skyrocketing, not just pricing elasticity. What's truly interesting is an article we wrote in 2024, at that time o1 came out. OpenAI released o1, this was the first reasoning model, created a wave of reasoning model boom, OpenAI, Anthropic, DeepSeek etc many companies are utilizing this direction, letting models do long-duration agent tasks.
Dylan: When o1 came out what we noticed immediately was workloads changed hugely. When doing chat, you send a prompt, maybe 50 words or maybe 500 words, model gives you a reply, context length might be a few thousand. For example 2000 context length. When doing reasoning, for every token generated, you have to read all weights into the chip, read all context into the chip, process one token, then iterate. Context is KV cache, it creates relationships between all tokens.
Dylan: Interestingly, on the weights side, whether context length is 1000 or 100,000, you have to read all weights. So reasoning memory intensity on the weights side is the same. But on the KV cache side, you read 1000 tokens and read 100,000 tokens, memory difference is huge, although computation is roughly the same. Computation can be controlled because of KV cache caching etc reasons, but memory costs skyrocket.
Dylan: In our o1 article, December 2024 discussed scaling laws, pretraining scaling laws how to give way to inference scaling laws, o1 was a step change. We talked about how KV cache explodes due to reasoning, therefore memory will be the biggest winner. Very bullish on memory multiple times in 2025. But truly in that note we wrote in January 2026, at that time people said memory already rose 50%, is it at the top? What we wrote was basically: no no no, I think you guys don't get it. Memory capacity will only grow 20-30% annually over the next three years, but demand is doubling, doubling.
Dylan: So what will happen? Memory prices will continue to skyrocket. Memory users less sensitive to price elasticity, less able to adapt to price fluctuations will exit the market. Smartphones, laptops, because costs skyrocketed too much, will exit the market, make way for AI. Prices must continue to rise, until reaching equilibrium, because capacity growth isn't fast enough.
Dylan: Our argument is: memory shortage will last several years, not a short-term phenomenon. Past Q1 remaining time and Q2, memory indeed kept skyrocketing. There were a few days dropped 7-8% because of some random reasons, but overall trend has been going up and to the right. We look to the future, prices will continue to rise, because we haven't seen high-end markets affected yet. Some Chinese mid-to-low end phone manufacturers like Xiaomi said shipments dropped 40%, but high-end markets haven't been affected yet. Next year iPhone prices must rise. Next year MacBook prices must rise.
Dylan: Currently if MacBook or iPhone rises 100 USD, market won't adjust too much. But memory will become more and more expensive, until AI is full. This means smartphone prices won't just rise 100 USD, have to rise several hundred USD. At some point will reach equilibrium, AI gets needed memory, mobile and consumer hardware squeezed enough. But people still need new phones and new laptops, so will still buy. We have to reach a new equilibrium point, because memory capacity growth isn't fast enough.
Dylan: When we expand vision to the entire ecosystem, what's truly important is many different components are in shortage. Whose pricing has elasticity, who doesn't? For example TSMC pricing has no elasticity, they are a very reliable company, very fair to customers, long-term cooperation. They raise prices just 5-10%. Memory companies are commodity markets, they let spot market and contract market supply and demand balance decide prices. So you will see these two three dimensional pricing differences.
Dylan: One day you will see pricing halved, because memory shouldn't have 85% gross margin. Although currently heading in that direction, we haven't reached 85-90% memory gross margin yet, but will. Then at some point will also fall back to 70 or even lower. TSMC doesn't have much of this volatility. ASML doesn't have much volatility in pricing either, they do equipment. But supply chain different links will fluctuate differently depending on how much AI terminal demand flows to them. For every 1 USD spent on AI, this product might only account for 1 cent, that product might account for 5 cents.
Dylan: So different terminal markets will benefit to different degrees. Plus market structure differences: is it monopoly or oligopoly? Is it fiercely competitive large market? Is pricing stable and has long-term agreements? Or commodity market relying on supply and demand pricing? These factors together determine a certain terminal market, whether memory or now people talking about MLCC shortage, PCB drill bit shortage, copper foil shortage, various random components, you will see online "this is the next shortage" claims. Important is actually how much demand flows here. Terminal market is doubling? Rising 50%? Or quadrupling? How much will pricing rise? These are the truly decisive factors in infrastructure supply chain.
Chapter 7: CPU Demand Inflection Point
Klay: If using that framework you just mentioned, because every year the market wakes up to some new so-called "shortage" you talk about. Earlier this year OpenClaw went viral on various websites, waking people up to the world of AI agents and all possibilities. Using the framework you just described, I'm curious about your view on the CPU market. First three years of AI I didn't hear the word "CPU", this year everyone is talking about CPU.
Dylan: Yes yes. About CPU, interesting is in our research for institutional clients, last November we started discussing this heavily. Because OpenAI and Anthropic started signing agreements with Amazon, Google, Microsoft etc companies, buying all CPUs in their fleets to rent. From end of last year to this year, CPU demand has been showing an inflection point.
Dylan: First say the reason. AI initially in training and inference phases, inference was mainly short context, mainly relying on computation and networking. But as pretraining shifts to reinforcement learning, as chat-style reasoning becomes agentic workflows, CPU demand increases significantly.
Dylan: Why? Pretraining is training the entire internet dataset into the model. Reinforcement learning is the model generates some synthetic data or reasoning trajectories, then verifies in an environment. This environment might be running code unit tests, might be a sandbox simulating a website, might be simulating an engineering system or other platform. Whether a website, shopping website or compiling code, these environments need lots of CPU. Whereas previously during pretraining, token processing itself didn't need much CPU, needed was environment checking.
Dylan: I generated these tokens, now are they valid in Python or C compiler? On an e-commerce website if I want to buy something, as agentic workflow, I constantly test these things, this needs lots of CPU. The other side is real-time inference. Previously doing chat, I tell it one thing, it gives me answer, done. I might ask a few more questions, just like that. But now in agentic workflows, models are doing tool calls: I go search this, I go query that in database, I go let Python interpreter run, I write a small piece of code to check my work, I write code then compile deploy. These agentic processes need more and more CPU, because they must interact with the real world.
Dylan: Previously was human interacting with model: I tell model what, model gives me reply, I look at it, copy paste to where needed. Now is model interacting with internet world, more computation in the loop, more AI, more CPU passing answers back and forth. So whether reinforcement learning or agentic workflows, both need lots of CPU.
Dylan: Now what happened? We need lots of CPU, but let's use the framework just now to evaluate. What's the market structure? Market has Intel and AMD, ARM now also released CPUs, ARM stock therefore skyrocketed, because they are new entrants looking competitive. Amazon is leader, Microsoft and Google also releasing self-developed CPUs. Nvidia also releasing their own CPUs. So there are many competitors, but until two years ago, all market share was Intel and AMD's. Now Amazon took quite a bit of share, Nvidia and ARM starting to take more share.
Dylan: Terminal market situation is: Intel actually can raise prices, AMD can also raise prices. Both raised prices, demand of course also rose a lot. Amazon because makes to rent, not makes to sell, can extract amazing profits from CPUs. Their Graviton CPU leasing is very hot, orders increased significantly. Nvidia previously only sold CPUs paired with GPUs, now sells CPUs separately through Vera. They gave 20 billion USD CPU revenue guidance. For Nvidia this is nothing, just a few percentage points growth. Just kidding. But when you look at Intel, AMD, ARM, Amazon these companies, who gets revenue instead of only sales revenue, there are big things happening there.
Klay: Dylan, based on CPU topic, some discussions I heard are, CPUs for agents are different from historical CPUs in some aspects. Cores more optimized for agentic activities, I remember Jensen hinted around Vera CPU. Also there are discussions about GPU versus CPU ratios, this obviously highlights CPU demand and direction. Can you give more color? Because high-level concepts everyone can understand, but there are some technical details maybe overlooked. I'm not sure if this is marketing or has actual meaning.
Dylan: About agentic workflows, CPU usage ways differ greatly. Some agentic workflows are: model runs, then sends all tokens to some CPU workflow, wait for CPU to finish something then send back to model, model continues work. Question is: is model running computation paused while waiting for CPU? In some cases paused, some not. In paused cases, running model computation is waiting for CPU response, at this time CPU architecture needs to be very different.
Dylan: Basic concept is: do I want more cores or faster cores? There is a rule in CPU architecture, if you make CPU cores two times bigger, means only half the core count on the chip, each core's performance won't improve 2 times, maybe only improve 50%. Of course there is lots of engineering complexity, this tradeoff isn't that simple, but simplified this is it.
Dylan: Looking at Nvidia's Vera CPU, less than 100 cores, but each core is faster than AMD's. AMD's flagship CPU has 256 cores. Core count difference is huge, but Nvidia's cores are faster, though not two times faster. So people make tradeoffs in this design space.
Dylan: For those workloads where AI computation must stop and wait for CPU, you need the fastest cores, even sacrificing multi-core performance. I don't need super parallel workloads, I need this one workload completed right now. In this case I'm willing to accept total core count less but single core performance high. This is certain types of agentic workflows.
Dylan: Other types of agentic workflows, like how I use Claude daily, or how team spends 11 million USD annually on Claude. I call Claude, Claude processes a bunch of tokens, but they aren't just serving me alone, they batch thousands of users together. If I get reply, now wait for me to execute, whether waiting for me or waiting for some CPU core to execute, this doesn't matter, because computer is still running, just not running for me, is running for others. So if CPU is slower but I have more cores, this is different type of task.
Dylan: There is another distinction: is it active use of AI, or using something AI generated then deploying? Interestingly, if we look at global GitHub commits, rose many times compared to last year. Rise magnitude is not 10% or 50%, is several times. This means lots of code is being generated into the world, people are deploying lots of code. Lots of code is garbage, but lots of code is being deployed. After deployment put on CPU to run, is standard code. Might be a web crawler, might be an analysis engine, might be some business process automation. This doesn't necessarily need super fast CPU cores, can use cost-performance CPU cores.
Dylan: Looking at this continuum: Nvidia made highest performance CPU cores, but doesn't necessarily give you best chip-level total performance (core count multiplied by single core performance). AMD and Amazon have more cores, several hundreds, but single core performance lower. ARM also on this end. Where you are in the continuum depends on workloads. Some workloads you indeed want Vera, some you want Graviton or AMD's CPUs. I won't say this is simple.
Dylan: As for the other question you mentioned, ratios. CPU demand rising is indisputable. We were the first to point this out in institutional research end of last year, also wrote in newsletter in January this year. After we published, some CPU stocks skyrocketed. ARM rose several times, Intel rose several times, AMD also rose. But now sell-side analysts simply don't understand technology, start making things up. They talk about CPU versus GPU ratios biased towards CPU more than AI computation. This is wrong.
Dylan: Reiterate, if you look at a Blackwell, fully configured about over 50,000 USD per piece. CPU about 5,000 USD. If it's 1:1 ratio, for 300 billion or 500 billion USD Blackwell sales, you would only get 30 billion or 50 billion USD CPU sales. So another point people ignore is: yes, this terminal market is skyrocketing, but most funds still flow to AI computation and memory. This market was previously undervalued, now more reasonable.
Dylan: What needs to be recognized is, CPU demand won't keep growing to exceed AI accelerators. This is more like a recalibration. 2023, 2024 sold millions of AI chips but few CPUs. Now CPU demand suddenly inflected, ratio should adjust from original position to new position. People are in catch-up mode. I need to buy a bunch of CPUs to catch up to the amount of AI chips bought before, plus what is being bought now. Once caught up to those historically bought AI chips accumulated CPU demand, that demand is gone, only incremental remains.
Dylan: If you imagine a ratio, like 1 CPU to 2 GPUs, each GPU 50,000 USD, each CPU 5,000 USD. Then for every 100,000 USD spent on GPUs, only spend 5,000 USD on CPUs. This actually isn't that good market dynamic for CPU growth. Much better than before, but if you look at it反过来, if I had 10 million GPUs in past three years didn't configure much CPU, then this 5,000 USD has huge catch-up space. This is what we are experiencing now: huge catch-up, plus ratio itself also shifting up, huge backlog being caught up. So you see demand crazy, but it will calm down, then reach steady state. We are currently in a CPU mini-cycle.
Chapter 8: Networking and CPO Timeline
Klay: Very helpful background. Next shift to networking, this is also an area many investors focus on, especially after they delve into optical supply chain and some constraints. We see some estimates saying co-packaged optics (CPO) might not be大规模 deployed until around 2028. How do you view the architectural evolution of optics? "Use copper if possible, use optics only when must" this concept. Jensen also talked a lot about this at Computex. Marvell etc companies also received lots of attention. What extra thoughts do you have on optics and data center network architecture evolution in the next two years?
Dylan: What's clear is, as models get bigger, how do we run across models? How to train models? There are many different fields in the optical stack. Telecom optics, companies like Sienna have been skyrocketing. Data communication, chip to chip communication, there are copper fields and optics fields, these are all rising, because network content growth speed is faster than any other content growth speed. Network proportion of AI chip related spending rose from less than 10% to over 10%. By CPO era network proportion will further rise to 20-30%. So network content has huge increase.
Dylan: But on the other hand, CPO is a huge step change for the industry, everyone recognizes this. But I think now people are a bit too excited. Currently a bit overly optimistic about CPO. I don't think it will arrive in 2027. Actually end of 2028, but 2029 is the true ramp-up period for scale-up co-packaged optics. There are many problems. This is a manufacturing problem. If could deploy at good cost today, that would be great, everyone would do it. But really hard. Manufacturing volume isn't enough, yield isn't enough, chips haven't truly been designed into place. This is a very complex, difficult thing to ramp up.
Dylan: So people will stay on copper as long as possible. This means Rubin all use copper. Feynman's GPUs also still use copper, Feynman is the next generation Nvidia GPU after Rubin. Rubin, Rubin Ultra, then Feynman. We haven't even reached Rubin shipping yet, Rubin just started shipping. So still several generations of chips away from using CPO on GPUs. CPO on switches will come a bit earlier than on GPUs or AI accelerators.
Dylan: But even without CPO, as clusters get bigger, each GPU needs more optical devices or active cables. We see this huge dynamic and shift. This Monday we sent a note to SemiAnalysis institutional research subscribers, about local timeline. Not saying terminal market, our consensus is CPO will come eventually, we have been pushing this direction. Our consensus is copper will eventually be replaced. But in the mid-term we are very bullish on copper, also very bullish on non-CPO optics, instead relatively cautious on CPO itself, because we see some delays in downstream chips. Feynman won't fully adopt CPO, there are other situations. Copper concept stocks like Amphenol, they do all backplane connectors and cables, will be much better in coming years than previously expected, because we previously thought CPO would ramp up earlier, but now delayed.
Dylan: Optics is a field if you close your eyes today, open in 5 years, will find much bigger. Much is already priced into stocks, much not yet. I think there are some local misalignments. This is the research we do, also one of the works cooperating with you: how to weigh? How much is CPO optics, how much is non-CPO optics, how much is traditional optical transceivers, how much is copper? Because copper actually still has a long way to go. There is lots of innovation in the copper industry pushing back CPO timeline. Why do CPO? Because integrated optics is much more expensive than electrical transmission. Unless electrical transmission can't transmit that far, need to add repeaters or optical devices. There is this tradeoff and continuum. CPO will come, but looks like delayed a bit.
Chapter 9: Power and Energy Infrastructure
Klay: Dylan, we might still have time to chat about one last big topic. We've already talked about models, GPUs, CPUs, memory, networking. It would be too inappropriate not to mention the elephant in the data center, that is power. How do you get electricity, how to convert electricity into the form chips need. You wrote about DC versus AC in the newsletter, and some elements, theoretically when hyperscalers spend so much money building data centers, even putting power plants on-site behind-the-meter. How should we view power demand and grid versus non-grid?
Dylan: Okay. Data center growth is huge. This year we deploy 20 gigawatts of data centers. Next year this number rises to 30 gigawatts, up 50%. Next year after is 50 gigawatts. Data center capacity growth is huge. There are many local misalignments to handle. Energy is one of the biggest constraints, another is political aspects, third is construction. Building data centers and getting permits and approvals is a bit difficult politically, someone is trying to stop it. But ultimately the biggest constraint is still energy.
Dylan: Energy can be broken down into several aspects. One is power generation, where do electrons come from? Two is transmission, how to transmit electrons from where generated to data centers? Three is conversion, because transmission voltage form is different from form chips can consume, chips need different forms. What is the conversion pipeline? All three aspects have very bullish aspects.
Dylan: Transmission side is hardest to be bullish on, because building more transmission capacity regulatory and political difficulties, how local power monopolies operate, if building a power line needs to be分摊 to all users not just individual users. There are various transmission weird misalignments. So building more grid capacity at transmission level is relatively difficult.
Dylan: But in power generation and conversion aspects there are two interesting things. Power generation aspect, obviously grid power generation is increasing. There is also a big shift towards generating power for data centers. We predict within a few years, half of data center new power will be generated on-site, not off-site. Behind-the-meter power generation is skyrocketing. We have a behind-the-meter power generation tracker in our data center and energy models.
Dylan: I mentioned our team has someone, she built a power plant in Kazakhstan. Her name is Ellie, leads our energy models. We have been building this model, covering the entire grid, every power generation asset, every transmission asset, all load assets, and all behind-the-meter power generation work. Interestingly we see huge boom in behind-the-meter power generation.
Dylan: There are many struggles in permits and regulations. Someone doesn't want to give air permits, someone doesn't allow building natural gas pipelines to site. Oracle data centers encountered this situation. There are many different aspects happening. But the ultimate state is behind-the-meter power generation is skyrocketing.
Dylan: Much of this is natural gas. Much is combined cycle gas turbines, from GE Vernova, Mitsubishi or Siemens. But besides this there are many different types of energy: reciprocating engines, industrial gas turbines, various diesel engines, train engines. People took train engines, ship engines, truck engines, converted into data center power generation equipment. We see an ocean of innovation. Not no industrial capacity. US can manufacture millions of reciprocating engines annually, just engines that burn fuel to rotate. Converting these from diesel to natural gas is very simple.
Dylan: You connect an electric motor on top, reverse drive it, then it generates electricity. You can do this at scale to generate electricity. We see over 10 gigawatts of data centers will be built using this kind of technology. Take diesel truck engines convert to natural gas, can be done very simply during production, reverse connect electric motor, put at data center site, behind a data center there are hundreds, then you hire a bunch of people from auto repair shops to maintain. These engines need maintenance, they just run around all day maintaining these diesel engines. You need some buffer, this way when certain engines stop you can maintain them, keep maximum power running. You also need to put some batteries in between, because data center ups and downs won't break the engines.
Dylan: You have this entire behind-the-meter power generation supply chain, very exciting. About 2 years, solar plus batteries will be cheaper than natural gas. Solar plus batteries supply chain has difficulties, depends on what level of reliability you want. If only enough batteries for night use, that's cheaper. But if you need enough batteries for three days呢? Because might rain for two days. How many 9s of reliability do you want? Solar plus batteries is becoming cheaper and cheaper because of China's manufacturing strength, speed is amazing. There are also some subsidies. At some point solar plus batteries will become very cheap.
Dylan: Then you also have space data centers. Don't need batteries, just put in space, have one solar panel is enough. You have this entire continuum from "convert diesel engines to natural gas engines" to "launch chips into space". There is lots of money to be made. There are many interesting dynamic things to do.
Dylan: This is why SemiAnalysis's largest dataset and research vertical field, you would think is semiconductors, actually is data centers and energy. We internally call it DEI team, Data Center Energy Industrial. This is an internal pun. Jeremy leads this team, he came up with this name. Data centers, energy and industry are our largest research vertical field, because we are tracking every data center and every power plant.
Dylan: When we identify a delay or something is happening, or how many data centers coming online in a certain quarter, this is something no one else in the industry can do. This is why it's one of our largest vertical fields. Everyone is interested. Google cares about how much Meta can deploy, Meta cares about how much OpenAI can deploy, but all these companies are also looking at supply chain who has capacity. Investors are also looking.
Dylan: This is a very fragmented market. Memory only has three companies, very simple. Accelerators only a few. Semiconductor wafer fabrication equipment only a few. But this field has hundreds of supply chain companies, doing various random small parts. There are dozens of companies building data centers. There are dozens of companies doing various different things, whether as independent power producers or doing behind-the-meter power generation, or providing some kind of battery services. This is a very complex supply chain. But there is lots of vitality and innovation.
Dylan: So although data centers in some sense will continue to be a constraint, but also won't be a constraint, because depends on how crazy you are willing to be. Like I said, you can take truck engines retrofit, hire a bunch of mechanics, operate a site like this. Not the best, many people say "this is too dirty, what about reliability?" "Too much trouble". But people are doing this, this works. Although painful, but works. All the way to "I'm going to launch it into space". Also painful, hard to do, but works. So data center problems have solutions, whether you go the full dirty route or the space route. And other supply chain links don't necessarily have. This is why this market is so dynamic. You will see people going up and down many paths.
Chapter 10: Conversion Chain and Ending
Dylan: Then another part, power generation and transmission said, conversion aspect is another thing. How do you convert power from power generation or transmission form into form chips need? Here is an entire supply chain. Whether IGBT, silicon carbide, various MOSFET, gallium nitride MOSFET. What happens when we go from 12 volts to 54 volts to 800 volts DC? What happens in the conversion supply chain? How will solid state transformers be in innovation? These are all happening. UPS, uninterruptible power supply, battery backup and supercapacitors various ways to smooth power, how to convert dirty, unstable power generated on left into super clean power on right, meanwhile right side chip power consumption is also fluctuating, how do you match? This entire conversion pipeline is super super exciting.
Dylan: We just published a blog about 800 volts last week. Recently also discussed some delays with institutional subscribers, Nvidia side delayed 800 volts from Kyber. Rubin Ultra's Kyber version no longer has 800 volts. What does this mean for supply chain? Delayed a bit.
Host: Dylan, I want to thank you very much. This is if you think in chapters, this is the first episode. First time inviting Dylan to be on the podcast, but definitely not the last time, because there is lots and lots of information, like he directly said, every link of the entire technology stack is changing. Really exhausting to keep up. But as many people know.
Dylan: I also want to say one thing, this supply chain is too crazy. Many times we talk about the big ones: memory, CPU, data centers. But when you delve into the supply chain, local fluctuations are small but many. There were months we were talking about PCB drill bits, just drill bits drilling holes on PCB boards. Also random things like copper foil on PCB. All these small things in the supply chain also have these misalignments. These companies are spread across the globe, might be listed in Taiwan, might be listed in Japan, might be listed in Korea, might be listed around the world. Not just easy to access for investors.
Dylan: I think this is where our cooperation and working together is truly exciting. We can influence what is happening, can discuss these supply chain disruptions extensively, and the framework I proposed before and the entire landscape we are trying to cover. Look forward to coming on the program more in the future, and our other cooperations.
Host: Of course. Finally I must declare to the compliance team: views and opinions expressed in this podcast belong to WisdomTree, may change at any time. Any content presented in this podcast is not intended as predictive research or investment tax advice. Does not constitute advice, offer or solicitation to buy or sell any securities. Listeners decide themselves whether to rely on this information. Please remember, past performance does not represent future results. Thank you everyone for spending time with us today, look forward to coming back in the future. Take care.
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
