Home

Beyond Compute: What It Takes to Keep AI Fed at Scale

Beyond Compute: What It Takes to Keep AI Fed at Scale

At SC26, Micron’s Prasad Alluri explains how agentic AI is making memory and storage as critical as compute, from data tiering and PCIe Gen 6 SSDs to planning supply years in advance. Organizations that design memory and storage in from the start will keep AI fed at scale.

AI conversations tend to center on compute, but feeding those platforms with data is just as challenging. As AI shifts from large training jobs to real-time agentic inference, systems need to move, store, and reuse data faster than ever. The simple model of writing to memory and committing to storage no longer holds.

At SC26 in Chicago, Matt Kimball, VP and Principal Analyst, Datacenter, at Moor Insights & Strategy, is joined by Prasad Alluri, Vice President and General Manager of the Core Data Center Business Unit at Micron, to discuss how agentic AI is reshaping memory and storage architectures, what system architects should prioritize, and how the industry can keep up with demand.

For infrastructure leaders, the takeaway is to plan memory and storage from the start. That means designing data tiers around how close each type of data needs to sit to compute, treating networking, memory, and storage as one optimization problem, and building supply into the plan as early as power and cooling.

Key Takeaways:

🔹 Agentic AI makes AI data- and memory-centric. Alluri says AI is moving from compute-centric training to inference that depends on moving, storing, and reusing data. Memory and storage now come in many tiers, with more emerging as agentic workloads grow.

🔹 Data awareness decides where everything lives. The hottest data sits closest to compute in high-bandwidth memory that moves bits energy-efficiently to avoid thermal limits. Warm data lives in main system memory, shifting from DDR5 to LP5 and SOCAMM, while colder data moves to fast or capacity-optimized storage tiers.

🔹 Storage scales for both speed and density. Micron was first to introduce PCIe Gen 6 SSDs, starting with NVIDIA GPU-based systems and now extending to AMD and Intel platforms. For high-density storage on AMD CPU-based systems, Micron uses 245-terabyte SSDs to reach multiple petabytes in a 2U system.

🔹 Architects should optimize the whole system as one unit. Alluri traces today's large AI systems back to distributed computing and coherent domains from HPC. As those domains grow beyond 72 GPUs, he argues networking, memory, and storage must be designed together, starting with how east-west and north-south traffic are partitioned.

🔹 Memory supply must be planned years ahead. Alluri says the metric that matters is now useful work-per-watt, -rack, or -dollar, and memory and storage belong early in the RFP process alongside permits, power, and cooling. A new fab takes 5-6years and a node upgrade about 18 months, so Micron is working across the supply chain, signing long-term agreements, and co-engineering with partners three to four years ahead of each annual silicon cadence.

Organizations that design memory and storage tiers into their AI infrastructure from day one will be better positioned to keep compute fed as agentic workloads scale.

Watch the full video at sixfivemedia.com, and subscribe to our YouTube channel so you never miss an episode.

Learn more about Micron at micron.com.

Disclaimer: Six Five Media is for information and entertainment purposes only. Over the course of this video, we may discuss companies that are publicly traded, and we may reference their equity share prices. Nothing discussed during this webcast should be considered investment advice or a recommendation to buy or sell any security. We are not investment advisors, and you should not rely on this content as financial advice. Six Five Media collaborates with technology companies and industry leaders to produce research-driven interviews and multimedia programming for enterprise technology audiences.

Transcript

Matt Kimball:

Hey everybody, welcome to Six Five On The Road, live from Chicago, Supercomputing 26. My name is Matt Kimball, and we are here talking about a lot of things supercomputing. As you know, AI has kind of dominated the conversation, and in that realm, compute has been, and really big compute platforms have been kind of the conversation that everybody seems to want to have. However, feeding those compute platforms is really just as big of a problem as those compute platforms themselves. So here to talk about this with me today is Prasad Alluri, General Manager and Vice President of the Core Data Center business at Micron. Hey, Prasad, thanks for joining me.

Prasad Alluri:

Thanks, Matt. Pleasure is mine.

Matt Kimball:

Let's have a fun conversation. Absolutely. Looking forward for that. Yeah. All right. So, listen. So, let's do the setup here, okay? Supercomputing, AI, it's shifted from these large training jobs to real-time inference, agentic AI, right? And that means I have to get real-time access to data, and I need to act on that data as fast as possible. So as you think about this at Micron, and you, you know, obviously the makers of a lot of what sits inside of that platform, how do you see the role of storage and memory changing to support this brave new world, if you will?

Prasad Alluri:

So Matt, certainly the AI itself is transforming into a completely new era. What used to be compute-centric, driven by training, is now moving to data- and memory-centric, is what this agentic inference is making that change to happen. So, when you look at that agentic, you know, taking the center stage, what's important is the ability to move the data, store the data, and reuse the data. That's what is actually causing the biggest change compared to more so than before. So what it means is that both memory and storage is undergoing significant amount of tiering within themselves. So what used to be, you know, back when computer scientists were taught, it used to be two-stage programming. You write into memory, then you commit it to storage. It's no longer that simple. Memory itself has many, many tiers now as a result, and more tiers are evolving. Perhaps we might be able to talk about it during this conversation. There are tiers that exist already, and then there are tiers that are going to be emerging as a consequence of the agentic era.

Matt Kimball:

So, I'm glad you say that. So as you look at, we talk about HPC, we talk about AI and kind of enabling AI across an enterprise, agentic AI, real-time inference, you have to feed that compute platform as fast as possible. And to what you're saying, you see new technologies emerging, right? With agentic AI, you hear about KV cache now, right? And that has a huge impact on memory architectures, storage architectures. As you think about having to support larger memory footprints, you think about capacity, bandwidth, efficiency. How do you manage the trade-offs that have to be made to get you to scale appropriately?

Prasad Alluri:

So what's key in this construct that I just talked about is the awareness of the data. What part of the data needs to reside a lot closer to the compute, and what can reside right behind it, and what can reside way away from it. So, that awareness is essential for us to be able to decide what kind of memory that we need to be building. So things that require the data that need to be very close to the compute means that it has to have ready access to the compute, high bandwidth, and not just high bandwidth, but in a manner where the bits are actually getting transferred energy efficiently. Otherwise, you'll run into thermal challenges, as an example. And then the next tier is about, okay, the next part of the warm part of the data, so-called the warm part of the data, is going to be residing in the computer memory, I mean the main system memory. Today, that is in the form of DDR5 transitioning to LP5, SOCAMM, more as a physical implementation. And then things that are even colder, so to speak, that is going to be going into the storage tiers. There are fast storage tiers, and then there are colder storage tiers. And so that is the kind of data tiering that we do, and it's absolutely essential to be able to be aware of what kind of data needs to be residing there.

Matt Kimball:

So speaking of that, getting into storage, right? Storage is obviously increasingly important. It's always been important. Increasingly important as you have to keep feeding those GPUs and accelerators in real time. Y'all have been doing some work with AMD and Intel, and you're showing off some of your kind of what you've been doing. Can you talk to that a little bit?

Prasad Alluri:

Yeah, absolutely. So if you step back and think about storage back even when the AI actually became mainstream, there was just high-performance storage that was basically attached to the GPU. And then you had the data lakes that was residing in big pools of NAND that was more optimized for capacity as opposed to performance. More recently, that has moved to QLC from TLC type of NAND. So that was the construct that existed. So now, with the agentic era coming in, as I talked about, there are more of the data tiers that are getting established. In the case of storage, what it means is that you have to have now different tiers of high performance, different tiers of so-called warmer and colder type of data, and so with both, AMD and Intel, there is work that's being done in both memory as well as storage. Since you asked about storage, I'll stick to the storage work that we're doing. And so in the context of high-performance storage, which is primarily used for two things. One is during training. You are doing a lot of the snapshotting so that the work is not lost, and you don't have to start the session all over from the beginning. It requires super high performance. It means that you want to be able to write to this state and so that becomes the last known state, and you can start the computer. High-performance SSD. We are actually the first ones to introduce PCIe Gen 6-based SSDs for the first time, and we've started with the NVIDIA GPU-based systems. Now we are extending that to AMD- and Intel-based systems.

Matt Kimball:

So the new Intel D270 GPU? Yes. And then in addition

Prasad Alluri:

We are also trying to build storage systems that are meant for high density. And using AMD CPU-based storage systems, we are able to use 245 terabytes per SSD. Oh, wow. And in a 2U system that you're talking about, multi-petabytes. Yeah, that's incredible. That's unbelievable. That's unheard of in the past. That's like science fiction almost.

Matt Kimball:

Yeah. Appropriate for supercomputing, I guess. Yes, absolutely. Hey, so step back for a second, okay? So we're talking about memory; we're talking about storage. Put on your systems architect hat, okay? I'm rolling out these next-generation systems that are going to solve the biggest problems and deliver real-time inference and manage my agentic environments. I'm thinking about compute; I'm thinking about storage, memory, networking, power. If I'm that systems architect that's sitting down and think about kind of how I deploy this, what should I be thinking about? How should I be kind of approaching that, how to deploy that big system out there?

Prasad Alluri:

So, very interesting question, Matt, actually. So, it's all about thinking computer as a distributed computing system. And that's, you know, by the way, it all started in HPC space way back when AI adopted it. And if you look at all of the GPU-based AI systems, they're all big distributed computers. And the whole concept of, you know, coherent domain, that all came from HPC, right? And so, as the AI, you know, itself started becoming bigger and bigger, bigger amount of data that are coming in, the coherent domain started increasing, right? So, what used to be, in the case of NVIDIA, NVL72, now it has, you know, multiplied from that 72 GPUs to many, many more GPUs as a coherent domain. So now what it means is that it starts with networking equally important. We, as a memory and storage company, tend to be overly focused on that. That's okay. Networking is where a lot of this comes to work. East-West traffic and North-South traffic. How do you repartition them? It starts from there, and then comes down to where you place your data in the memory and storage. And it's not just about thinking about networking and memory and storage and discrete steps, but it needs to be thought about as one single optimization unit. So that is the biggest thinking shift that's been happening. And so that's...

Matt Kimball:

It's funny, because I remember, man, it must have been seven, eight years ago, NVIDIA was on stage and talking about the new measurement of compute was at the rack level, not at the server level. And it's so true. And I agree with you that, from a networking perspective, that's foundational. It's the plumbing. But it all has to feed up into the platform itself. And memory and storage is obviously so critical to that equation. So, you know, we kind of, you move forward, you're looking at, you know, labs are expanding, putting out, you know, the latest and greatest infrastructure. AI is taking over hyperscalers, you know, neoclouds, enterprises, all this new infrastructure's going out. It's putting a big demand on supply, right? How do you see, y'all at Micron, you're kind of on the front end of the innovation, right? It kind of all starts with you. But how do you see that how that ecosystem needs to work together from kind of the research to manufacturing to supply how does that you know how do you see that kind of does that get better in the future and how does it get better in the future to better meet the needs of the

Prasad Alluri:

Yeah, so maybe we should step back and start with what matters. I think it used to be that peak performance was always the criteria that was used. You know, performance per watt, tokens per watt, whatever, right? But I think from there, the shift now is about usefulness. Compute per either watt, or per rack, or whatever other metric per dollar. I think that's the kind of new kind of metrics that are evolving. It's not just about meeting certain peak performance spec, but it's about what's useful for the user.

Matt Kimball:

And that's, by the way, not to interrupt you, but what's interesting about that is even within an organization, for some business units or some workloads, it will be absolute best peak performance. For others, it's best useful outcome. For best, it’s: how do I do this? How do I run this the cheapest? Absolutely.

Prasad Alluri:

That's why it's useful; it’s what I use. And what is useful is probably different for you than me? And so now, talking about the supply chain aspect of what you asked, in this, you know, thinking of useful work and, you know, output, useful output for whatever the denominator is. And you'll have to think about the whole, you know, aspects of when you first start, when you start. You have to be thinking about memory and storage being the bottleneck now. You have to treat it as the early part of your RFP process, as opposed to actually thinking about it as an afterthought. So what it means is when you're thinking about your permits, when you're thinking about your energy access, power access, thinking about your cooling systems, all of that is when you have to be thinking about memory and storage as well, right? And so Micron is doing its part to address this shortage that you alluded to, but it does take many, many, many years, right? And it's like, you know, we are. Trying to develop the next node. We are also trying to build our next capacity to address all of this demand change. And for a new factory to be built, you're talking about five, six years from scratch. And for an existing factory, if you're actually trying to upgrade, to a newer node, you're talking about 18 months, right? I mean, that's the kind of length that it takes for us. And so, despite the fact that we have the visibility, all of this visibility kind of came much later, right? And it came as a surprise to most of the ecosystem. And so what we are trying to do is work with all of the ecosystem players and get down to the, you know, way to the front end of the line, and then making sure that we have visibility across all of the supply chain. And so we are efficiently able to actually allocate all of our bits. More importantly, we are also making sure that the customers are actually signing up for long-term agreements such that we have the visibility, and we have the ability to actually build all of the factories needed to provide all of that supply. That's something that we are doing to address that supply-demand challenge.

Matt Kimball:

One last question for you as a follow-on. Do you find, you know, we've gone in the industry from an every three- or four-year kind of cadence of new silicon coming into the market to every major vendor is now trying to hit that annual cadence, right? Does that have a big impact on Micron, and how do you kind of, how do you look at that and kind of adjust or, you know, support that?

Prasad Alluri:

Yeah, absolutely. I think if you look at CPU silicon roadmaps and GPU roadmaps, they have been. I mean, big hard shift roughly about once in two years, but then there is a minor shift that happens at either a package level or so almost every year, all of which requires us to work closely with them. And in the case of, since it's a topic is all about AI, Co-packaging means that we have to co-engineer together, whether it's a substrate or the TSV pitches and all of that. And so while you get to see the silicon coming in every year at a cadence, you're talking about us working with our partners at least three, four years ahead of time. Because you have to co-engineer both of them together. That's a big, big change for us. It used to be that, you know, we could actually put a, develop our memory and storage sitting behind the standards-defined interface, PCIe for storage and, you know, standard DDR, JEDEC-defined interface, but that has changed now. I mean, you have to be co-engineering and integrating years ahead, and that's what's actually causing all of the partnerships and

Matt Kimball:

It's crazy, man, and while I don't envy you, I admire Micron incredibly for keeping pace and actually staying ahead and leading the innovation.

Prasad Alluri:

It's all physics-level inventions is what we have to do, right? Pitches are coming down, transistor pitches, in the case of logic. For us, it's the bit density. You're talking about, you know, nanometers per scale. And that is, you know, tens of atoms across a critical dimension. So engineering at that level is a lot of new physics is invented, new materials are invented, and that's what we are actually doing.

Matt Kimball:

I don't want to call it rocket science, but it's kind of rocket science. Yes. Prasad, this has been a great conversation. I can't thank you enough for taking the time. And for all of you tuning in, thanks for listening in to this Six Five On The Road at Supercomputing. Please like us, follow us on social, and we will see you at the next recording.

MORE VIDEOS

Why Most AI Agents Never Make It Past the Demo (MongoDB)

MongoDB Principal Technologist Devin Dickerson joins Jason Andersen to explain why AI agents fail on the path from demo to production, as data moving across disconnected systems adds latency, drift, and cost. For enterprise platform leaders, collapsing operational data, retrieval, and memory into fewer systems creates the trust and cost predictability that production agents require.

How Far Away Is Quantum, Really? Timelines, Encryption Risk, and the Talent Gap

On The Main Scoop with Daniel Newman and Greg Lotko, quantum industry veteran Denise Ruffner explains why 2030 is an aggressive timeline for quantum computing, why post-quantum security can’t wait, and how the industry can widen its talent pipeline. Organizations that start their quantum-safe transition now will be ready when quantum moves from research to advantage.

Intel on Why Agents Need More CPUs, Not Just More GPUs

At OCP Global Summit 2026, Intel’s Jeff McVeigh explains how two years of x86 ecosystem collaboration and open standards are shaping AI infrastructure as agentic workloads put CPUs back at the center. For enterprises, shared interfaces make it possible to deploy AI inference inside existing data centers.

See more

Other Categories

CYBERSECURITY

QUANTUM