Shaping the Future of AI Memory

Shaping the Future of AI Memory - Six Five Summit

‍

How are cutting-edge semiconductors powering the massive scale of today's AI workloads?

‍

At The Six Five Summit, host Patrick Moorhead is joined by Samsung Semiconductor's President Paul Cho, for a conversation on how semiconductor innovations, particularly in memory and system architecture, are vital to scaling ever-increasing AI workloads.

‍

Key takeaways include:

‍

🔹A Historic Moment for AI & Semiconductors: Unpack this unprecedented era, marked by rapidly scaling AI model capabilities and the immense industry excitement fueling these technological breakthroughs.

‍

🔹Memory's Evolving Role in AI: Gain Samsung Semiconductor’s unique perspective on how memory is central to AI advancements, particularly in overcoming critical power limitations and the persistent memory bottleneck.

‍

🔹The Strategic Importance of Packaging & Foundry: Understand why advanced packaging and robust foundry capabilities are increasingly significant in Samsung’s strategic planning and customer collaborations, reflecting broader industry trends.

‍

🔹Powering the Next Wave of AI: Discover how Samsung's innovations are directly enabling the scaling of AI workloads, providing the essential building blocks for future AI solutions.

‍

Learn more at Samsung Semiconductor.

‍

Watch the full video at Six Five Media, and be sure to subscribe to our YouTube channel, so you never miss an episode.

‍

‍

Or listen to the audio here:

‍

‍

Patrick Moorhead: Welcome back, everybody, to the Six Five Summit where we are talking about unleashing AI. You're watching the Semiconductor track, where we explore the core technologies for the AI era. And you know, on the Six Five, we love semiconductors. I think we probably talk 30% just about semiconductors. They are so important, whether it's memory, packaging, foundry innovation and everything in between. We're opening up with a global leader in this space, Samsung Semiconductor. And as AI workloads scale very quickly, memory has become one of the biggest bottlenecks. Not necessarily the GPUs, but if you don't have the right type of memory or enough memory, your GPUs or your accelerators are not going to work as well as you want. And I'm joined by Paul Cho, president of Samsung Semiconductor. We're going to discuss how the future of AI is shipped by innovation in memory, packaging and systems architecture. We know the Six Five audience loves semiconductors. I think you're going to enjoy this. Paul, welcome to the show.

‍

Paul Cho: Thank you for having me. Great to see you.

‍

Patrick Moorhead: Yeah. It's just been, what, a couple years here? I mean, it seems like the pace is just torrid. We're moving forward. The industry is creating new innovations. And listen, I always knew semiconductors were amazing. I've been in and around semiconductors for 35 years this June, but it took a while for other people to catch up. That's right, yeah. So it really is a historic moment here. Generative AI and semiconductors. You can, you know, whether it's CapEx, whether it's market valuations, whether it's the buzz in, in the industry on the use cases. Let me ask you, what excites you most about the direction the industry is, is heading right now?

‍

Paul Cho: Yeah, Pat, as you know, AI is evolving at an extraordinary pace. From generative models in the data center to robots and autonomous systems at the edge. It is moving faster than any technology before it. Think about it. Cars took 62 years to reach 50 million users. Phones took 50 years, television took 22 years, and ChatGPT did it just under two months. It's incredible to think about what will be the next. So in just a few years, we've seen companies like OpenAI, Google Meta release powerful models trained on massive data sets, some with hundreds of billions of parameters. Model accuracy has improved dramatically. The the MMLU score, which is a key benchmark, has more than tripled in a few years from around 28% to 92%. So these advances are transforming the entire stack from compute to memory to packaging. Driving a need for faster time to market, lower power and TCO and highly customized architectures. But what excites me the most is that semiconductors are at the core of this transformation and Samsung has the right building blocks, memory, foundry system packaging and some selective logic design capabilities all under one roof. So we can help customers optimize for performance, efficiency, scale and time to market wherever AI leads us to.

‍

Patrick Moorhead: Yeah, it is amazing. Not that I keep doing a history lesson here, but when the dot.com era hit I actually worked at a dot com and I was part of the run up. In fact I worked for the number one search engine and, and I also, you know we saw the dot bomb where it came down. But you know, the difference between these two eras is that the green shoots of benefits and whether that's with consumers and what they're, they're experiencing by the way, part and parcel to Samsung phones as well. You are the first to widely adopt AI in your phones or, whether it's the industrial edge or related to businesses where they're seeing benefit, let's say on the edge with, with their CRM systems or the ability to do ERP better. It's pretty much everywhere and it's pervasive and there's benefits today a lot bigger than being able to buy dog food off the Internet in the year 2000. 

‍

So yeah, that's what gets me super excited is not that this, you know, there's fads and there's trends and this is clearly a trend. And if I look at the capex that's being spent today currently with the hyperscalers which I believe will move to the enterprise data center which will then move to the edge. And again it's not a serial process but I'm just looking at the amount of chips, the amount of compute required and even industrial edge. Five years ago we were talking about, you know, we called IT industry 4.0 and we all said hey, the edge is going to change overnight. Well, it didn't change as much as we, we, we would have liked it did but now we have a hundred times performance per watt AI at the edge and I think things are going to be dramatically different. I actually have a question here.

‍

Paul Cho: Right. Events changing soon is so exciting but will keep you dizzy.

‍

Patrick Moorhead: Yes, no, no, absolutely. Good for analysts by the way and analyst firms like mine. So hey, I want to drill down into memory. Okay. You know, typically it was oh, I have CPU compute and I can put multiple types of memory in and they don't even need to be that loosely coupled and. But now with the efficiency and the performance required, memory is more important than ever. Can you talk a little bit about the future of memory in a world where we're pushing the limits on performance, power, and even form factor? Like, do we reach a point where you're like, hey, Pat, we're done, we're here. We don't need to innovate anymore.

‍

Paul Cho: So we are not done yet. That's a great question, in fact. Indeed, memory poses the biggest architectural challenge in AI compute today. As model sizes scale, compute grows exponentially, but memory bandwidth lags behind. So bandwidth improves by tens of percent per generation while compute jumps by multiples. So that gap is widening. And at the system level, AI performance often depends on how efficiently you move huge amounts of data between memory and compute. So the future of memory lies in bringing data closer to compute. Because the real challenge isn't just bandwidth, it's power. And in the data center, power consumption has gone through the roof. Just to train ChatGPT4, for example, it has 1.8 trillion parameters. It took 148 gigawatt hours of energy. That's like fully charging 2.5 million Tesla Model Ys. So with Samsung High Band Bandwidth memory, or HBM, or increasingly custom HBM is the answer so on. Package HBM dramatically reduces the energy cost of data movement. And with custom HBM, we tailor performance and capacity for specific AI workloads. HBM as such, is core to our AI memory strategy. We are delivering today and working side by side with our customers to build custom solutions that meet their specific needs for future applications.

‍

Patrick Moorhead: Yeah, it's been fascinating watching how HBM started. A long time ago, very high performance memory was GDDR that was attached to graphics cards. And it still is for that, but we needed something even faster with lower latency. And you invented HBM and HBM in many cases, because I compare the different vendors of GPUs, any of the different ASICs that get connected, many times power and performance is directly related to the capacity and performance of the HBM. I can tell you there's one graphics card that outperforms another. I guess they're not cards anymore. They're more, you know, big blocks of silicon that because it has more memory, it outperforms on inference. So, and as I talk to the folks who buy my research that I advise, even on the ASIC side. Right. Who are, you know, whether it's, you know, whether it's TPU or any variant of that, their strategies are all around HBM. So it's been, it's been amazing to watch.

‍

Paul Cho: That's right.

‍

Patrick Moorhead: So I want to talk about packaging and then we'll talk a little bit about foundry. I mean the market need for. And I always like to say really good markets have three competitors in them. Foundry and whether it's packaging. Can you talk to me a little bit about the. Let's start off with packaging, the importance of, of packaging. What does it mean to your strategy and how are you working with customers? I have tracked the multiple foundry customers that you've had. A lot of mobile folks, a lot of industrial folks. You had Nvidia on graphics and it's been fun to watch.

‍

Paul Cho: Right. So packaging today isn't just about connecting chips to boards anymore. It now defines the system architecture and must handle growing complexity and performance demands for AI and hpc. The market wants packages combining logic chips and HBM and more on both. So that requires larger package sizes. For example, with Samsung we have IQUBE S that supports up to 3.3 reticle sizes. IQUBE E and IQUBE R are being explored for even larger configurations. To meet rising performance and power needs, Samsung is developing package architectures that connect high bandwidth streamlets and ensure power integrity. And as for our customers, they need a strategic foundry partner that provides scalability, innovation and long term reliability, especially in the face of growing geopolitical concerns that are impacting global supply chains. 

‍

So Samsung Foundry is focused on addressing these needs through a comprehensive technology portfolio that spans both more mature and advanced process nodes backed by additional system packaging capabilities. So this flexibility allows us to support a wide range of applications from traditional mobile and consumer devices to automotive and emerging AI. We also understand that supply chain stability has become a critical consideration here for businesses today. With fabs in Korea and Texas, we've developed a globally distributed manufacturing network that puts us closer to where our customers are and helps them minimize risks. So with our US FAF investment in Texas hitting advanced node milestones and a robust ecosystem of design solution partners, we are building momentum in foundry.

‍

Patrick Moorhead: Yeah, I love to hear that. Texas. Go Texas, and go Austin. It's funny for me, from my home I can see the Tesla Gigafactory. Saw it being built and I can see it in operation. Every, every morning that I wake up, I'm. I'm a little bit too far away from your facility here, but I hope to visit it someday once you start cranking out what wafers. Isn't it funny? Packaging? I mean, 20 years ago when I was at AMD and I was knee deep in products, we probably spent 95% on the wafer and about 5% on the package, you know, of resources. You know, it was kind of let's throw it over the wall here. But you know, as node shrinks, as new technologies have become more difficult and architectures have become more distorted, distributed particularly on the larger SoCs, packaging has become a first party citizen because squeezing out every ounce of performance per watt and cost, getting the right die done in the right process, right, everything doesn't need bleeding edge. It just, it just doesn't. And pulling this together has become a strategic advantage. 

‍

On the geopolitical stuff. I deal with the CEOs of, you know, your customers, your potential customers on a weekly basis and it is very much on their mind. And I don't think regardless of whatever terror, wherever we're going to wind up with tariffs, I think in the en, these chip designers want to have a diversified manufacturing port portfolio. They don't want all their eggs in one basket. They don't want all their chips being done in a specific country. And competition is good competition. It breeds innovation, it typically lowers the overall costs. So it's always a good thing. And we need three strong players in wafers and, and in packaging. So that's my sermon, Paul.

‍

Paul Cho: I like that. So it's surprising, it's a surprise for me to learn that you've not been to our Taylor campus.

‍

Patrick Moorhead: Not yet. I was at your, I was at your grand opening party that was about 10 miles away. But I am gonna, you know, get in a car as soon as I get that invitation and, and we'll meet you down there, right.

‍

Paul Cho: Pat, I'd love to have you there. You know, I envision the most beautiful, most beautiful semiconductor fab complex and campus in the world, right in Texas.

‍

Patrick Moorhead: So exciting, so exciting. Paul, is there anything else you'd like to share with our audience here? We're, you know, our audience is all the way from enterprise IT to investors to tech aficionados. About half of our viewers are deep into semiconductors. Is there anything else you'd like to share?

‍

Paul Cho: I think we have a lot in common but the most important keyword today for everyone is AI and Samsung Semiconductor. We are trying our best to be part of that. Not just a part, but the most pivotal role that we want to bring to the market and the ecosystem. When it comes to memory, it's the high bandwidth, low power memory architectures that we bring on the table. What's the customization capabilities that we also bring together so that our customers can have their own custom solutions that will serve their workloads in the best way possible? Because performance, power, reliability, manageability, all those things, you never want to miss any single of them. And when it comes to transistors and integration, we are focusing immensely on honing our advanced logic, process technologies and system packaging capabilities. So you bring all these things together, what ends up is you get the best product in the best time to market. So that's what we want to have and offer to our customers.

‍

Patrick Moorhead: It's a great way to close this. Paul, I want to thank you for coming on the Six Five. We'd love to have you on again to just drop these truth bombs here and you know, Samsung semi's been a little bit quiet lately and people want to know, hey, what's going on there? So thank you so much.

‍

Paul Cho: I'll be happy to come back anytime.

‍

Patrick Moorhead: Excellent. I appreciate that, thanks. So hey, thanks for tuning into this kickoff session for the semiconductor track. Paul did a great job, hopefully learned a lot and he disclosed some stuff to me that I wasn't aware of. Maybe I wasn't paying attention the first time, but it was very informative for me and hopefully you found that as well.  Memory is not just a component, HBM is core to everything. Foundry and packaging are as important as, you know, in the Six Five audience that we've discussed here. So we'll be hearing from other folks across the industry as we can explore the chip system strategies powering the next generation of compute. Check out the full lineup on the website. More insights coming up next. 

‍

‍

‍

Disclaimer: The Six Five Summit is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded, and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors, and we ask that you do not treat us as such.

‍