Advancing Optical for AI

Does connectivity need to change to power an era of AI where millions of devices seamlessly interact?

‍

Find out at the Six Five Summit: AI Unleashed! Xi Wang, Vice President and General Manager of the Connectivity Business Unit at Marvell, is featured as an Intelligent Edge spotlight speaker. Watch as he joins host Will Townsend for a pivotal discussion on advancing connectivity solutions in the age of AI.

‍

Key takeaways include:

‍

🔹Connectivity's Critical Role in the AI Revolution: Explore the profound evolution of connectivity and its indispensable function in accelerating the AI revolution, from its very inception.

‍

🔹Designing Next-Gen AI Connectivity: Delve into the major challenges involved in architecting connectivity solutions for the next generation of AI, particularly as systems grow to encompass millions of devices.

‍

🔹The Challenges of Rapid Data Center Expansion: Understand critical strategies for addressing scalability issues and navigating supply chain concerns amidst the explosive growth of data centers.

‍

🔹The Future of Optical Integration: Get a preview into the future of optical integration within servers and chips, a crucial advancement driven by the relentless progress of AI.

‍

Learn more at Marvell.

‍

Watch the full video at Six Five Media, and be sure to subscribe to our YouTube channel, so you never miss an episode.

‍

Or listen to the audio here:

Will Townsend: Hi everyone and welcome to the Six Five Summit AI Unleashed. I'm Will Townsend. I lead the networking and security practices for Moor Insights & Strategy. And joining me today is Xi Wang, senior vice president and general manager of the connectivity business unit at Marvell. And we're going to discuss the intelligent edge in this special spotlight on the Six Five summit. Xi, it's a pleasure to talk to you today.

‍

Xi Wang: Thank you.

‍

Will Townsend: Well Xi the theme of our topic today is AI in the million XPU era. Let's talk about that a little bit and how important networking is to support modern AI.

‍

Xi Wang: As we speak. The operator are already deploying AI networks with 200,000 of GPU altogether. And then to support that number of GPUs you need a fast low latency and big network is a part of the ecosystem partners to support that enablement and the operator are moving even faster to 1 million XP altogether. There are a lot of challenges and opportunities for everyone in this industry and we're continuing to bring innovations such as low latency networks, higher bandwidth optics, and also technology to enable multi-site, high bandwidth, large scale networks. And we are very proud to work with all the ecosystem partner to make that a reality. And you can see that in the past we have announced several new products, but there are more technology that in the kitchen that we're putting.

‍

Will Townsend: I'm wondering, there's a lot of debate around this, but has AI accelerated the adoption of connectivity or has it just simply changed where it gets used?

‍

Xi Wang: Well definitely AI has been a huge driver for connectivity. The global market for optical interconnect has doubled since 2020 to nearly 20 billion and it's expected to double again by 2030. Because of the increase in bandwidth and the growing size and the number of clusters, hyperscalers are deploying more and faster optics. The industry KR is around 18%. AI also jumpstarted the transition to higher bandwidth optics. Traditional compute servers, IO bandwidth lacked behind the network IO bandwidth by at least one generation. However, when you move to ai, the AI server bandwidth leap forward by one generation. So this transition triggered much higher demand for higher bandwidth connectivity. Since AI is also based on parallel computing, the physical dimension of each node also increases that increase. It requires longer distance interconnect. 

‍

So which is the advantage of using optical transceivers or active electrical cables. The vast majority of connections three meters or longer are supported by this technology. This next generation AI will be particularly interesting because we're talking about clusters with thousand to 1 million XPOs. So how do you make a fabric with high bandwidth and low latencies while simultaneously improving costs and efficiency? This is where new innovation is needed. On the connectivity side, you can see innovations happening across the board at the border level. New generations of a service IO are being deployed and developed at a faster cadence, at a cluster level, optimize electrical and optical solution for each different application. Scenarios are becoming available across locations. We're delivering coherent transmission technology, which was reserved for long haul telecom, but now being used into data center of campuses. So all this innovation and has become an enabler for next generation AI. This is not about who has the best accelerator, but who also has the best connectivity solutions.

‍

Will Townsend: I couldn't agree more, Xi. Connectivity is the bedrock to move data around these AI models, whether they're large language hosted in the cloud or as AI moves to network edges requires the efficient transfer of data. And so that presents a lot of challenges. So I'm wondering from your perspective, what do you see as some of the biggest challenges with respect to network and just supporting those AI workloads at scale?

‍

Xi Wang: Indeed, I think first is we'll need more GPUs that work together and we'll need a bigger fabric to connect them. A cluster is 100,000. GPUs might need 500,000 interconnect along with thousands of servers and switches. A million GPUs could need 10 million interconnect, but collectively spend several kilometers. Power could approach a gigawatt. So this transition to higher bandwidth also accelerate instead of two times bandwidth every three years we're doubling every two years. Hundred gig per lane interconnect are being deployed with 400 gig on the horizon. So if you look at a system that enables AI, there are two separate and connected network scale up that enhances the capacity of a single server or system by combining all the resources together, scale out, bring this servers to the network in scale up. The overriding concern will latency followed by power scale up is also a new market for connectivity. We expect lots of innovation happening here in the next few years, such as ultra low latency networks, co packaged optics and the photonics iOS in scale out. The big issue is scalability. There we see potential innovations such as light 400 gig optics and you need to support a new modulation schemes to support all these use cases.

‍

Will Townsend: No, those are fascinating insights and from my perspective, there's another dimension to scalability and it comes with the dramatic increase in the pace of data center buildout. So there are commitments around the world just recently in the kingdom around the investment in AI infrastructure to create new use cases, drive new monetization opportunities. I'm wondering from your perspective, what's the industry doing to sort of ease the potential supply chain in management concerns that comes with this hypergrowth?

‍

Xi Wang: Well, this is a very interesting issue. I mean usually it doesn't get a lot of tensions than the power consumptions and then the economics, but it is just us important. I think the short answer is the ecosystem, right? The number and the size of AI cluster have grown rapidly in part because of the ecosystem. The operator need a flexible and interchangeable solution that scale to support their need. And then if you think about it, the combination of a continual innovation of CMOs and OK technology pushes lower dollar per gig at each generation. And then we're moving from five TER to three nanometer technology that improve the power and cost efficiency, a stable and a mature pluggable ecosystem, flexibility and scalability both on the operational side and also the commercial side. So in our mind, this vast network of ecosystem partners that support pluggable ecosystem is the go-to solution for large scale network and for them to continue to scale.

‍

Will Townsend: And certainly Marvell is participating in many of these areas. And as we close our discussion, Xi, I'd like to talk a little bit about optics for inside the server or inside the chip. You touched on co packaged optics. People have been talking about this for decades, but it's still not quite here yet. So will AI accelerate that whole notion and how will it manifest itself do you think over time?

‍

Xi Wang: Well, this is a really good question as well, right? So I think things are happening, but over time, currently the whole system design is focused on disaggregation memory processors and networking, they are all position in separate domains that are connected together. But there's a strong push that allow this separation to have, enable faster transitions and then time to market for new technology. We're expecting a convergence of the interconnect technology to end points that can facilitate that. So those technology include, as you said, advanced packaging. It all comes down to the implementation of that underlying technology and where you pull them together. This consideration must be done at the overall system level, including thermal power, cost to market, and ease of operation. The semiconductor industry, like where we are in, has always been enabling higher level of integration at the right TCO intersection point. So this time for updates going inside the server won't be any different. That is our will.

‍

Will Townsend: Yeah, and you touched on power. That's one of the biggest concerns that enterprises are facing right now. Just the immense power that it takes to support these modern AI workloads. And Marvell is doing a lot of work in that regard, providing higher degrees of performance, while also bringing that power consumption down to make AI infrastructure more sustainable. But with that, Xi, I just want to thank our audience for joining this Intelligent Edge spotlight at the Six Five Summit. Stay connected with us on social media and explore more conversations@sixfivemedia.com/summit. More insights are coming up next.

‍

The Six Five Summit is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded, and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors, and we ask that you do not treat us as such.

‍