Cisco, Nvidia Tackle AI’s Growing Data Center Bottleneck

[00:00:00] Speaker 1: I'm trying to get into the sort of balance between the layman's and going into the detail of USFP, but a new optical transceiver format, right? What's the problem you're solving for here? [00:00:11] Speaker 2: So the problem we're solving. So first, if we just take a step back and say what is happening right now in AI, the amount of traffic that's going to get generated with these agents, which is the second big phase of AI that we're in now, especially with these personal agents, is going to be growing exponentially. And so what you're starting to see is you no longer just have to make sure that you have the rack itself scaling. But now what you're starting to see in data centers is you're going to need to have hundreds of thousands of GPUs over time within a data center that need to act as one coherent cluster. And so what you need to do is you need to make sure that those emissions can't just happen with a wire. They need to happen with optical, you know, kind of transmitters, which is basically light rays that go, you know, across, you know, go over the fiber. And so what we've done is we have partnered with NVIDIA to have the next generation pluggable optics. It's a standard called ultra dense small form factor plug, which is USFP. And what that allows you to do is think about this is it's just going to have the exact same highway where you can actually have many, many more cars go through. And so it's going to have at least 4x the bandwidth and you're going to be 1.3% or 1.3x more power efficient than what you used to have. So so much, much more traffic patterns that can actually be accommodated with this new new collaboration that we're doing with NVIDIA. [00:01:49] Speaker 1: You know, G.C., there are so many people watching the show that always like want to be inside the data center and that, you know, it's really hard to comprehend what's happening. The module converts that electrical signal from the network switch into light. You know, that that's the technology. Huge amounts of data traveling over fiber. Critically important. I'm trying to get to what's new in the USFP 400 gigabits per lane versus like prior generation technologies that weren't at that level. Just go with that a little bit. [00:02:19] Speaker 2: Yeah. So I think the other standards that are emerging don't actually what this allows you to do is, you know, it's built and deployed and surfaced and kind of serviced and managed. In the same way as pluggables are. And so you don't have to have a full re architecture of your data centers. That's a huge advantage of this kind of standard and this this format. And you have up to, you know, kind of 16 lanes or 32 lanes or 400 gigabits per second transmission that's going through. So it's just a lot more bandwidth. It's much faster. And I think these switches, if you take a step back, what you need to make sure that you have is these switches within a data center need to transmit, you know, traffic at very low latency so that your GPUs don't remain idle. And that's what we're trying to do over here is make sure that you actually have extremely fast transmission at very, very low latency and very high power efficiency. [00:03:17] Speaker 1: Why did Cisco need to work with Nvidia on this? Nvidia specifically? [00:03:21] Speaker 2: Look, I think this isn't the one thing that you have to keep in mind is this is an ecosystem play in general in this market. It's it's very hard for a single company to just go out and do everything. We need to make sure that we're working with partners. And Nvidia makes GPUs. They make they make these rack systems and they really go out and equip the data centers. And we make the networking that goes out and connects these data centers. And we have we have these coherent pluggable optics modules. And we wanted to make sure that the standards that we're coming up with, we're actually collaborating with others to do that. And so we're really looking forward to collaborating with Nvidia. The nice part about Nvidia is we have a great partnership with them. We've worked with them for many, many years now. And we continue to keep finding different avenues where we can keep pushing the kind of challenging the limits of physics and keep pushing this forward. [00:04:15] Speaker 1: Right. Your starting position was sort of that, you know, AI is supply constrained, right? But supply relative to the demand, I find where the demand from a tokens perspective is coming so interesting. The agents are using 5x the tokens that a human power user is, right? How is that sort of exacerbating the problem that you're trying to solve for? [00:04:38] Speaker 2: So it is a very interesting problem that we actually face right now, which is as we move from this era of chat bots to agents, these agents are, like you said, far more consumptive on network bandwidth than even a human for conducting the same exact task. It's it takes about 500% more for an agent to conduct the same task with network bandwidth compared to what a human would take to conduct that same task. And so what you're starting to find is a tremendous amount of need for more and more kind of technology innovations that can push the envelope specifically in the data center. And so now what we are at the point of is these today sub 2% of human beings within the world are using agents in any kind of power user capacity. And my prediction is that within the next, I don't know, 15 months or so, you'll probably have 30 to 40% of humans use these agents in a power user capacity. And so there's going to be an exponential rise. And with 2% of the humans using these agents, we are still completely saturated in supply. So you have to make sure that you don't just build more data centers, you make the existing data centers more effective. [00:05:49] Speaker 1: So let me jump in on that, GT. The basics are, you know, USFP is a super fast plug connects all the AI supercomputers in the data center. But to your point, you also have to discuss the economics. So this technology that you've worked on with Nvidia, how does it change the sort of speed to get something online, but also the economics of running that data center? [00:06:13] Speaker 2: The economics of the data center, the total costs of ownership actually keep going down as you think, as you start to think about cost of tokens. So the cost of tokens actually gets far more efficient than what it has, what it's been. And what you also see over here is the power efficiency is 1.3 X. So it's 1.3 X more power efficient than what you would have, which then would have a direct correlation to what your cost of tokens would be within this entire, you know, kind of equation. So 4X more bandwidth, 4X more bandwidth density, 1.3 X more power efficiency, 16 lanes of 400 gigabits of scalable traffic, or you can go up to 32 lanes. And so that's the, those are the stats. And the key point over here is it's built, it's deployed and serviced and managed the same way as today's pluggables, so customers can get more out of the way that they've already architected the data centers.