Cisco AI & ML Blueprint for Data Center Networking — Intelligent Infrastructure at Scale

[00:00:00] Speaker ?: AI/ML Data Center Networking Blueprint [00:00:10] Nemanja Kamenica: My name is Nemanja Kamenica and I'm Technical Marketing Engineer at Cisco and I'm here to talk about AI/ML Data Center Networking Blueprint. It's a set of stuff which we created to explain how you can create the network the best possible way for your AI/ML workloads. So what I have today on the session is why AI is important today and what it will bring in the future and then explain to you how you should create networks for your AI cluster. I'm going to show a brief demo of everything which I'm going to talk up to that point and then I'm going to show you the set of collateralers which we have created to enable you as a customer to do this. So first, let's talk about AI and what AI could do today and what we'll do in the future. There are a set of use cases and industries which will benefit from AI. you also have all have seen ChatGPT, you probably asked some of the serious questions, some of the silly questions and it always respond to you. So that's one way to entertain yourself. But there are other industries which will benefit from it. For example, healthcare will be able to do some medical research and medical risk research. In financial services, they might be able to advance the trading algorithms and do more trading using AI. In public sector, they might be able to optimize public transport paths and have more users using those paths and public transport in the cities. In media entertainment industries, creating subtitles, doing translation using AIs is a great help. So manufacturing, finding out the flaws in the products will be enabled by AI and be able to do that in a scalable way. And in the end, retail, personalized recommendations for any of the retail stuff could be done with the AI. So capabilities, use cases are vast, wide, and as such will be deployed by many enterprises. So when we talk about enterprises deploying this, there are kind of two major type of AI clusters. We have something we call distributed training cluster, which is basically a cluster which trains your AI model. So imagine that you want to have an AI model which recognizes a particular color. I'm simplifying this, oversimplifying this, but let's say it trains the color. You need to give it a set of colors. It's going to go do that task and eventually will say, "Hey, this is red, this is pink, this is yellow." So what that cluster kind of requires from infrastructure is that node-to-node communication in this case is high. The reason is they take those samples and then exchange. They do calculation. Then they exchange the calculation. They make an average and then they update the model. So all of this, which I said, it happens between the node communication. It happens over the network. And as such, the network is required to do high bandwidth, low latency type of a transport in this case. Also, the key metric for this type of a cluster would be training time. So shortest period of time to train your model. What does it require from the infrastructure? It would be a large network with GPU, a lot of GPU and CPU power. So that's our distributor training cluster. There is another way to create a cluster. And that's happened when you have finished training your model is that you have a product inference cluster. So now you train the model to recognize a color, red color. Now you're going to deploy that model and users will be able to go and ask that model. Hey, what is this color? Or is this a red color? So it's a different type of a cluster which will do this task. What is required from this model is to be real time and highly available. So anytime anybody of the users asking this, you need to provide it an answer. Obviously, you're not going to have a single user doing this. You will have hundreds or maybe thousands of users asking this for model to do. And as such, you might not have a lot of bandwidth, but you need to have a lot of high or a lot of availability, high availability of that cluster. So what is required from the network? It's a smaller network with smaller amount of devices, but it has to be real time. It has to be always available. So with this, how do you create those clusters? First, you can create that cluster on-prem in your own data center to serve your own enterprise needs. What would be a benefit of this is that cluster is always available. So there could be somebody always using that cluster. Also, you have a flexibility to create that size of the cluster however you want. Different parts of the enterprises can use that, kind of have a timeshare of that cluster. So use that cluster at a different time. And all your data is stored on-prem. So you do not export that data or you do not send it anywhere. It's present in your cluster. Another way for you to do this if you're an enterprise is to do it in a public cloud. So all of the public clouds do enable you or do have an offering to do your AIML training cluster in public cloud. Flexibility is a benefit of that. So I'm going to go, I'm going to get a number of instances with a particular bandwidth. And I'm going to pay only when I use it. But what will happen is that how you grow your cluster, your cost is going to grow. So it might be fine for a certain time, it might be fine for forever, but there are different cases and different use cases which can be solved. So let's say you are an enterprise and you've decided to build a cluster in on-prem situation. So you're going to build your own data center which will run AIML workloads. So what would be a key set of challenges which you need to understand before you do that? First one is that model which you're creating and data which that model will use will double every two months. So as such, you would need to scale, you would need to improve your infrastructure to be able to keep up with that model. What that bigger model provides you is more accuracy. So if you are training for something complex, it's going to probably take time to train that model. If you increase the size of the cluster, your time is going to go down and so on and so on. However, today, most of the training happens on the cluster size up to 512 GPUs. So I guess if you would start creating, start looking for something which is 512 GPUs or higher. The next challenge which you need to solve would be key components which you need to have in your cluster. So what do you need to have? Compute nodes. Those will contain GPUs and CPUs, but also you would need to have a network. That network needs to provide obviously transport between all of those nodes. And then you would have some kind of storage system which will contain that data and will be able to provide that training data into the cluster and later basically collect all that data from the cluster into a storage device. Then there are two software components. Then there are two software components, job scheduling and orchestration component, which will basically tell you which GPUs and where they are and who of them or which of them will work in this process. So some of them may require lower latency path. And as such, you might want to have them on the same switch or nearby switches. Some of them may not require low latency and a higher scale, so you might distribute them as well. So the next component would be software framework for AI model. So that's how your AI model will work and how it will be trained. And that's what you can buy. Since we are here and talking about data center networking, obviously the focus of presentation will be on the network in this case. So let's look what network for AI cluster will provide in this case or what kind of requirements are put to the network to do this. So when we talk about AI training networks, there are two requirements or two components which are there. The first one would be transport which you provide. In this case, this is ROC EV2. I don't know if any of you know what ROC EV2 stands for RDMA or Converged Ethernet Version 2. I'll explain what that means in just a moment. But as a technology provides low latency, high throughput type of a transport from endpoints and does require, does have some requirements for the network. From the network side, we in Nexus product line, in Nexus 9000 switches, we do have a set of switches which do provide low latency and high bandwidth forwarding. With ASICs up to 25.60 present in our equipment and their purpose or their could satisfy all the needs for from the AI ML cluster. So just brief overview, I'm pretty sure you all know, but if anybody missing, the Nexus 9000 is a data center portfolio from Cisco. It's a flagship data center networking. It does run speeds from 100 meg to a 400 gig. And then it does run two operating systems. It does have standalone mode or NXOS. And it does have an ACI mode as well. So you can run whatever you choose. And all of the switches are based on Cisco silicon. So there are certain features which are enabled, particularly on this product line, which we use over here. So let's go to the next component, which is ROC UE2. As I mentioned, RDMA or Converge Ethernet. So if you look at the bottom packet in this picture, you would see that it does have a layer two header. It does have an IP header, which means source MAC, destination MAC, source IP, destination IP, UDP header. And after that, the whole transport is InfiniBand. So you kind of have InfiniBand encapsulated Ethernet packet or frame. So what that provides you, because it's RDMA based, transport based on InfiniBand, you have low latency, high throughput produced from the network from the host itself. What this packet allows is that you can route it, you can switch it, you can do whatever you want with it. And Ethernet network will forward this packet. How this works in a kind of high level end-to-end system. So if you do have an AI cluster using GPUs, ROC UE2 allows you direct memory access and direct memory communication between memory chunks in that GPU cluster, GPU compute nodes. So how this is done through RDMA, NEC is capable directly to go into the, into the memory of the GPU, get that information, put it on the network, send it wherever it needs to go. Next, NEC receives it and puts it into GPU memory. So that's how they exchange. Benefit of this is that you are bypassing the stack, you're bypassing the kernel, and as such you provide low latency, high throughput transport from the endpoint itself. So what is required from the network basically to provide similar capabilities. So in this case, we use RDMA or ROC UE2 in the network for end-to-end communication. What is required from the network? It's non-blocking transport. So you have to create a non-blocking network. Then you have to create a lossless Ethernet. How you create lossless Ethernet using priority flow control, PFC, and ECN, explicit congestion notification. And I'm going to go there in my next slide. So brief overview and brief explanation how all this works. What we have here is we have configured both ECN and PFC to manage congestion. And if I look in this network, I do have three hosts. Host A, host B, and host C. And I have three leaf switches and I have a spine switch in here. What each switch have been configured is to support WRED ECN and to support PFC. So for WRED ECN, we have set these two thresholds on the bottom, the yellow one and the red one. The yellow one is WRED minimal threshold. So just to explain that part as well. So every time that buffer utilization goes over the minimal threshold, the system will go and mark IP packets based on, sorry, it will mark IP packets with ECN or congestion experience bit. So that ECN bits are part of the IP packet in the TOS field, the last two significant bits. So just so we are clear. And then we do have two PFC thresholds, the green and the higher red line. Basically, if your WRED ECN doesn't comply with congestion or doesn't solve the congestion, the PFC will kick in and basically resolve that congestion. So in my example, what I'm trying to do here is that I have host A and host B talking to host C. In this case, they're sending packet, sourced and designated from each of the hosts to the host C. And you will see that they are carrying the ECN enabled bits. So they're carrying ECN 1, 0. This to the network will say, hey, these packets are ECN capable. If you do experience congestion, mark them with bits 1, 1. So in this case, we do congest the port, which goes to host C. And you will see that packets are coming with 1, 0. They reach leaf 3. And then they will be marked with 1, 1. The host C will see this and say, hey, there was a congestion would happen on the path. What will happen here is that I'm going to generate a CNP congestion notification packet. And I'm going to tell to both of the hosts, host A and host B, that they should slow down. So you see that source is host C, destination is host B and host A, both of them receiving CNP bits. If this is, let's say mild congestion, you haven't had a lot of traffic, it's just like a temporary thing. So WRED will solve this congestion. Basically at this time, the host A and host B will slow down and kind of I'm done with congestion in this case. This provides kind of lowest latency for the system or lowest latency overhead. And the system will be able to co-op with the congestion, basically resolve it. However, let's say that our host A and host B in this example haven't slowed down. They still keep pumping a lot of traffic into this network. So what switch three or leaf three will do in this case, it's going to keep buffering and buffering. And eventually it's going to hit our PFC thresholds. Once it hits a PFC threshold, it will stop sending any traffic down to host C. So, hey, I'm not going to pump any more traffic to you. And I'm going to generate PFC, priority flow control packet up to the spine one. So in this case, there is no traffic coming from spine one. So what spine one will do, stop sending traffic and keep buffering that traffic, which is incoming from host A and host B. Eventually, it will go over WRED thresholds. It will mark ECN, but that will not mitigate congestion. So it's going to go over PFC thresholds and it starts sending PFC down to leaf one and leaf two. Basically, we do have a per hop behavior. Leaf three sends to spine one. Spine one sends to leaf one and leaf two. Leaf one and leaf two starts buffering at this point. They go over PFC thresholds and eventually they send pause frames down to the host A and host B. Are there any buffer overlap there then? There could be some. So the whole system is made so both of them do stop because they could contribute to congestion. In reality, I'd have simplified this example a lot. There might not be PFC frame coming to both of them because they might not contribute the same amount of traffic to this congestion. So some of them may not receive any pause frames or any ECN packets and some of them may receive a lot. So there could be that distribution between them. So in this case, host A and host B have stopped. So we have mitigated congestion. At this point in time, the buffers will drain and eventually we will reestablish a full flow of traffic in this case. But the whole point is there is no packet has been dropped in this case and there is no any of the data wasted in this situation. So what I'm going to do, I'm going to show you the demo of this congestion management mechanisms with PFC and ECN. But before I do that, let me explain you the topology and what is happening in the demo. So I do have a network which have been created in the system. So I have six leaf switches. I have five hosts and I have for leaf six or leaf 206, I have attached storage device here. So let's say that our, that my hosts one to five have finished training and they want to write something to a storage device. So all of them start sending to that storage device at the same time. I have storage device connected to, to leaf switch interface one slash 11. This is important for the demo, but all my hosts are connected to the network with a hundred gig connection. And my first four leaves are connected with hundred gig connection to the spines. And then I have 400 gig connection from leaf 205 and 206 down from the spines. So when they start, when hosts one to five start sending and writing that information down to the host, to the storage device, what will happen? They will send to the spine and spine will basically send everything to leaf 206, congesting that port one slash 11. So what I have here is configuration management tool from data center group called Nexus dashboard fabric controller, which will help me configure network end to end. So what I have here is. So what I have here is fabric, which I showed you earlier. So it does have, it does have those six leaves. It does have those three spines. We have all the connections that are present in this case. Um, everything, which I mentioned, um, the main action will be on this leaf 206. So I go to my fabric, I look at the switches and I do find my leaf 206. And what I'm going to do on that leaf is going to show you the QS configuration. So I'm going to go and do show run IP QS, looking at the running configuration for QS. And, uh, when I execute this command, there will be none. So in the first line, I'm going to highlight it right now. The system has absolutely no QS configured. So let's go to our hosts. Um, and I'm going to show you how they're reacting to this as well as how switch is reacting. Um, give me one second. Just gonna zoom in. So we see all what's going on. So here I do have my five hosts on the left. So you will see that they do write something. So W, uh, means I'm writing to my storage device and you see that bandwidth is fluctuating. So at a point in time, they have zero, 600, 1200, 2000 megabits per second. So they're writing at the same time, but very inconsistent. What is happening on, on my switches. So here in the bottom, I do see that switches constantly dropping some, or my, uh, switch to a six on that interface one slash 11. It's constantly dropping some of the packets. Um, why this is happening. I don't have any QoS. I don't generate any ECN packets. I do not generate any of the PFC packets. Um, and this number over here on one slash 11, you see that I have received some of, uh, PFC packets. What that tells you is that my host is configured to do all of this, but my network is not. So host will be saying, Hey, slow down, slow down, but network doesn't understand. So it will not do anything about it. So I'm going to back to my fabric and I'm going to go and configure QoS. In this case, I'm going to go to advanced tab. I'm going to find my QoS configuration, and I'm going to go and select AI cluster QoS template. So a little bit of story about this. What I'm doing here is NDFC. If anybody in audience knows the number. Uh, it's 12.1.2, which is a current NDFC Nexus dashboard fabric controller release. This template is a custom template, which I have created. Uh, however, in the future release, in the next release, which is 12.1.3, we will have multiple templates configured for, uh, multiple hardware devices, because some of them might have a different Q levels and, and to do this in a proper way. So in the future, in the next release, in the upcoming release, we will create a set of templates corresponding to all the hardware, which we have in the portfolio. So I go and I, uh, save this configuration or save this intention. And I'm going to go and deploy that, uh, configuration onto my switches. Um, here as everything which we are doing is on leaf 206. So I'm going to go and check what configuration we will push down to the switch. And I'm going to explain you this. So here, what I have is, uh, my QoS configuration. Uh, anything which is coming with DSCP 24 is my Rocky traffic. Uh, so any, any traffic coming out of the server, it will be Rocky traffic. Anything which is matched by DSCP 48 is my CNP traffic. I want to treat them differently. So my data plane will be delivered in, in fast and reliable way with the QoS. But any congestion notification, I want to reach my sources as fast as possible without waiting for anybody. So I'm going to put it in a strict priority queue, which is my QoS. So if I go scroll down further in configuration, I do have on my QoS 3 configured WRED ECN. So, uh, you see over here QoS 3, um, and then random detect minimal threshold. So that's that yellow line, which I showed you in the previous diagram. Then maximal threshold, that's the red line, lower lab red line in the diagram. And then the last keyword, this ECN tells me, Hey, if, if you do detect any congestion, if you're a WRED and detect any congestion, marked all of those, all of those packet with ECN. Higher here, you see class type Cout 8QQ7 priority queue level one. That's where my, uh, CNP packets strict priority queue going straight down to, uh, the sources in this case. So, and further down, I do have a PFC configuration for Q3. Um, saying for the, um, network QoS Q3, PFC, uh, pause PFC cost 3, which tells, Hey, Q3 needs to be paused. If congestion happens for PFC, um, the thresholds are adjusted to serve the purpose. And how I explained in the previous example. So, uh, PFC have stayed, uh, default. We have put our W thresholds fairly low in the queue. So any minor congestion will be mitigated by ECN and WRED. Uh, any severe congestion will be mitigated by PFC. So what I'm going to do now go to interface. And, uh, here I have told to interface, Hey, enable PFC, which is a priority flow control mode on. Then I'm going to do a watchdog. So watchdog is a feature which we have. Um, and, um, it's served for cases where, uh, PFC storm happens. So PFC storm is that you have malfunctioning a device. It could be a host. It could be a switch, which is constantly sending pause frames. Eventually those pause frames, how I showed in example, will be propagated to everybody in the network. And basically your network will stall. So in order to prevent this, we do have a watchdog, which sets a timer and waits to see if any packets is staying longer than the timer. If they do, they will be cleared out from the buffer. So you prevent, uh, PFC storm in this case. And I do attach classification policies saying, Hey, any traffic coming into this interface or this interface, please treat it. Uh, please put it in the right queue and apply QS on that. So, um, I'm going to go and deploy this configuration and I'm going to go back to my hosts, uh, at this case, in this case, and what will happen here. Um, so pay less attention on the left, more attention on the right. I'll try to zoom in. So this is, uh, clear. So, um, you see that our host is generating PFC. And at this point in time, we do have some of the generate, uh, ECN packets being marked. So remember, I told you first ECN starts, and then if ECN is not capable to resolve this congestion, PFC will kick in and basically get, um, resolve this congestion. So what will happen here? The system will know about and host and the network knows that QS it's configured. So in this case, um, for a moment, hosts will try to gauge what's going on and start sending the traffic. And you will see that ECN continues to, um, to go up. You see that PFC continues to, to be received by the switch and generated and sent to the spine. Um, and then you see over here that there is no packet drop present or the drop counter doesn't increase anymore, which tells you that, that PFC and ECN have done their own job and have prevented any congestion. Um, so we have a lossless network to provide, uh, for that AI cluster. So I'm going back to my NDFC and, um, um, I'm going to show you that, uh, configuration has been applied. What I wanted to push. So I'm going back to leaf 206 and I'm going to do show it on IPQS, similarly to, uh, what I have done before. And, um, um, when I execute there, you will see that same configuration, which I pushed earlier. So we do have, uh, classification policy, uh, sorting out rocket traffic, CNP traffic. We do have WRED configuration on this switch. Um, and then we have, um, PFC configuration saying, Hey, generate PFC for Q3. And please configure PFC watchdog as well classification and QS for, for this leaf. So we do have a set of features, uh, which are present in our Nexus 9000 switches. Uh, the first one will gain you visibility in your network traffic and what's happening in the network. Uh, so the first one is a flow table, or this feature is called flow table, which means system goes and tracks every packet traversing the network traversing the switch. Is able to connect, collect information like five tuple information, flow information, uh, interface and queue information. Um, any, um, um, flow start and flow, flow start and flow stop time. And then it will, um, um, um, indicate if packet is dropped or there was burst, which has happened, uh, with a particular flow. How you can leverage these features is that, um, you can export them using, uh, using hardware. It will be direct hardware, uh, export into our controller. And what we have is Nexus dashboard insights, which is collecting all of those information as well information about PFC interface counters. Uh, basically using, uh, flow, uh, uh, streaming, uh, streaming telemetry from the switches to import that information into Nexus dashboard insights. So with, with Nexus dashboard, you are able to, as I mentioned, look at PFC counters, ECN counters. Uh, you can use those to tune your network basically in a, in a stage where you, uh, establishing the, the cluster. Um, you can understand how those thresholds relate to each other, how they relate to congestion, which congestion they should provide, prevent more, um, um, um. Um, severe, which congestion they can, they can prevent, um, more earlier basically. So, um, what we have done, and this is coming in subsequent release, release in August release, uh, for the Nexus dashboard insights. We do have a nice graph on the interface statistics and, and traffic, which is going, but if you look here, we do have a congestion parameter. Uh, so we do have, um, um, we account for every PFC transmitted packet, every PFC received packet, uh, and any of the ECN, uh, packets. So, um, um, eventually you will be able to kind of correlate congestion, uh, go and drill down on a particular interface and understand what that interface have sent or receive, um, and how it's behaving in the network. And from that perspective, you will be able to, uh, manage that congestion. Um, so what we have done is that we created a blueprint, uh, explaining everything which I have told you in this session, uh, but going in way more details, um, um, giving you way more information. So you can go and, um, um, read this document, uh, we also have created, um, an automation script based on ensemble. So let's say you, you are deploying this type of infrastructure. Um, but, uh, you have deployed, you have automated, uh, provisioning of your endpoints hosts, um, and you have an ensemble script for that. Um, then, um, you want to deploy the network. So you can use this script, um, to deploy the network in the same way, how you're deploying the endpoints. Uh, the script will go and push the configuration down to NDFC and NDFC will go and push it down to the switches. So you have, um, that part automated. And the last part, which I want to mention is, um, customer, um, which have used our next switches to deploy this and have their AI. Uh, their AI cluster working, uh, to, uh, do, uh, sorry, to perform all the functions, which I have explained. So none of this, which are everything, which I have mentioned in, in this presentation is shipping is available, except a few items, which are mentioned. Uh, so you can go deploy this ethernet network for your AI ML cluster. [00:35:48] Speaker 2: Um, and I kind of feel like this maybe would have helped give us a little bit more context in the beginning. You spent a great deal of time talking about, uh, traffic control and congestion management. Could you talk to us a little bit about how, uh, AI clusters generate traffic that would require something like this? I mean, are they, are they, are they, are they bursty? Are they just flood, big floods? Are they sustained? Help us understand why all of what you've shown us is necessary. [00:36:22] Nemanja Kamenica: So good question. Uh, in nutshell, it comes down to what algorithm you will use to, to explain your cluster. So each of the functions of the cluster will have their own capabilities and all properties. Um, if I talk generally, um, a lot of cluster may have all to all type of communications or clusters may have all to all communications, which means let's say I have, um, I don't know, a 512 GPUs, which I mentioned as an example. Uh, and while they are training the cluster or they're, uh, training the model, um, they basically going to come and pull information from the storage device. Uh, that data, which they use to train the cluster. Um, once they pull, they have something which is called data parallelization. So what I do is that, let's say I picked up a picture, which is a dog or whatever thing it is. Uh, each of us, each of 512 GPUs, we're going to go piece of that, uh, that picture, and it's going to process it. So I'm training, let's say this, this cluster to recognize eyes on a dog. I don't know, like, let's put that as an example. So a picture is of a whole dog. So somebody will get tail, somebody will get leg, somebody will get belly, and somebody will get eye. And they're going to find out that eye. Once they kind of go get each of them get that piece of information, they're going to look for it. And majority of them will respond to each other. Hey, I don't have it, which means that data or that information is sent to everybody in the cluster. So as such, you might have, so definitely pulling it from, from the storage device, it's going to be somewhat bursty, but you're going one too many. So that's kind of traffic pattern. Uh, then once I have got my piece, um, I did calculation, I figured out where the eye is or that I don't have an eye. I'm going to update to everybody else. So I'm going to send a packet to everybody in the cluster. Um, as such, I need to have like a rich connectivity between those clusters as well. Um, so the next, uh, point is that those who have recognized the eye, they're going to go and send to everybody else. Hey, I do have an eye. So we can update the model. This is how I recognize that eye, uh, that model will come together and it will say, Hey, sorry, the cluster will come together, conclude where the eye, what are the parameters for that eye of the dog. And it's going to go back and write that information to the model, which is again in a storage device. So all of them come together and write and push that information down to that single storage device. So as such that model is, is done, the operation is done for this particular example. So I can start, um, I can push a new picture, do repeat that same operation again, and again, push it down to the storage device. So this operation happens many times per second. Um, and the whole training, um, can last multiple weeks even. So how does everything which I have talked, uh, in this session about. So imagine you are like a hyperscaler and you are doing very, uh, uh, complex learning or training of that AI cluster. Um, so what will happen? Your training happens for, or lasting for, let's say, a two days, even short period of time. But sometime first day, like after 12 hours, congestion happens in the network. Let's say it happens in the network. You drop some amount of traffic and your, uh, cluster stop working. Basically it drops down. So you need to know that cost of that each GPU. Uh, let's say you had five, 12 of them. They work for, uh, for 12 hours. Each hour costs you, let's say $20. So calculate the, the expense which you just wasted. So if you do have, if you have designed the network in a proper way, um, you basically preventing yourself of that money wastage. And you will be able to finish that operation in, in a proper way without, um, without implication of the network. [00:40:49] Speaker ?: And you will be able to, um, without implication. And you will be able to, um, without implication of the network. And you will be able to, um, without implication of the network. And you will be able to, um, without implication of the network. And you will be able to, um, without implication of the network. And you will be able to, um, without implication of the network. And you will be able to, um, without implication of the network.