Exclusive INSIDE the Secretive AWS Networking Lab

[00:00:00] Speaker 1: We got to see switches up to 1.6 terabits per second per port and other ones that could encrypt links at 400 gigabits per port. There were custom one gig management switches, serial consoles, and even custom timing. We saw devices that could jam multiple 800 gigabit per second links into fiber spanning over a thousand kilometers. We also saw how you can connect 64 fiber links in like a second. We got to see all of this and more when we got to go tour an Amazon Web Services networking lab where they make all kinds of crazy innovations happen. Sam and I had to get on a plane and travel to a nondescript building that you'd have no idea that all of this innovation was happening. So we have to say that AWS is sponsoring this video. With that, let's go check out some crazy switches. Now behind me, I have AWS's switches ranging all the way from the 100 gigabit per second per port all the way up to the 1.6 terabit per second per port generation. This right here, this blue one, is a 3.2 terabit per second switch, which means you have 32 ports of 100 gig QSFP28 on the front. And this is really now an older generation switch. Now something really cool is what happens when we move from the blue 3.2 terabit per second switch all the way to the green 12.8 terabit per second switch. Now on this switch, all of the ports have migrated from a 100 gig to a 400 gig platform. You're going to see that we have QSFPDD and this is something that is deployed, I think, pretty widely at AWS right now. Now you're also going to notice that inside a lot has changed. Now of course this generation moved from NRZ up to PAM4 and you can see the switchboard, which is always kind of neat to see. You can see also, because we have a higher power switch chip, you can see that we have a larger heatsink on that as well as larger heatsinks on the QSFP56 cages. Now one other just kind of fun thing here is you're going to notice that inside the switch, there's a lot more going on. Because then you're going to see that we have a fancier Annapurna Labs management processor, you're going to see that we have our SSD, and then this board right here, which is something that AWS tells me is one of the keys to actually getting higher reliability in the newer generation of switches and optical modules, or at least in their operation. This board over here communicates with all of the optical modules using I2C and it's able to go and get telemetry data so you can figure out if a device might be having an issue. And also if a device is having an issue, it doesn't impact all of the rest of the optical modules that surround it, which makes it a lot easier to go troubleshoot when something's going wrong. It may just look like a PCV, but I've been told that this is a huge innovation at AWS. Let's keep moving on though to the new switch, which is over here. Now if you're watching this video when it comes out, this is what AWS is really deploying a lot of right now. This is a 51.2T generation switch. You'll notice on the front that we have our OSFP and this is using, of course, PAM-4-112, so it gives us OSFP 800 gig ports on the front. Then we have a much larger heatsink because that I2C board that communicates with the optical modules, there are two of those under this giant heatsink along with the switch ASIC. Now, when we move behind there, you can see that we have our management processor. In the back, we have our power distribution and our fan control. One other thing that you might notice is that this is a silver switch, whereas we showed you blue and green. This one, I'm told, is a peach color and I'm going to tell you about the next one after this. Because after the 51.2T generation, we get to the 102.4TB per second generation. Now, each one of these OSFP ports is using PAM-4-224, which means that you get a 1.6TB per second OSFP port. And of course, since we were pushing 16 times the traffic over each one of these ports that we had on the first switch that we looked at, and we have twice as many ports, you're going to see that we have a much larger heatsink again. And also, just a fun thing that we've covered a couple times on the STH main site is that the optical modules actually have their own cooling built into them. Now, as we're filming this, this is a really cutting-edge switch. This is the kind of thing that will be deployed over the next quarters and years, but this is definitely something that you can look at. And you can see that there's a lot of similarities between this switch and the 51.2T generation. You're going to see that we have our management module. Then behind that, we have a much larger power distribution as well as a fan controller board. And when these are deployed, the actual ones will be some kind of like whitish color. So we'll go from blue to green to peach to a whitish color. Now, there's another one that we should definitely show you, which is this one over here. Now, this is the 12.8T generation Maxx switch, which means that each of the ports is encrypted because, of course, you'd want to go and encrypt all the data that's leaving your data center. These also handle much higher power optics because they can take 400 gig ZR optics. We've, of course, shown 800 gig ZR, ZR plus optics on STH. But the same idea where you want to go long distances using fiber, you can plug the modules directly into this type of switch. And, of course, everything's being encrypted. So something you'll notice is that there's a lot of similarities between this and the green 400 gig 12.8T switch. But some big differences, of course, are that this is a two RU switch. You're also going to notice that the management processor has been flipped and there's 16 FIs that handle all the encryptions. And so that means that you have, of course, more power and that means you need bigger cooling and more cooling. And that's why you have a two RU switch. Now, every good network needs a management network. And that's really what we have over here. This is AWS's one gig management switch. And, of course, you know, a one gig management switch is not like otherworldly these days. But on the other hand, it is important because if you don't have that management interface, you don't have that telemetry data, then you can't operate a highly reliable network. Inside, you're going to see that you have your PHYs, you have your transformers, you have a switch chip and then your management processor and, of course, a log SSD as well. Very similar, but just in a much lower bandwidth package you can see in the higher speed switches. Now, another super important part of any management network is, of course, your serial console server. And AWS makes their own one of those too. And that's what you're going to see right here. So even though AWS is building some of the most technologically advanced switches and going at the pace of technology, which is going crazy right now, they're still using 1970s technology in some ways to go and reliably manage their network. Okay, this next to me is a timing appliance. And the purpose of this is to keep accurate time among fleets of thousands upon thousands of servers. The idea here is that you get accurate time coming from satellites that have accurate clocks within them. And then you need to distribute a pulse every second to all of the downstream server devices. And so to do that, AWS has its own timing network independent of all of its other networks that runs through appliances just like this one. On the front, you're going to see that there are 48 ports. And you'll see that these are SFP ports, but they're not Ethernet. Instead, this is solely AWS's proprietary IP on how they distribute time. And the reason that you need accurate time in a large infrastructure like AWS is because you need to ensure that whenever you do transactions or other types of operations, and even just looking at logs, I guess, you would need to know when things happen very specifically so you know the order in which things happen. And so, for example, if you have a transactional database or a distributed database that's reading and writing from different nodes, you need to know when something was written so that way you know if you're reading the most current version. And so if you're just using plain run-of-the-mill NTP, you might have accuracy within about 500 microseconds. But AWS, with this timing appliance and its entire separate timing infrastructure can guarantee time down to 20 microseconds. And I've heard that that might be pretty conservative. So on the front of the timing appliance, you're going to see that we have our UART, our console, as well as our management port. Then you're going to see that we have multiple antenna inputs. And the reason for that, of course, is that we need redundancy throughout this entire thing. So something happens wrong on one side, we have redundancy. Then there are four upstream ports, as well as all the rest of these ports are downstream ports. So that way you can create a tree topology and distribute time to all of the devices downstream. And so taking a look at the actual board, you're going to see that these two FPGAs are really focused on the I2C. So making sure that we can have the optical modules and can monitor those. There's another FPGA here, which runs all of the proprietary logic to make this thing work. Then behind it, we have our SSD. We have our Annapurna Labs processor. And then we have our power distribution. We have our fan control board back here, followed by our fans and our power input. Now, AWS also makes its long haul optical networking gear as well. So instead of just buying something off the shelf, instead what you're going to see here is the DWDM for dense wave division multiplexing modules that they have. Now, just to kind of give you an idea, this is the 400 gig per lane generation. This is the 800 gig generation. So what you're going to see here and over here is the client optics port. And you can think of this as these are the DR4 optics that are going to the switches that are just in the data center. But then when you need to go a much longer distance, not just, you know, 10, 20, 30, 50, 100 kilometers, you need to go thousands of kilometers, for example. What you need to do is you need much more powerful optics and you also need a different encoding. And that's exactly what these do. So they go from the client side and they go and they convert it to the long range optics. And that uses 16 QAM, which you're going to see over on this laptop over here. This is actually what it looks like. So instead of just the normal eye charts that you see with either NRZ or PAM4 signaling, which would be what the client side uses, this uses the 16 QAM signaling. We have another video on STH on ZR plus optics, but the really cool thing here is that Amazon is able to put up to 64 different wavelengths on a single fiber. So when you are going a long distance, you only have so many fiber strands that are actually connecting different data centers and different sites. And so it's important to be able to put as much information on each strand of fiber as you possibly can. And so this generation over here actually does 800 gigs per wavelength. You can put 64 wavelengths onto a single fiber and that allows you to get massive bandwidth that spans long distances. And that also means that Amazon needs to have a number of different types of modules all the way from lower end 10 gig optics, all the way up to 1.6 T optics, and then into the ZR plus modules. Now, when we say that AWS makes almost everything, that includes even down to the fiber because AWS is using hollow core fiber on some of its longer haul routes. And what do we mean by hollow core fiber? Well, instead of going through glass and putting an optical and light through glass to transmit data between data centers, instead, they're using a hollow core. And that means that light is just going through air. Now, let me show you a little bit of how this works. So this is your normal single mode fiber behind me, which you see in a lot of data centers and AWS frankly uses a lot of, and I don't really think that that's too exciting, but the hollow core, well, that's over here. And that's really the exciting stuff because the hollow core fiber is 125 micrometers wide, or the diameter is 125 micrometers. And so instead of trying to cut that up and show you guys, we have this nice little handy model in front of me. Now, this handy model shows you what the structure of the fiber looks like. You'll see that there's all these structures, but really the big part is the center portion because that's the center channel where all the light passes through just air. And in case you're wondering, these little structures around here, these are the VNF structures that help with all the reflections and all the kind of weird optical properties that you get when you try putting light through a cable. Now, let's go take a look over here at some of the implications of that, because you'll see that we have the VIAVI tool over here, which shows that we have a little segment with the reflectivity of the single mode fiber over there. Then we have the hollow core fiber, which is this little dip. And then we have single mode again over here. Now, the reason that a company like AWS uses hollow core fiber is because of what it allows them to do. Within AWS, there's multiple availability zones and those availability zones all have to have tight latency requirements within them. So hollow core fiber gives you about 33% better latency than if you go and put light through glass, which is the typical way you would do it. And so by having lower latency, it means that you can, as AWS, design your data center footprint to have a wider radius because, well, you have lower latency on your longer haul links. Now, while AWS is not deploying this everywhere, of course, inside data centers and what have you, still single mode fiber. And also, if you have a data center that's close by, you can still use single mode fiber, and that's really easy. But for those data centers that are a little bit further away, they can use hollow core fiber and have been deploying this technology since 2024. AWS also has to innovate in terms of how they deploy fiber and their connectors and all that kind of stuff. It doesn't just stop at the switch. So let me just kind of show you a couple of things, right? These are the new MMC connectors. And you see that there are four MMC connectors in here. And this is a normal loopback device that you might see where you're just kind of doing the RX and TX sides and looping those back. So that way you can tell, you know, is the fiber connection, does it work? And all that kind of stuff. And this is what something would look like, but you have to actually go and manufacture it with all these little loops. And that's just kind of painful. So instead, AWS has this. This is called the Firefly device, which is a molded solution where you have a, not just a loopback without all these little fibers that are expensive to manufacture. And this little Firefly device, not only does it provide the lens-based loopback, but it also protects the fiber during transit. So you don't have something go and jam up on this and break the connectors, or you don't just get dust in it. So to get this off, you actually just pinch it like this and out comes the ganged MMC connector. Now this has a total of 64 fibers. And one of the cool things about this is that if you want to install it, well, you can install it all at once. And let me show you what that looks like. So let me just kind of show you what installing this looks like. You plop this in here, you latch it like this, and I've just connected 64 fibers in like a second, right? And let me show you what this connector looks like if you were using old school LC connectors, which are still widely used in the industry. So instead of getting 64 fibers here, you could just use LC connectors, in which case it's gonna take this much room. Look at how much more room that is. So when you're trying to get much higher density of fiber, well, obviously it makes sense to go to higher capacity and higher density connectors like MMC using the gang connectors, so that way the installation is much more reliable and faster. And so that's one of the big reasons that you have something like this. And one other cool thing is that even if you install these as a 64, a ganged MMC, you can also just service each one of the four MMCs here by just plopping them out just like this. And so you can still service them individually if you need to. And what all of this does is it allows AWS to deploy fiber quickly, test it, and make sure it's ready. So that way when the gear like the AI servers, the standard CPU servers, any of the IT gear that comes in, that expensive gear isn't waiting for the network to get installed, tested, and ready to go. Instead, AWS has the network ready for the gear to arrive. While AWS data centers are controlled environments, what you see behind me is the walk-in thermal chamber. Now, it is darn loud in there, you're going to see some B-roll about this. But the idea is that AWS is able to take components that are manufactured in different sites and at different vendors. They're able to put them together end-to-end and then do a lot of the temperature and humidity, thermal cycling and humidity cycling to see how they would respond if environmental change happened. This lets them test scenarios that, while unrealistic, give some like really crazy boundary conditions onto what something might see in the field. And just in case it happens, they've already tested it. Now, whenever we do these tour videos, it always takes a lot of folks. So I want to say thank you to all the folks at AWS for making something like this possible. But also, if you want to get involved in building the next generation of infrastructure, well, I bet you when you watch this, AWS is going to be hiring. So go check it out. Hey guys, I hope you like this look at a crazy hyperscale networking lab. This is something you don't get to see every day. And so if you want to share it with your friends and colleagues, well, definitely go do that. But also give it a like, click subscribe and turn on those notifications so you can see whenever we come out with great new videos. As always, thanks for watching. Have an awesome day.