I Can't BELIEVE They Let Me in Here!

[00:00:00] Speaker 1: - If you already know who Equinix is, you know the treat you're in for today. For everyone else, Equinix, who sponsored this facility tour, was founded in 1998 and currently operates over 260 data centers globally, through which flows over half of the tier one internet backbone. In a nutshell, if you're using a computer right now, which you probably are, like this guy, you are more likely than not touching Equinix infrastructure right now. And guys, I get to take you deep behind the curtain. I'm talking power, data, cooling, security. It's pretty much everything that we do on our channel. Yes, even pool server cooling, except legit. You see, as pretty as it might look, the building behind me was purpose-built from the ground up to be a function first data center. And that manifests in a lot of interesting ways. For starters, it was not built from the ground up, but rather the first floor was lifted a few feet off of ground level. Why? Security. Like any Equinix facility, that building needs to be able to withstand a breach attempt from a vehicle ramming into it, but the city wouldn't let them surround it with ugly barriers. So instead, they lifted up the foundation itself. So if you drove anything less than that city bus into 45 Parliament Street, Toronto, you would eventually regain consciousness and find the building to be quite unbreached. Honestly speaking, you'd probably have better luck getting inside if you just walk through the front door. JK, once you get in there, you'll find yourself in a series of airlock-style doors like this one with at least three biometric security checkpoints before you so much as catch a glimpse of a customer rack. Ooh, but I get to go. But what's the big deal, you might think? It's just some servers and network switches, right? Well, yeah, but really important ones. Equinix houses critical infrastructure for everyone from telecommunications providers to cloud service providers to securities exchanges and even governments. To put it in perspective, some poor schlub accidentally overcooked a bag of microwave popcorn at this location, and the city sent seven fire trucks on the off chance that Equinix was on fire and hadn't already managed it with their own fire suppression systems. For more context, their service level agreement with diesel fuel providers is on par with the local hospital because, well, think about it. Realistically, is that hospital getting anything done without data services? Probably not. Let's head up. Here's something cool. Everything in this building, except the generators we're about to see on the fifth floor, was brought up in this freight elevator. It has a fully loaded capacity that's equivalent to about four loaded Mini Coopers. And this is really important for cloud providers who pre-build their racks offsite. It allows them to wheel a rack into their cage, hook up power, hook up cooling, and walk away. We'll take a look at some of those cages later, but first we need to go all the way up to the fifth floor. Where possible, Equinix uses a top to bottom approach to building design. So we're gonna go in that order. Why do they design top down, you might ask? That's because if you spring a leak, it's gonna go down. So this floor contains everything that's needed to run the whole building in the event of a major outside interruption, and nothing can leak on top of it, which is especially important for this. Inside the aptly named battery room is enough capacity to run the entire building at full load for up to 15 minutes. And if they need more time, that's where these come in. These diesel generators are designed to run continuously at a capacity of 2.5 megawatts each. Yes, my friends, 2.5 million watts each. Getting these up here, by the way, took some special planning, which I'm told involves cutting a hole in the side of the building, renting a crane and then sliding them in. Absolutely badass. But there's one small problem. Now they've gotta fuel them. I mean, they've got this little reservoir here, but this would run one of these bad boys for what? An hour? Well, as it turns out, the main tanks, which include an onsite polisher to remove water and other impurities that can spoil the fuel, those are underground. And this was done because diesel fuel is lighter than water, and Equinix wanted to design this building such that, if the entire first floor was flooded, the tanks wouldn't float away. And wicked smart. The last thing they had to manage was the noise. Oh, the noise, noise, noise, noise. Now, obviously, Equinix isn't gonna hit their 2030 renewable energy goals by running diesel generators 24/7, but they do need to perform monthly checks on them. And I'm told that when you fire those things up, you can't even hear yourself think in here. Now, given there used to be a public library on the other side of this wall, that was a problem. So, Equinix designed these acoustic baffles that bring the noise down to the same as a typical city road. And you can... Oh, wow. You can probably hear the difference, right? When I go in there? Oh, yeah. Crazy. Kind of necessary if you consider that this is about the equivalent of having five bloody trains running in here. Fun fact before we leave this room, by the way, diesel generation kicks off enough smoke that conventional smoke detectors would be giving off constant false alarms. So this room uses flame detectors, special cameras that can see through smoke to see if there's actual fire. Moving into the chiller room, we've got the exact opposite problem. You can fit so much water in this bad boy. That is why absolutely no customer infrastructure is allowed under this room. In here are three main cooling loops. The one behind me is filled with antifreeze that goes between these liquid-to-liquid heat exchangers and the large evaporative cooling fin stacks that are up on the roof. We're going to take a closer look at those in a minute, but for now, I want to look at the other loops. Those two are connected to what is, in effect, the biggest frickin' air conditioner that I have ever seen. Except instead of cooling air like your air conditioner does, it cools water. The hot side here carries heat to those same heat exchangers that we just talked about, while the cold side is connected to the rest of the building through what Equinix calls their bus architecture, which is kind of just a fancy way of saying that it's all redundant, it's all interconnected, and it's all controlled by a BMS or building management system. And I'll show you why that's cool. Right above me is our chilled water return, which is bringing heat up from the data center floors below. Okay, so far, so simple, but where does it go? Well, as you can see, it gets a little complicated. When it's below four degrees Celsius in Canadian winter, it essentially passes straight through with the compressors barely running and goes to the heat exchangers on the roof. Sweet! Free cooling. Thanks, Mother Nature. But then spring rolls around and the outside air needs some help. That's when our highly efficient magnetic bearing compressors ramp up their cooling efforts, bringing our thermals under control. But wait! Now it's midsummer and there's a heat dome! We need more cooling, Kevin! Thankfully, we've got it! Now, most of the time, the misters are not needed. But for a building like this, you don't design for most of the time. And these rooftop fin stacks can either be run bare or they can be lightly coated in water to dramatically increase their cooling capacity. It works in the same way that your arm gets much colder if you lick it before you blow on it. The best part is that everything we've seen is fully redundant, including cold spares of key components. That gives Equinix's team lots of options for load balancing during maintenance. And it means that in the event of an equipment failure or another event, Equinix won't risk giving up their nice 6.9s uptime target. That's an average of about 32 seconds of downtime per year, by the way. Wow! Hilariously, sometime in the near future, All the cooling we just looked at will be rendered redundant, though, when the fourth pipe goes online in the DLWC or Deep Lake Water Cooling System. If you haven't heard of it, it's basically my pool water cooling setup, except done properly with massive pipes that draw 4 degree water from the bottom of Lake Ontario and then pump it into the city to cool office towers and arenas and, well, data centers, that sort of thing. Super cool project. Since we're up here, by the way, these freaking blew my mind. Now as consumers, when we want the lowest latency, say for gaming or video conferencing, we think fiber, right? Well, what if I told you that there's a tier above the speed of light? Well, you'd probably tell me there is not. And all right, you got me. But here's the thing. Light does not always go the same speed. But here's what's going on. And fiber optic cable slows it down to about two thirds of its theoretical maximum vacuum. Now, that's not enough to matter for your game, but for capital market transactions, where microseconds can be the difference between making your trade and losing a huge opportunity, every competitive edge matters. And that is where these microwave arrays come in. These provide direct line of sight links north to the Toronto Stock Exchange and south to New York. Now, of course, for anything where throughput matters, nobody's using these. So let's go back inside and look at good old fashioned fiber. By the way, you guys might have noticed some extra pads and footings for additional equipment up here. That's another nice thing about a purpose built building. You can plan out your future capacity and then you can build it as needed. This, for instance, is a spot for another fin stack. And here in the electrical room, you'll see much of the same thing. Look at the glorious extra conduit for future installs. Isn't it beautiful? This room takes in 13,800 volt power, steps it down and distributes it throughout the building. But all right, enough about infrastructure. How about some customer systems, right? The next two floors are very similar. So let's go with number four. This is the oldest floor and therefore uses some older design paradigms. For starters, cooling is handled using cold aisle containment rather than hot aisle containment. And let me show you what that means. The water from above us is getting pumped into this air handler corridor that chills the air. Chilled air then gets carried through ducts to the center of the customer's cage, where a thermal probe will inform the system when cooling is needed. It opens up the motorized vents and then blasts cold air downward. It then gets sucked up into the rack, which is going to be on either side, and then exhausted out into the room where it gets sucked into that corridor. Rinse and repeat. Now, the main drawback of this approach is that it allows some mixing of the hot and cold air. But Equinix still managed a solid power usage effectiveness of 1.42 for this site. Since you're wondering, by the way, I should probably acknowledge that this is an unleashed space and there are normally strict rules about access to these cages. For example, exchanges cannot have their cages entered at all during trading hours. And even for Equinix staff, unless there's an emergency or contracted work that needs to be done, they are not supposed to be entering customer spaces. So they built these cool access panels where they can run any connections that the customer requests to the outside of the cage and then the customer can come in and route whatever they need for whatever they're doing. Access to the cages is controlled by access cards and biometrics. Either a hand or a fingerprint that's stored as a hash for customer privacy. And any use of your card is logged on top of the logs for the biometric sensor, the contact sensor on the door, the motion sensor near the door, ah, there it is. And of course, the camera pointed at the back of the rack space. And if all this seems overkill, you ain't seen nothing yet. Many of the cages not only have physical access barriers, but even visual blockers so no one can see any of the equipment in use. And for the extra paranoid, a top cage can be installed to ensure that nobody can climb over the top. Though, of course, anyone trying to do that would have been spotted by one of the hundreds of cameras in this place. But then, as you've probably picked up by now, in this business, redundancy is the name of the game. Some cages go even further, requiring what's called double access. Meaning that to get inside, you need both an Equinix employee and a customer employee. And we even saw one with a touch sensitive cage. So if you so much as leaned against it, a ticket would get automatically created and Equinix security would get an irate phone call. But wait, there's more. They told me that some customers will even post a guard who stands outside their cage 24 hours a day, seven days a week. Financial institutions and government agencies will often operate with a high degree of security like that. Speaking of financial institutions, certain customers have noticed that if they're closer to each other, they can have a physical advantage in connection times. Well, that's not necessarily a good thing. To solve that problem, then, Equinix has a system called equidistant trading that neutralizes any latency advantage by running the exact same length of fiber to every customer who requires that. The spools of excess fiber optic cable are one of the most protected things in the entire facility. Now, I was able to convince them to open the door earlier for a quick B-roll shot, but they wouldn't let us any closer. There's a camera pointed at the cabinet. And if anyone so much as goes near it, security shows up and it's a whole to do with reports to the SEC and the whole kazoo. Moving on to the second floor, this place is sick. Now, we weren't able to shoot as much here due to customer privacy concerns, but we did manage to get some footage of one of the liquid-cooled cages. Here, building cold water goes into a heat exchanger that pumps chilled water to their high-density servers. These kinds of deployments are becoming much more common, but they're kind of an ongoing challenge for standards lovers like Equinix at the moment. These guys dial in flipping everything down to the colors of the metal posts in their facilities. By the way, blue means you hang stuff off them and gray means they hold stuff up. The more you know. But the lack of standards in the liquid cooling space right now means that these cages often need special provisions. And that's something that they're working with others in the industry to solve as rack power density continues to scale to ludicrous levels. Looking at you, AI. While we're here, there's a couple other cool things about this cage. There's leak containment troughs in the ceiling under any water pipes to ensure that even a small leak will be caught and can't fall onto the equipment below it. And if you look carefully at the floor, there's actually little yellow ropes running around the outside of this cage. And here on the first floor, I see another one. Now, I wouldn't blame you guys for thinking, gee, it seems uncharacteristically sloppy of these guys to leave this kind of thing lying around all over the place. But it's actually not a rope. They are everywhere in this facility where there's water and they're part of a leak detection system that tells their team not only if there's been a leak, but exactly, not only if there's been a leak, but exactly, and exactly where it was. Boy, would that ever have been handy. Boy, would that ever have been handy. Boy, would that ever have been handy. Since we're here, by the way, this happens to be the newest floor in the building. Remember, top to bottom build out, right? This one was done about a year ago and they refer to this data hall as a Mark 1 design. But now that AI has exploded power densities to numbers that would have been unthinkable even two to three years ago, the Mark 2 designs that they're currently constructing are aiming at around 10 times the power per square foot that they targeted here. So maybe that's something we could see in another video. Equinix and Nihoo, this floor uses a very different system for handling air. Instead of pushing cold air to the cages, now they pull hot air out of the cages into these chimneys that go up into the plenum, which runs into the air handlers. Now, it might seem like an unimportant distinction, but the higher the temperature difference, the more efficient the thermal transfer. And when you're dealing with very high draw racks, this ensures that the hottest possible air goes into the handling. It also all but completely prevents hot and cold air from mixing, which is a major win for efficiency. Now, I can practically hear you guys shouting, if you've got all the hot air in a chimney, why don't you do what a chimney does best and blast it out of the building? Well, you can do that, but then you need to bring in air to replace it air that needs to be filtered and that needs to be humidity controlled to handle that at the kind of volume that you would need. It's impractical. So, even under very extreme conditions, like we saw last winter in Chicago, I'm sorry, but you can't just swing open the doors of a data center and hope that it'll run cooler. By the way, we didn't show this on the other customer floors, but there's actually two equivalent air handling systems on opposite sides of each floor, each with the capacity to handle the entire floor in the event of a failure. By the way, before we leave the customer systems area, I was very surprised to hear that Equinex does not use exotic fire suppression methods, opting instead for standard water sprinklers. But then I found out they're not quite standard. See, the pipes are not flooded with water all the time, but rather they're full of inert gas and water only flows into them if two out of three systems go off and a plastic head melts. So they use heat detection, smoke detection, and the world's most expensive PVC runs. These are part of a VESDA or Very Early Smoke Detection Alarm System that can not only detect smoke, but also triangulate where it's coming from to ensure that only the necessary sprinkler head goes off. They made this choice because oxygen depletion fire suppression can pose very real risks to any personnel who get accidentally trapped. So, good guy Equinex, let's go to the basement. Above me is where utility power comes into the building. Or rather, this is one of the two rooms because redundancy, right? Now, interestingly, the power company can actually control this unit remotely, allowing them to turn it on and off and even scale how much power comes in. That is not a normal thing, but because there's so much power going into this building in an emergency, they might need to call Equinex and say, hey, we need more power. Could you guys just turn all of this off and island yourselves? This kind of collaboration is really important because Equinex estimates that this building draws somewhere in the neighborhood of 0.3% of the city's entire power budget. If a multi-megawatt load kicks in suddenly without warning or drops suddenly without warning, it is a big problem for grid stability. With that said, they do turn it off and on regularly, though. Every year, they do a scheduled pull the plug test. And quarterly, they do load tests where they will bring in load banks to ensure that the building can actually power everything that it's rated for. And these things sound pretty cool. Basically, if they don't have enough servers running, they bring in big blocks of metal and they fire up the generators and make them glow red hot. Flippin wild. Finally, there's the fiber in room. It's funny. They thought that this was going to be underwhelming and they were all like, are you sure you even want to see it? But it really goes to show you how numb you can get to cool stuff if you spend your whole day around it. Now, in fairness to them, they are kind of right. I mean, it pretty much looks exactly like the fiber room in our building back home. Line comes in from outside. Line loops around, as is tradition, and then gets spliced or patched to wherever. The difference is that I could probably fit my entire family hula hooping inside these freaking loops. And where they go to is even cooler. They go outside to one of the seven manholes that get shared by the approximately 60 carriers that route into this building. That is where the carrier's access ends. Equinix splices runs from the manholes to this room and then they run it up to the floors above. The carriers only get access again once it reaches their cage. This is how Equinix enforces the principle of equity of access that inspired their name. Oh, and of course, there are two of these rooms with fiber coming in from two different sides of the building. Because in the event of a construction worker accidentally cutting one of these lines or something, they got to have redundancy. I got to tell you guys, there is so much we weren't able to fully cover. If you noticed QR codes on everything, those are cool. They use AR to direct technicians for fault discovery and to map out customer connection requests. The 18-wheeler dock can be completely closed off for privacy and security when you're unloading racks. They have employee facilities on sites when an emergency. You can shower, shave, sleep, you know, your three S's. They turn off the lights whenever no humans are detected to save power. They've got grounding strips freaking everywhere. They have direct links to their other buildings to turn their entire flock of data centers into a giant redundant mesh. Guys, there is so much more, but I think it might have to wait for next time. For now, massive thanks to Equinix for sponsoring this video and giving us such an in-depth look at the kind of space that most people are never going to set foot in, let alone get a comprehensive tour of. Hopefully we'll get a chance to do it again sometime. If you guys enjoyed this video, hey, you might enjoy this kind of high security, nobody gets to see it kind of stuff. So maybe check out the Intel fab tour that we did last year.