A Fun Data Center Tour at PhoenixNAP

[00:00:00] Patrick: Hey guys, this is Patrick from SDH and today I am in Phoenix, Arizona to take a look at a data center of PhoenixNAP. At SDH, we review server storage and networking gear all the time and that gear really resides usually in data centers. So I thought, well, why don't we go and take a tour of a data center? Wouldn't that be some cool content? And the challenge there is really just the fact that filming in data centers is usually not allowed. Now PhoenixNAP wanted to show off their facility, so they managed to sponsor getting us down there. But as always with SDH, this is being done completely independently. They're not getting to see this video before it gets published or the accompanying main site article. And specifically, you're going to see that we're taking a tour with Frank. And then also we have some interview questions with Ian McClarty, who's the president of PhoenixNAP. Given the fact that this is done during COVID times, I'm actually sitting just off camera and just know that these are all, you know, since we do this editorially independently, these are all my questions. They're not, you know, their questions that I'm just parroting back or whatever. And the first question I had, and I'm sure a lot of you had the same question is like, [00:01:09] Ian McClarty: okay, so like, Ian, why Phoenix? Phoenix is actually one of the largest data center hubs in the world right now. Space is relatively inexpensive here. The tax rates are very viable here. And we also have a disaster free zone. So no major earthquakes or hurricanes. This is why also a lot of good data [00:01:27] Frank: runs through the valley. We actually have two separate fiber vaults coming into the facility. We have one here on the south entrance and we have one on the north entrance. There's two entrances in here. We weren't able to put a fence out front for the customer parking. We put these bollards in. These are still reinforced, go six feet down, right? So no one can come into the facility. If we look over here, this is our guard booth. This is bullproof glass. It's Kevlar line. And it's also sandbags. [00:01:55] Patrick: Now, James's security guard here is a very nice guy, but he's also a very large human being that he didn't want to get on the wrong side of. So we did manage to only film for a short period of time in [00:02:06] Frank: the lobby before we moved inside. If we walk this way, we'll see our NOC, our network operations center. Due to the pandemic, right? Most people working remote. We actually have staff here. We have staff in Atlanta, Ashburn, Virginia, Chicago, Amsterdam, Singapore, Malta, and Serbia. We kind of follow the sun when it comes to support staff from our, you know, supporting our customers, right? Okay. So we're at our entrance mantra. So we'll scan one at a time. When we scan, this will turn green. Which turns green, you can go in. It has anti-tailgates. More than one person goes in, the alarm sounds, security shows up. [00:02:43] Ian McClarty: We don't want security showing up because we saw them earlier. They carry tasers. What makes PhoenixNAP different is that we're not a real estate company. We are a data center operator, and we are a really good network hub, but we are a web scaler at heart. I mean, that's who we are. We have a lot of the core component pieces that a hyperscaler has. But on top of that, we also offer a more intimate engagement. You can actually talk to a person. You can actually pick [00:03:06] Frank: up the phone. It's been said that we're the most connected data center in Phoenix. Yeah. Right. And the reason you would want to be here, the reason it would make a difference to you for an Amazon direct connect, is you actually have a direct connection right to Amazon, right? You're not [00:03:18] Patrick: going through another pier or anything else. The AWS direct connection cage was one area that we just could not film anywhere near. And so, you know, there are some restrictions and AWS is very private in terms of their infrastructure. And so we just couldn't film that. Sorry, guys. [00:03:32] Ian McClarty: We have a true network hub. It is the place where all the big enterprise customers that do high bandwidth services come in. We service two of the major cloud on ramps. So one for AWS and one for Google compute. We service those very well. We also service the supporting infrastructure around that. So people that are like mega port and packet fabric that are carrier grade and are really the next level cloud on ramp as well. So when it comes to the connectivity side, and if you think of like edge from a carrier perspective, [00:03:59] Frank: we are the place to be. We also offer blends. So we have a blend that we can make up multiple carriers. So let's say there's a fiber cut in Seattle, LA, wherever, right? That may impact one carrier. That's fine because our blend will have multiple carriers in it. So you'll never lose connectivity from that perspective. Yeah, Joe got you now taking pictures. So this is our water ionizer roadie system that takes all the impurities as well as all the connectivity out of the water. And the reason [00:04:28] Speaker 4: that you have that is really to make sure that you don't have impurities that will corrode or build up [00:04:33] Frank: inside your pipes, which keeps the water flow. Exactly. Yes. Very important. All right. So now we're going to go on to the data center floor. It may be a little loud out here, so hopefully we'll be fine. All right. As you can see, all the connectivity, all the powers run above. We have a lot of clients that will want to have to run diversion paths to their cabs or cages in case something happens. They still have a connection. Some clients are very security conscious. So you see these cables up here. That's actually Kevlar line cables. Those are armored cables, right? Okay. So we'll accommodate [00:05:11] Speaker 4: whatever the customer wants at the end of the day. There's some really cool gear as they're walking through here. I know we're not supposed to show on camera, but very cool. Yes. I will give a perspective [00:05:23] Frank: of the size of this facility. If you look down to that wall down there, about 65,000 square foot of raised floor space. And then we have a second data hall on the second floor, which we'll visit as well. [00:05:38] Speaker 4: Hey, so what's the maximum rack height that you support? There's 48U's in here. I think there's some 52s. They're about 10 feet. It's 10 feet from the floor to the ceiling. Okay. Because most of my see are 42s. So we have various sized cages on the floor. We don't constrict customers to a pod or [00:05:57] Frank: something like that. Like this customer here is a processing customer. So they require two-factor [00:06:04] Speaker 4: authentication for PCI purposes. So to get into this one, because it's a payments one, you have to go through the front security. Then you have to go through the mantrap with the iris scan pin and the badge. Then you have to do another pin, which is probably not the same one. And then badge as well. That's correct. Okay. So quite a bit. And this is a full cab. So this cab's capped to the top [00:06:31] Frank: and it's underneath the floor. So if you raise this floor tile, you can't get into that cage. [00:06:36] Ian McClarty: We've been very comfortable with bare metal hosting. Before public cloud was public cloud. Now it's becoming more mass market. As before, it was more unique and boutique type of workloads. The enterprise customers demand these services now. The reason why we went to a bare metal cloud philosophy was to really align ourselves to DevOps so that we can deliver strong API. We could deliver workloads that they're looking at bypassing the hybrid scaler with, meaning they actually want to run their own Kubernetes clusters vanilla or have a lot of control over that. HPC type workloads require that. We actually deliver that truly from a TCL perspective at the best rate out there. And we know we can be a market [00:07:14] Frank: leader in that space. This is actually one of our offerings, our bare metal offering. We have a couple of cages on the floor. It's probably about 4,000 services in here. With the super micro blades, right? You get 12 blades in a 3U space. Pretty dense, right? We keep this facility very cold, 72 degrees, plus or minus five degrees. Relative humidity, 45%, plus or minus 15%. Now, how we do that for the humidity piece, we have these electrostatic humidifiers throughout the facility. They're all zoned. So every once in a while, you'll see some moisture come out, right? A pump of smoke, which is actually just putting moisture in the air. So I don't have any kind of static electricity as I'm touching things. So again, we have full cabs, we have half cabs, quarter cabs, right? They get their own power space bandwidth. We do the traditional cold aisle hot aisle when it comes from a security. We don't do any hard containment, as you can see. If you look at the cold aisle, you'll see preparated tiles. So as that cold air is being blown onto the floor, comes up, comes into this gear, and then we have a hot aisle. All the hot air is exhausted out. And you look up, you'll see the egg crate tiles. So that hot air is being put up there and then sucked back into our crate. So how we cool the facility? We have these 80-ton Stoltz Cray units. So I mentioned 36-inch raised floor. So the chilled water pipe comes through these coils, right? And then this is 80-ton Stoltz Cray unit. These are all controlled by our building management system. So it'll maintain, right now, says the room is 72.1 degrees, 40-percent relative humidity. These are all controlled by the building management system. It'll ramp up. If load increases on the floor, it'll ramp down, like it's pretty ramped down right now. If you look down, you'll see multiple Cray units. These 80-ton Cray units, right? So that's how we're enabled to do up to 500 watts per square foot. From down here, we have actually two separate water mains coming into the facility from a redundancy perspective. We have one coming off the street over here, one coming off our main street here. Okay, something happens. [00:09:20] Speaker 4: That's sort of like something happens, like somebody's drilling in a street and hits a pipe and you don't want to lose water to the facility. That's why. In addition to the two water mains, [00:09:28] Frank: we also have two 75,000 gallon water tanks on the premise. Okay. And we have contracts in place that'll come in and actually provide water to us, right? Okay. In the event that something happens. [00:09:40] Speaker 4: So Frank, it's pretty hot here. So I'm guessing this is the chiller and this is how we cool the data center, right? Absolutely. Supposed to be about 100 degrees today. We're going to go into the chiller [00:09:49] Frank: plant. We're going to take a look at the chiller and see how we cool the facility. We have two 700-ton chillers here. These were the first ones that came. About three leaves later, we have these two 720-ton chillers. These are fully magnetic, so we get a little bit better ride through. So then this water comes in, it gets chilled down. There's refrigerant, 44 degrees. So everything coming out of here is 44 degrees in the chilled water loop. And that's how with going through the crays, the 80-ton solstice crays, right? Creates that static pressure and keeps the facility chilled at the 72 degrees Fahrenheit. And then you can see the input temp of the water coming in is 55 degrees, even though it's 100 degrees outside, right? And I'll show you how that's accomplished out back with the towers and the water droplets coming down through that media fill. Each of the chiller has two towers. So when that water comes back through, coming down through this media fill, which then cools the droplets down, right? That's how it comes back into the chillers at 55 degrees. And then, you know, it gets chilled down to 44 coming out. Then we also have a thermolytic tower for each chiller, 8,500 gallons. Actually, we'll keep the water chilled for about 20-25 minutes. So something happens, we have some nice ride through, we still have chilled water that we can disperse through the facility. We get about seven inches rain a year. We do have a monsoon season. Soil doesn't hold any water, so it just runs off. So if you notice, we're already on an incline here, away from the facility, right? You have these grates here, you have these water grates. So we're able to drain that water out and away from the facility at all times. So you don't park your car behind this wall in the rainy season? That's correct. So you can see all the thermistors around here for lightning protection. I've actually been out here in an electrical storm. Safest place to be. This is steel reinforced walls. We have several dishes, satellites, towers on the roof that are our various customers. Line of sight, wireless, all that kind of stuff. Some are line of sight, some are microwave, some are satellite, whatever it is. [00:11:55] Ian McClarty: We are very fortunate that when we built PhoenixNAP 10 years ago, that we had some foresight into power consumption rates. And we're actually going to run out of physical space before we run out of power and cooling. Today we can cool down 44 kW racks with no special containment needed. So we are well suited up and well organized for supporting a higher GPU cluster type workload, which today we see plenty of those [00:12:16] Frank: workloads. One thing that's a little unique about this facility, this is all medium voltage. So when you look at this UPS, this happens to be 12,470 volts, right? So this is all medium voltage. We're actually two in through the UPSs, which means we run separate feeds from the utility. They run divergent pass coming into feeders and they feed completely separate infrastructure. Okay. So two in. What you see over here, we have battery banks for A side, B side. So at full load, they would hold for approximately 60 seconds. Should the event happen within three seconds, we have six two megawatt diesel common generators. They actually will get a signal. They'll start typically within that three to 15 second range. They're fired up. They're in parallel and then we can run indefinitely. [00:13:13] Speaker 4: So right now we're upstairs at a power distribution block of the second floor facility data room. [00:13:21] Frank: Correct. And we mentioned downstairs, substation takes it from 1247 to 480. When it comes into the facility, one of those substations feeds this multi distribution point, right? This is all 480 volt. And then each one of these breakers will come out here and feed a PDU. That's how we get power to the [00:13:42] Speaker 4: various cabs and cages. Hey, Frank, are you seeing that customers are moving from 120 volt to higher [00:13:50] Frank: voltages more often these days? We don't get a lot of requests for 120s unless it's in our quarter cabs or something like that. The norm pretty much is 208 volt. As I mentioned earlier, this facility is designed at 200 watts a square foot. We can go up to 500. So we have some cabs over here. Our normal cab would be about 5kw. We have some 25kw cabs over here. I have some 43kw cabs downstairs, right? So we can handle, I never worry about cooling or anything like that, right? [00:14:21] Patrick: Or power for that matter. Hey, Frank, I just want to say thank you for showing us as well as our viewers on YouTube and readers on the STH main site all about Phoenix Knapp and all about the facility and explaining all of the bits that go into it. Definitely takes a lot of work. My pleasure. [00:14:36] Frank: And I appreciate you guys time to come out and film us. So, you know, we take pride in our facility. We love our facility. So absolutely. It's been great. It's been a pleasure. Again, I just wanted to say [00:14:45] Patrick: thank you to the Phoenix Knapp folks for having us down to Phoenix and showing us around their facility and making this whole thing really happen. And of course, if this is something that you're interested in, just, you know, they have things like co-location, they have bare metal cloud, they have dedicated servers, they have all kinds of different offerings. So go check them out. We'll have a link for that in the description. And also, if Phoenix isn't your thing, they do have facilities outside of the Phoenix area as well, which we mentioned in the video. And hey, if you liked this video, well, why don't you let us know in the comments below? Give this video a like, click subscribe, turn on notifications so you can see whenever we come out with great new videos. As always, thanks for watching and have an awesome day.