My Datacenter Evicted Me

[00:00:00] Speaker 1: We're here between a Walmart super center and a Canadian tire on the edge of Vancouver with a trunk full of servers, because sometimes it's just better to do it yourself. For almost a decade now, we've built and run our floatplane video platform on rented cloud infrastructure. And honestly, it's been great. We've been able to build up a robust and scalable service for both ourselves and other creators without the upfront investment of buying a bunch of expensive hardware or the ongoing cost and responsibility of maintaining that hardware. Sure, our monthly bill crept up over the years as we added more resources for a growing suite of internal tools used by labs and other various departments, but it's been manageable. Or it was until our host decided that our five-digit monthly bill was gonna go up by about 40%, whether we liked it or not. And spoiler alert, we did not. So what are we gonna do about it? Well, the answer's in there behind these, which I'm told are segues to our sponsor. [00:01:06] Speaker 2: Snapmaker, their U1 is a multicolor 3D printer and true tool changer. It keeps four colors or materials ready in separate tool heads for faster, less wasteful multicolor printing. Grab yours for Black Friday at the link down below. [00:01:30] Speaker 3: I'm here to grant you three wishes. For your first, I give you free cash. If you buy this $130 float plane credit, you'll get $60 of it back to spend on LTT store. You can catch up on amazing exclusives on float plane while also getting some amazing products. That's double amazing. What would you like for your second wish? [00:01:51] AJ: Affordable RAM! [00:01:53] Speaker 3: Well, I'm not omnipotent. [00:01:56] Speaker 1: This is CoLogic's Van 3, the largest network neutral data center in Vancouver. I can show you here, and here, and here again, but seated. So why a co-location center? You can think of it like a super technically advanced mini storage facility. We're renting space, but they're also providing redundant power, blazing fast networking, and multiple layers of security. All things that would be crazy expensive for us to do ourselves. It's $35,000 just for a generator. This is it, our blank slate. It's bring your own rails, power delivery, routing, switching, servers, backups, etc. All you get are hookups into the facility's power and networking infrastructure, and this sweet, sweet five square feet of floor space. AJ, Dan, what are we installing today? [00:02:52] AJ: We're installing networking first, because without networking, there's not much happening. So, sorry, Mr. Luke, but we will evict you. So yes, this one is the first switch. You will have seen this switch before in a previous video. So this is the secondary switch, all 100GB, 32 ports. We're looking at two for obviously redundancy. If you notice right now, they're actually pointing backwards, and there's two reasons for that. One, it's way easier to route your cable to your servers. And these will actually act as top of the rack switches. And the other reason is that we got models that actually pull air from the back and push air in the front. So if we want to conserve the right airflow with the co-location facility, we need to make sure they're in the right [00:03:45] Dan: airflow orientation. Hey, AJ, weren't those two switches like, you know, in our building? Yeah, I might have [00:03:51] AJ: pulled them off to pull them off of production. Right now we we have no redundancy at the office. The only issue we have with these is that they have pretty significant limitation with routing. What we're expecting to do is switch those with beefier, more appropriate switches down the road. So stay tuned for that. This is the out of bound or yeah, sorry, the out of bound switches. So the reason we do that is that if these switch or the main internet breaks, you want to be able to access your hardware remotely and fix things. So that allows you to do that by, you know, having a VPN to another internet connection that you don't necessarily control or own or through LTE, which we will try to do. But this facility is a big concrete box. And this is a big metal box. So will the signal go through? We'll see. And not a VPN in the way that like PIA is a VPN? No. So a site to site, we use something called cloud player zero trust for all our meshing networking. So that will technically land a mesh node so we can access it remotely. Why do we call it out of band? We call it out of band because we want to have a network connection that is out of the primary band that we pass all the traffic that is for clients. That way [00:05:13] Dan: you have an isolated out of band because you've got bandwidth. Yeah. And then we're like over here. Yeah. [00:05:19] AJ: All right. So next thing we'll be installing is a multi-blade server. This one is actually something that, again, you might have seen in another video, the redundant 1X server, I believe. So it's not redundant anymore. It's in my hand. It's getting reborn here for Flowplane. It's going to host videos. [00:05:40] Speaker 1: For now, it's really heavy. So we're going to remove some of the blades just to make this easier. And we'll [00:05:45] AJ: put the blades back in when it's in the rack. What are you doing with all those sexy, sexy blades? What we're going to be doing is installing a software called XCPNG, which allows us to pool [00:05:59] Dan: all the resources together. All four of those blades are going to act as one computer then? [00:06:04] AJ: Yeah. And also the two other servers. So it's all kind of quad redundant rather than dual redundant? In a sense. But since we're going to be running so much services, we only keep one as a redundancy. I see. So we can run three total capacity and then one for redundancy. [00:06:20] Dan: Kind of like RAID, but with compute in a way. In a way. Yes. [00:06:25] AJ: All right. Here's our fifth node. A 92 core CPU from AMD. Oh, actually, it has two. So a hundred... Actually too many. I don't want to count. But yeah, it has a lot of cores and it has 1.5 terabyte of RAM. This thing is a beast. And you know what's a beast? This second one. It's exactly the same. Our sixth node here is worth the car. But it's also going to be running our backups and stuff. So for the out of bound or OB network, we're actually going to install a UniFi UDM Pro that we had lying around. This thing is going to work pretty well for us. And since it supports WireGuard, we will be able to connect [00:07:22] Dan: it to our network remotely. All right. You've taken our switches, but how are we going to like [00:07:28] AJ: connect to the office? For now, we're going to use a VPN link, but we're planning something where we will actually have a, we call it a layer two. So a direct fiber link between this data center and the office. [00:07:43] Dan: Our own pipe? Yeah. Like our own truck that we can put things on? [00:07:47] AJ: Effectively. It's a bit more complex since we're actually renting a wave. So we're renting a part of a fiber, but it's up to 10 gig. And we might even, you know, go up to a hundred if needed. [00:08:01] Dan: One of the reasons we're using two different colors is because we've got two redundant power supplies in just about everything. And if one side fails, the other one keeps going. It also lets us move cables and reroute things without having to shut anything off. So we can re-cable manage maybe the blue side and then finish that off and then go do the red side [00:08:21] AJ: as well. This building has multiple power feed, which allows us to actually power our servers from two feeds. And if one goes down for any reason, the servers will stay up because they'll keep getting their power from at least the blue side or the red side. Pick your team in the comments. This is our out of bound network. We'll just slap this here, plug this here. And what this should allow us to do is hopefully connect to the internet wirelessly. In the case that the internet or the transit that we're using goes down, we still want to be able to monitor or do fix remotely. So that allows us to connect through 5G with our provider and then enter and do any changes that's necessary to get things back up. So this facility actually monitors the power usage of every rack and also the temperature going in and out. That way they can actually live tune their fan circulation to account for servers consuming more power and tune them down when there's less consumption. That way they can save power when there's no big need of cooling. [00:09:34] Speaker 1: As much as it is a facility that just draws a lot of power, if we can save any of it, that's great. Yeah. [00:09:40] Dan: That's cool. We're in a data center. We've got internet coming in here. Can we be an internet service provider now? For linus detectives. Yes. So we're going to provide internet to the office. [00:09:51] AJ: Yes, correct. That's the plan at least. [00:09:53] Dan: And we have three internet connections right now. So that's probably going to save us a lot of money, right? [00:09:58] AJ: Yeah. We will keep some for redundancy. Lisp will also allow us to have a direct local connection between the colo and the office. So it'll be a LAN connection? Yeah. So technically, from the office, we'll be able to connect to the network directly as if it was at the office. No more VPN. No more VPN. With Dell switches, something that you need to be careful is the DAC that you plug in. And something like FlexOptics DACs, what's cool is that we can use their FlexOptics programmer to program them to effectively be like, "Hey, bro, I am a Dell DAC. Please let me in." And yeah, right now it is doing it. And once that's done, we can just lop it in. If I can align it. Here we go. [00:10:48] Dan: Looks like we cheaped out a bit and got a factory recertified drive. This is a Seagate 26 terabyte though. I don't know. Hopefully it works. We'll find out. [00:10:57] AJ: We're actually also going to connect with VANIX. And VANIX is the local exchange here for internet traffic. So what we did with them for cost-optimized connectivity is we went with BiDi connection, meaning that every strand of fiber is actually carrying both RX and TX or receive and send. So right now, this should be able to connect to both switches at the same time. Once I hook everything up, we will be able to VGPP with every partners that there is on the network. And that includes Cloudflare, which we do a lot of traffic with. Everything is Cloudflare protected. That means that we don't really go out to the internet for that traffic. We will be able to just directly peer with Cloudflare. So the other end of that fiber is effectively going to the rack of VANIX. [00:12:00] Speaker 1: A lot of this is we're just trying to do initial setup. We can continue to tidy and optimize and improve over time. But we want to get things going. We have limited time with AJ here. So, you know, certain cable color combinations, tidiness, whatever will improve in the future. [00:12:21] AJ: This here is a MPO cable. The really nice thing is that it contains multiple strands of fiber. This can transport eight different signals at the same time on eight different fibers. And what's cool is that this allows us to break 100 gig port into four tens. That way we keep more [00:12:42] Dan: of our capacity going forward. If we're moving all this stuff off of OVH or at least working with it, I know that we host Kubernetes there. Are we going to put that here too? [00:12:52] AJ: We have about a thousand something containers running. I love containers. We do run a lot of stuff like databases for labs. All this is going to be get migrated there and potentially even get a boosted performance because we'll have technically more horsepower and better drives. I can have more workers. That'll be nice. And more bandwidth actually. All our racks were at, or servers were at 25 gigs and we're upping them to a hundred. [00:13:18] Speaker 1: Very nice. Lots of room for improvement. There's going to be quite a bit more work done in here. Hopefully you guys liked this video and hopefully we can keep releasing more of these that you guys like. Thanks for watching. Get subscribed. And, uh, I guess follow this segue, uh, to our, to our, to our, to our sponsor. Follow the segue to our sponsor. Get out, dad. He's gotta, he's gotta keep the servers working. [00:13:48] Speaker 2: Bye. Snapmaker. If you're looking for a multicolor 3d printer, Snapmaker's U1 keeps four colors and materials in separate tool heads and switches between them in about five seconds for faster and more efficient printing. This means more material will make it into the things you make and less will end up in the trash bin. It's real tool changers swaps between those four different heads. So there's no shared nozzles and no time wasting color purges. If you're new to 3d printing, they also have Snapmaker space, a great hub for you to find things to print. So you don't have to stumble around designing everything from scratch. After running a successful Kickstarter campaign, every order was delivered. And since then they've sent out over a hundred thousand units worldwide. Snapmaker U1 is down from 999 to just 699 for black Friday. So why not save $300 and grab yours at the link in the video [00:14:37] Dan: description. If you liked this video, go check out this cool playlist of data center tours we do. We're going to have some more coming down the pipe, down the band. Okay. Bye.