PON in the Datacenter: Hyperscale for Management and Console

[00:00:00] Speaker 1: Our next person up is MJ Mike Joseph, who will be presenting PON in the data center, hyperscale for management and console. MJ is currently the tech lead and manager of the infrastructure network engineering team at Meta, and we're happy to have him speaking here today. [00:00:22] MJ Mike Joseph: MJ? Thank you. I've been a member of this community for quite some time, but this is my first time presenting, so it's an honor to be in front of you, and I see a lot of familiar faces here in the crowd. So first, I want to talk a little bit about what my team does at Meta. We're responsible for providing the management out-of-band and initial provisioning and disaster recovery infrastructure for Meta's POPs and data centers worldwide. We're unique in the Meta production and operations organization because we actually hit every single rack and every data center in POP, which makes us having the widest scope of any other team doing networking at Meta. We're also the first in team to Meta data centers, and we're generally the team that gets called first when we have to recover a data center due to an outage. One good rule of thumb for my team is that if it's managed by production, and it's one gig or less, it's probably us, because we provide management Ethernet and console to our devices. Now, most of my team's infrastructure is actually in the data center side, although my team organizationally is part of backbone organization. And as a result, today's talk is going to mostly focus on our data centers, which is where we're deploying the solution first. In our data centers, we currently have a number of data centers around the world, as most of you know, and our current technology for serving management Ethernet and console is probably pretty similar to what many of you are familiar with in a data center environment today. We have an end-of-row device, an Ethernet switch, and a serial terminal server, although in Meta's case, we mount them on what we call goalposts, which are 19-inch double-wide racks elevated above your head at the end of each row. But aside from their unique placement, the topology is probably pretty familiar to most of you. Well, this doesn't have a pointer. Well, as you can see on these pictures, on the lower left of both of these racks, you have an example of an Ethernet switch and serial terminal server. Coming out of them, you have these large copper bundles. These bundles then go down into ladder trays, which go out to the racks that are served by this goalpost. As you can imagine, in a full row, it's quite a bit of copper, as you can see on the top. Now, in addition to the copper load here, we also have to deal with some other limitations. First, this is fixed. We have one switch and one terminal server in those pictures, and some of our rows, we have two, but we can't put in an infinite number. Now, we could get a bigger rack. We can choose not to hang it from the ceiling, but ultimately, we're going to hit some limitation at some point. More to the point, though, each rack today, we have to decide how many Ethernet and how many serial drops to put at a given rack position in advance. To change that number, we would have to go reconfigure the row or even the data center because of how we need to keep our environments automated and consistent. In addition, as you can imagine, this is quite a few devices to manage, tens of thousands of them across our data center footprint. So this doesn't give us a lot of flexibility. As you can, as you know, Meta is building out huge new data centers to support our AI demand. And going with the AI racks, we also have network racks. All of this increases the amount of Ethernet and console we have to serve at a single rack location. This is a very difficult problem to solve with our current topology. So we've decided to deploy PON. Now, many of you know, PON or GPON is a residential access technology, typically, used for residents, SMBs, some cell phone towers. It's typically an outdoor access method. It's not generally used in data centers, except when you're doing testing to certify equipment for use in a residential network. And yet, we decided to do something wild and deploy it in data centers. And not only go wild and deploy it in data centers, we decided to deploy it in all of our data centers. For us, it solves a number of unique problems. First, it gives us great flexibility. As I mentioned, we have a lot of new racks coming into our data centers. With our current PON solution, we can deliver 40 copper drops to each rack in some mix of Ethernet and console. Now, we can vary that mix and we can vary the number of drops within that limit, using our existing rack equipment today without any change to the data center's topology. Moreover, we're working on additional rack products coming out in the future that allow us to increase this number even further. If we swap a rack out, which happens with some regularity in our data centers, all we have to do is unplug that rack from the fiber topology, plug in a new rack without any changes to the data center itself. Scale is another big consideration. Given our current footprint, we can deploy up to 1280 Ethernet and 800 console per row. It's about a 10x improvement as to what we currently do maximum today from our end-of-row solution of goalposts. Additionally, we can manage up to 12 rows on a single pair of OLT aggregation devices. That's a huge savings for us in terms of number of devices managed, especially because although we do have automation, every device under management could be subject to a firmware upgrade problem or a configuration drift issue or any other problem that requires operator intervention. We also are able to improve our workflows. Now, many of you may have seen some of the media where we showcase some of our rack automation. Our racks are rolled into our data centers off trucks. They get scanned in at the loading dock and then either placed into a rack position by a human or, more often these days, by a robot. Once the rack is placed, a technician simply comes up to the rack, scans the Pano and use on the rack, scans the rack itself, hits a button on their tablet, and the rack self-provisions. This automation workflow fits very well with how Meta conducts all of its other operations and is much easier to handle than, for example, having to reconfigure an end-of-row switch every time we need to change a topology. It's also not dissimilar, by the way, from what you see in residential deployments upon, which is one of the wins we get from adopting this type of technology. We also get to improve our efficiency. Now, the switches I showed at the beginning of the presentation, those are 48 port switches and so are the serial terminal servers. Now, if you happen to have 48 ports to your row, that's a good size, but if you need more than that, you end up having to deploy another set of switches and console servers. Those can be expensive, both in terms of cost and power. PAN devices are fairly optimized for low port count. In fact, most of the Pano and use we use are single SoC devices that integrate PAN BOSA and up to four Ethernet on a single ASIC. This allows the devices to be both power and cost optimized for this particular use case. We currently deploy our capacity in four port increments. This works quite well when you have a rack that might need one or two ports. And if we need 10, 15, it's easy for us to do by simply deploying additional ONU and console devices in that rack. By deploying capacity exactly where we need it and the size we need it, we're able to reduce our cost and improve our power efficiency. One of the things we did in this particular design is we also improved our space by utilizing unused space on top of the rack for what we call a zero OU deployment. OU being the unit of measure in an open rack compared to RU in a traditional 19 inch rack. I mentioned all that copper we handle. Well, one of the great things about our PAN deployment is we've actually eliminated all copper trays from new metadata centers. So none of our data centers go and have any copper trays anymore. We don't have to carry copper throughout our racks. All the copper is self-contained to the rack between the PAN devices at the top and the equipment inside the rack. I mentioned about the PAN SOCs being well optimized for our use case. In fact, beyond that, PAN uses a set of protocols called PLOM and OMCI for managing the ONUs. These are ITU standard protocols and they have interoperability between ONU makers and OLT vendors. This allows for easy extensibility when we need to add new ONUs even with the same vendor who might be using a different SOC. This is beneficial because it gives us rapid onboarding of new rack products. And in fact, it also proves our supply chain diversity. If one of our manufacturers needed to switch to a different chip or a different design for an ONU, it wouldn't be a problem for us to support because it's quite easy for us to adapt that into our topology, given that all the complexity is handled between the OLT and ONU over the OMCI protocol, which is well standardized. The ONUs themselves, they're not managed. In fact, we don't even have console on them. You can't log into them. You can't SSH to them. You can't get a shell on them. You can't even manage them on a config. In fact, if they break, we take them out and throw them away because they don't have configs on them either. Because the ONUs are stateless and transient, they're entirely managed by the OLT. And I'll talk a little bit more about that later. One of the surprising benefits of moving to PON is actually improving our staffing. Now, I talked about efficiency. We're able to drive our network from a much larger network footprint given the AI demand with a currently small team. But it also allows us to offer cross-training for meta staff who want to join INE and gives them an opportunity to build up industry skills. And it also allows us to go higher from the industry. So for folks who are maybe coming off of a residential access ISP, it's a good skill set. We can hire into meta and they get an opportunity to work on web scale solutions using their existing skill set. So it gives us a lot of efficiency in terms of how much we can support from a relatively small team. Although we are hiring. Now let's talk a little bit about PON fundamentals. Now, as I mentioned, PON is traditionally a residential access technology. It is shared medium, similar to DOCSIS with cable modems, with an OLT or optical line terminal at the head end. And ONUs at what is traditionally the customer premise or in our case, the racks. The OLT controls all aspects of a PON network. It controls timing and gives bandwidth grants for the ONUs to transmit, much like DOCSIS. It also admits new ONUs to the network and authenticates them. Finally, as I mentioned earlier, using PLoAM and OMCI, it manages the full configuration and even the firmware images on the ONUs. This is actually all transparent to us. We don't have to trigger these events through any of our systems. This is integral to how PON functions. When an ONU comes up, the OLT recognizes it, admits it, upgrades or even downgrades its firmware, and then pushes configuration to it from the configuration that's pre-staged on the OLT for that given ONU. Here's a good diagram showing you how a given PON topology might operate. On the left, you have an OLT. And on the right, you have a set of ONUs. And then you have the ports beyond the ONU. We are called UniPorts or User Network Interface Ports facing the equipment. In the middle, we have what's called the ODN or the Optical Distribution Network. This is passive. This is a fiber plant. And PON is based on optical splitters. So as you can see in this picture, we show two splitters. An optical splitter in the PON world is pretty much like it sounds. You have light coming in on one end. And then in a case of a unity splitter, which is mostly what we use, you have equal parts going out to, say, 2, 4, 8, 16 ports on the other end. And so, yes, you have a reduction in power, optical power, between the left and the right. And in fact, actually, you also have a reduction in power from the right to the left. But that's taken into consideration when we design PON networks in terms of the power budget. Both the PON OLTs and ONU lasers have a fairly high output level. And the receivers are quite high sensitivity. So that gives us quite a large power budget between what we can afford to burn, both in terms of fiber distance, splices, and especially the split ratios that we handle. Now, there's a number of PON technologies grouped largely into two different families. Most of what I'm talking about today is in the G-PON family, which is standardized by the ITU. G-PON, which is the original technology standardized by the ITU, is basically a 1-gig technology with 1.25 up and 2.5 down. And then there's various iterations on top of that, like XG-PON, NG-PON, or what we use on the right, XGS-PON, which is a 10-gig symmetric service. In addition to the ITU PON family, there's also the IEEE PON family, or E-PON. There are somewhat competing standards, although most, both ITU and IEEE generally have standards for PON speeds at equivalent points. And in fact, a lot of equipment can be operated both in G-PON or E-PON mode, depending on the device. In our case, we use XGS-PON, which operates 1270 nanometer wavelength up and 1577 downstream. And the reason that is important is because PON is a bidirectional technology. Every device, both OLT and ONU is a single fiber. We transmit with bidire optics, so the single fiber is used to receive and transmit data in both directions. Because PON is well-contained to particular wavelengths, it means that in a PON network, you could actually have multiple technologies on the same ODN. Often you'll see that by operators who are migrating from, say, G-PON to XGS-PON, where they can operate simultaneously because they use different wavelengths, but other operators also put sometimes other services on their ODN. In Meta's case, we simply use XGS-PON and we don't share with other services. Now, the ITU defines a number of redundancy standards. In this picture, we depict type B, or more specifically what the ITU defines as dual-homed type B. Type B redundancy works by allowing for a second OLT port, or what we call warm standby port, to become active if the first port stops working. And dual-homed type B, that second port is actually located on a different device than the first one to protect against both interface and device failure. Meta has extended the type B standard with what we call type F redundancy. In type B, the way an OLT detects failure is if it stops seeing light coming up from the network, it can assume that the other OLT port probably failed because, remember, in order for an ONU to transmit, it has to be given a grant by the OLT. If the OLT stops working, it stops giving grants, so there's no northbound light. Therefore, the other OLT can assume that its neighbor probably died. In type F, however, we add an additional layer of protection. Even if a given OLT doesn't completely fail, if it loses visibility to a certain population of ONUs, we direct that OLT to abdicate or to stop transmitting so that the other OLT can take over because we believe that it might have a better view of the topology. It's not perfect, but it gives us a little bit better protection against, say, losing a whole rack. Now, this brings us to Meta's design. Now, we worked closely with our initial PON equipment vendor to bring all of this vision to reality. We were able to collaborate effectively with them to drive our requirements and also help them develop the new product line that they're offering for PON in the data center. Everything you see in this slide deck are, for the most part, active equipment that users can buy today from that vendor, and we invite other participants and other vendors to the ecosystem to help drive what we believe is the way forward for device management at scale. This is a meta talk. It's not a vendor talk, so I'm not selling anything, but I do have some examples to show you as we go further along in our talk. Now, as I mentioned in our data center deployments, we build PON data center at a time. And in each of our data centers, we have our head end space. This is the space that comes up first. I explained earlier that the INE team is the first in service to a new data center. So when a data center gets built, the first rooms that get power are the spaces where our core equipment goes. Now, some of this equipment is used to reach our out-of-band backbone, but some of the equipment is also used to provide the PON services to the racks. Here's an example of what we call our OLT aggregator, which is basically a layer three switch at the top. And then, in our case, the OLT technology we use is based on what are called micro OLTs. And these look like SFP+ modules. And in fact, they do speak SFP+, but they're actually far more than just optics. They implement the entire OLT stack directly on the module, and it actually embeds a three-port switch. And it translates between the Ethernet MAC framing from the router switch into which it's plugged in to a PON framing on the optic side. Now, in our case, these OLT aggregators aren't just simply routers or switches. They also actually run the PON control stack that drives all the OLTs embedded in that switch, as well as manage the ONUs that are connected to that aggregator. Below you can see a picture from one of our labs of a sparsely populated stack of OLT aggregators. This one is just getting turned up, so it only has a couple of the micro OLTs popped in it. But you can see those are already starting to get plugged in in this picture, and that's only four of them. In a larger data center, we would have quite a few more than that serving the data center, but still a lot less than we would have of those end of row switches. As I mentioned, we can serve 12 rows. For us, it's about 500 racks from a single one of these OneRU devices. Now, at the top of the rack, we have what we call our canopy chassis. This chassis is able to use unused space in our data centers. You see, it's not that much unused space between the top of the rack and the first cable tray, but it's enough that we can fit this chassis in. And then in this chassis, we can have various modules like ONUs and console cards, as well as optical splitters. And as you can see, both the top picture and the bottom picture, you can see those fibers in the back, and that's a pair of optical fibers coming down, and those are the two uplink fibers for that rack. If this rack were to be rolled out, for example, those fibers would simply be disconnected from the splitter, coiled up, and a new rack put in its place, and those fibers connected to the new rack. All the complexities contained within that rack. And that uniform two or four fiber drop to the rack position is all that we need to serve a given rack. Now the chassis itself is entirely passive. It does have a PCB on the back, but that PCB simply has copper pores for power distribution. Meta uses open rack, as you're well aware, given that we're a key developer in the open compute environment. And we use, most of our racks are open rack V3, or one variant of open rack V3, like ORV3, ORV3HPR, ORV3N. Those racks all use a native plus 48 volt bus DC, and this canopy chassis extends that 48 volt bus across its back plane to deliver power to the cards. There's no communication on the back plane, however. It's purely power distribution. This allows us to put different cards in the chassis without sharing fate. Of course, if the rack power goes down, all the cards go down too. But if the rack is unpowered, we're not that concerned about managing it either. One thing I'll note on this picture, the fibers here, by the way, not very clean. This is a lab. This is one of our early deployments in a miniature scale lab that we built up in one of our lab facilities. We would never have coiled fiber like this in our production data centers. It would look a lot more cleaned up, something like this. So, we have ONUs. Here's one. These ONUs are quite a bit bigger than you might see in a traditional residential deployment. They are made of metal for EMI, because our data centers have quite a bit of interference. And at the back, they have power. What they don't have is any backplane connections, because as I mentioned, all of our connections are on the front. Next to the ONU here, you can see there's a console card. The console card is effectively a daughter card for the ONU. Here's one here. Now, as you notice, the console card is quite a bit shorter than the ONU. It doesn't interface to the backplane. In our case, it has a USB port that interfaces directly to the front of the ONU and serves for both power and data. So, the ONU itself manages its console card. We also had these handy little USB connectors made to interconnect them. Now, this is not the only form factor, but this is the one we're deploying right now. So, I think it's important for people to understand how we're building out our current data centers. Now, as I mentioned, these ONUs, they're not managed, at least not directly. They're managed by the OLT, using Plume and OMCI. They do run SSH, that's for the console ports, but there's no way to get a shell on the ONU itself. And the nice thing is, if we had to swap out this ONU for a new one, stateless. The field pops it out, pops in a new one, scans it, and comes right up with the same config. And so, these devices are also relatively inexpensive. Meta considers them commodity. If one fails, we don't RMA it, we just throw it out. We also have the ODN, that's the Optical Distribution Network. I mentioned that's also passive. And that has fiber, as well as splitters. Now, at the end of that canopy, you probably saw there's a bunch of fibers going into another module. That's a splitter. That device has no power. It's just basically a physical enclosure that fits in the canopy, that has either one or three splitters in our design. Here's an example of one here. I wore cargo slacks for this presentation. And so, we would have the top of the fiber coming in from the rack, simply going into the top of the splitter. And then, this splitter is capable of serving up to eight ONUs in the rack. That's how we get to that 32-port Ethernet in some of our rack designs. Now, in the lower picture here, we have an example of our goalpost splitter. This is a cassette that goes into a fiber enclosure in the top of our goalposts that handle fiber distribution for the row. This actually embeds three large splitters, three 1 to 16-way splitters that provide all the fiber distribution for one side of the row. I say one side because we actually have two goalposts on the end of each row, providing the A and B paths for our fiber. Finally, in the upper right, you can see an example of an industry standard LGX cassette. And that's a standard splitter that we use in our head ends for one of our PON technologies. We call our HA technology, which I'll describe in a second. And that's to provide head end splitting right off the front of the OLT, where we have low density PON, and we need it to serve multiple rows from the same port. Now, as I mentioned, we have a couple different PON networks we run in our data center. The first and most common is our normal availability network. Most of our devices are on this network, and this network serves every single RAC. And this network uses a pair of OLTs and diverse head end locations in the data center. And they serve every RAC position on the A and B side, using the type F redundancy that I talked about earlier. That means we have fibers coming out from both OLTs, shown here, going to the goal post splitters. From the goal post splitters, we have fibers. And the goal post splitters are on both ends of the row. So they come across the row and then drop into each RAC position. They go to a RAC splitter, this one here on the A and B side. And then they drop down to our individual ONUs inside that chassis. And so we get protection all the way up until that RAC splitter on the optical path. And we also get protection from the OLT and the PON port on that OLT in case it fails through the type F protection. One thing this doesn't protect against, however, is it doesn't protect against the failure of an individual ONU serving a piece of equipment or that piece of equipment's Ethernet port itself. But for most of our devices, that's perfectly fine because we have two paths to manage the equipment. We have the out-of-band and the in-band. Some of our equipment, however, does not have two paths. For example, a cell-based switch might not have in-band IP at all. One of our design rules is that we have to provide two management paths for every piece of network equipment in a metadata center. In order to handle that, we use an HA network. It still uses a pair of OLTs, diversely placed at our head ends. But we don't use type F redundancy in this or even type B. Instead, these are two entirely separate PONs, or we call them HAA and HAB. And they're independent all the way down to the ONU and even down to the port on the client device. So in this case, there's no failover between the OLTs. They're active-active. They do get split at the goalposts just like the NA network does. But they get dropped simplex to each rack that needs them. And then within the rack, there's a dedicated pair of ONUs for the HA equipment in that rack. I will note that most of our racks don't actually need HA. Only a few of our high-density network racks that use, for example, cell-based switches, which don't have in-band management require an HA solution. Now, one of the things we've noticed in the industry is that most OLT vendors have been moving over the last decade or two towards off-box solutions for control, for their control plane. So a lot of times you'll see PON controllers or OLT controllers in server racks, either near or adjacent to an OLT spec or sometimes in central data centers operated by the ISP. For residential ISP, that actually can make a lot of sense. After all, you're not building your network for super high redundancy, because if your network doesn't work, your customers aren't online anyway. You're building it for ease of operations and low cost for your carrier operations. But in our case, we're not using it just for user access. We're using it for provisioning and disaster recovery, which means we need it to be up irrespective of any part of our server footprint working. To this end, we basically issue off-box controllers in our design. Instead, we worked with our initial PON vendor to implement their OLT control stack directly into that OLT aggregation router. And in fact, it's not only integrated, but it actually is redundant between a pair so that we have failover. And for our deployment, this gives us much greater reliability and survivability in the face of failure and allows us to not have to deploy dedicated servers next to every OLT that we put out there. Now, one of the nice things about our design is that our OLTs are entirely managed through our automation system, which uses a mix of NetConf and GNMI to manage the entire OLT fleet. I mentioned earlier how easy it is to provision a rack. We simply have a technician scan the ONUs on it. And once those ONUs are scanned, the serial number and identity of that ONU is populated in our database. That gets copied over to the system that provisions the rack. And that OLT then gets configured updated with the ID of that ONU so the rack can come up. And this happens continuously through our continuous automation and deployment infrastructure. My colleague, Kang, will be speaking about this more in-depth later this afternoon, so I do suggest you attend his talk as well. Now, where does this leave us? So, currently, PON is the plan of record for all metadata centers going forward. We do have a few LESC data centers still being built this year, but basically anything we're breaking ground on now or in the future is going to be PON oriented. We actually have a number of PON data centers currently under construction and a handful of them already have PON headends running. So, that picture I showed you earlier with the LLT stack, they're actually operating in a few data centers already. Now, they don't have their racks yet. Our data centers come up in stages and the earliest stage is the headend space. So, they're live now. We expect the first production rack to go live probably next month or so. And then, in the next several quarters, we expect a lot more production racks to start streaming in. We also anticipate breaking ground on a number of new data centers, which will all be PON first. So, that takes us to finally, where do we go from here? Well, as I mentioned, we're working on next generation rack products. These are pretty cool, but they're four ports and some of our racks need even more than 40. We have racks that might need 80 or 90 copper drops to serve the rack. And, in those cases, we're working on higher density PON solutions with our PON O and U vendors, as well as in-rack chassis that might need to handle more O and U than just a canopy can. Additionally, we have some emerging rack types that don't even have space on top because of their unique power and cooling needs. So, we have to come up with unique form factors to solve those. We're also working on an RU form factor to be able to use in spaces where we don't use our standard server racks. For example, like in our network management spaces where we might have our optical equipment or core networking equipment itself. One of the other things that my team does, in addition to operating the management network, is we also operate what we call our data center facilities networks. These are the networks that operate things like, say, air conditioning systems, generators, power transfer switches in the data center, all of our sensors in the data center. These are often one or two port devices spread out throughout a very large data center campus. This actually speaks, this actually is a design that practically calls out for PON because it's very similar to what you would see in a residential deployment. Disparate, diverse connections, far apart, quite farther than, say, the 100 meters you might get on a traditional copper ethernet. Low bandwidth, bursty connectivity, managed by some central controller. And so, that's one of the areas we're potentially looking into for next steps that we may expand the PON solution to. Additionally, we have opportunities to push PON into other spaces in the data center, like those core networking spaces that I talked about. And finally, into our POPs. There's opportunities to simplify our POP design by deploying PON into a variety of racks in our POP spaces to make them more consistent with our data centers. But that's probably a few years down the road. So, that's what I have. Happy to answer any questions that anyone may have. All right. Well, this equipment, I'm told, will be available from the vendor at their booths and beer and gear. So, if anyone wants to take a look at it. [00:32:14] Speaker 3: Someone's coming. [00:32:15] MJ Mike Joseph: Oh, sorry. [00:32:16] Speaker 3: One question for you. [00:32:17] Speaker 4: So, how big of a deployment do you need to have before you think that this type of solution makes sense? Are you talking, you know, a few hundred servers, thousands of racks, hundreds of racks, thousands? [00:32:33] MJ Mike Joseph: Yeah. So, I wouldn't say servers so much. And one thing I'll note, and I probably should have clarified that when I started. We don't actually, my team doesn't actually provide connectivity to servers. Although we do provide connectivity to server racks. But we service things like the rack management controller and the top of rack switch. The servers themselves don't really need out-of-band because of the net, the primary in-band network that manages the servers is not working. There's very little point in managing the server itself. So, number, how big would you need to be to have this? Well, that's a very good question. And I think there's two different answers to that. For Meta's case, because we already have the infrastructure and knowledge in the team to operate upon. Our threshold for a given site size to deploy it is very low. I mentioned we are considering even putting it in POPs or smaller spaces where we might have, you know, only, you know, dozens or hundreds of racks compared to our data centers with thousands. Because in our case, there's not much overhead in one additional deployment when we already have the software infrastructure and knowledge to operate that. The overhead of the PON head end itself is quite small. It's just like, you know, a couple 1RU devices. But for an operator that might globally only have a handful of, you know, a couple hundred racks, it might not be worth it. It's probably hard to say, and I don't have a number in my head. I will say that for us, it's not just scale. It's also the complexity of the rack. The fact that we can handle different racks coming into any given rack position without having to change the topology is a big win for us. So if you have a topology like that, your number might be lower where it becomes worth it even if you only have, say, you know, hundreds or a couple thousand racks rather than, say, you know, many, many thousands of deployments like we do. Great. Thanks. [00:34:13] Speaker ?: Sure. [00:34:14] Speaker 5: We have one from online. Matt Pitek, underemployed. Great talk, MJ. Do you have data on reliability of the ONU devices you're putting in place? It looks like the same ONU provides both the management ethernet and via USB daughter card, the serial console. Does that mean if the ONU fails, you lose both management ethernet as well as the console? [00:34:39] MJ Mike Joseph: Yes and yes. Although we don't have to, but that is how we choose to deploy it today. We don't have empirical data ourselves because, as I mentioned, we've operated this in our lab for a couple, two or three quarters. Our production deployment is just starting, so we don't have our own data. I can tell you the manufacturer stated MTBF of the ONU I think is around 250 years and the manufacturer stated MTBF of the console card is about 2,000 years. Now, extrapolating that out to one of our data centers, we would expect upon stack to fail in a given data center about once per year. So that means that we would expect our technicians to visit a data center rack and have to service the equipment in that rack at about that interval. Because of that, when we designed the canopy, actually, one thing you'll note, I'll go to that picture really quick. One thing you'll note here is the cable management crosses the cards. We had to preserve our vertical space to get the maximum amount of footprint, so we did not put horizontal cable management to keep the fibers out of the way of the cards. We did this intentionally because of the high expected reliability of the components in the ONU and console cards, so that we know that when we do have a failure, we actually will have to pull out adjacent cabling as well. But we're willing to take that risk on, given what we expect to be the low failure rate of the cards. Also, in this type of equipment, the most likely failure opportunity is going to be at initialization, right, a card that had a manufacturing defect and won't come up or was damaged in transit. Once operational, we're expecting that failure rate to be quite a bit low. Thank you. [00:36:16] Speaker 3: Hi, Rich Compton, Comcast. ISPs that use Pawn, they pretty much always have encryption enabled. I'm curious if you guys have encryption on that Pawn network or do you not have to worry about it because it's, you know, all within your own data center? [00:36:34] MJ Mike Joseph: I'm sure there's meta security folks here who will tell me that we should encrypt absolutely every link at all times. I will also point out that there is no encryption for RS-232 coming off the consoles anyway. However, we actually do use encryption on Pawn because for one thing, we opportunistically want to encrypt where we can. And for another, there are deployments which we are looking at, which isn't really relevant in our current data centers, but is relevant for some of those future deployments like in POPs, for example, where we might have the Pawn fibers running through untrusted conduit and we might want to have, depend on the encryption there, even if the rack itself is considered in a trusted environment. So, yes, one thing we're actually working on with our initial vendor is implementing device attestation of the ONU so that most Pawn deployments today commercially come up with either, typically are serial based, serial number based, right? It's not a very strong method. The ONU has a serial number and that's how it's authenticated by the OLT. Some operators implement an additional layer of protection where there's a password or other authentication token at the customer premises to activate the ONU. In our case, what we're working on is strong device attestation so we can use a certificate based, certificate to place during manufacturing, so we can immediately identify the ONU and have a sort of a trust point on the ONU during provisioning. So that's, I don't actually have the date for that feature, but I think it's coming up in the next, probably the next two or three quarters, hopefully. Now, that said, you know, today, our ONU's, our Pawn head end is typically in the same building, so we're not super worried about it. But actually, in some of our campuses, we actually serve adjacent buildings from one common building. And because of Pawn's ODN being quite large, I mean, in residential appointments, you routinely see ODN's 20, 40, or 60 kilometers wide, we're able to serve adjacent buildings quite easily. But even then, currently, our data center campuses are considered secure. They have tall razor wire fences and guards at the edge, so we're not super worried about people getting to the conduit between the buildings. But it is something we do run and we are looking to enhance. [00:38:48] Speaker 6: Patrick Delmore, I'm affiliated with TIGR. You mentioned that you hadn't put this into any of the POPs. Do you, and if you said this, I'm sorry, but I was trying to pay attention. But you see any problems going forward when you put into third-party data centers? Are you worried about, like, trying to get the thing on top of the racks if they do the racks? We won't go on top of the rack in a POP. [00:39:09] MJ Mike Joseph: Okay. So, as I mentioned, other rack products are under development. We'll probably use, like, a 1RU design. So, for those that aren't familiar, an open rack standards, 1RU is both wider and taller than an RU. In our data centers, that's a reasonable footprint because all of our racks are open rack. But in POPs, we use industry standard 19-inch standard racks. So, we would probably, and we, in fact, many of our racks in POPs don't even have roofs on them in which to mount the pond canopy. That's awesome. So, to that end, no, we would expect in POPs to go in with an RU device. [00:39:40] Speaker 6: And do you see any problems with that? Like, getting 80 or 90 serial ports into one RU seems... [00:39:48] MJ Mike Joseph: Well, I'll spill the beans a little bit. We're actually working on some emerging solutions. Our mechanical engineering teams have been working with our cabling vendors to come up with a higher density ethernet... RJ21s? RJ21s are hard to source. So, MRJ21 is hard to source. It's only made by TE connectivity and they're... It's unclear about the long-term longevity of that solution. But we've been coming up with some creative higher density ports. I think the current possibility that they're floating is repurposing a mini SAS connector for a three-way breakout. And our current CAD designs are able to get quite a few of them on a faceplate. Thank you. And then one thing I do want to mention as well, because Patrick reminded me, in terms of going into RACs is today, again, the pond chassis is passive. But if we go into, say, a POP, we would probably have to use active power supplies to drive the ONUs. That would compromise our statement about no shared fate. But in POPs, we don't generally use, sort of, cell-based switches. So, we don't really have to worry about the HA network sharing the same chassis as the NA network. We probably wouldn't put an HA network into a POP. And one thing I will mention, too, about going on top, the pond canopies mount natively on top of OpenRack V3. And so, META, as part of our OCP work, is enhancing the OpenRack V3 standards so that the pond canopy you saw in the picture will have mounting features on the OpenRack V3 rack. So, if people buy OpenRack racks from OpenRack V3 manufacturers, they will have mounting features. So, you could put a canopy enclosure on them if you wanted to. [00:41:30] Speaker 1: All right. Thank you, MJ. That was really informative. Thank you very much. [00:41:33] MJ Mike Joseph: Thank you very much.