Large Scale Optical Switching in Google Data Centers

[00:00:00] Speaker 1: First thank you very much for having me. Thank you for Tim, Paul and obviously the committee for asking me to speak. It's a pleasure to be here. I've known about this conference for a very long time but I've never been able to make it out here. What I will be sharing today is the result of not quite a decade of work in stealth for a long time. People have been wondering and asking me non-stop, what are you doing there, Kevin, beyond eating free food? So today you're going to hear about it and before I go into my talk, some of you may be aware that we're having our Google Next Developers Conference right now and our Senior VP of Infrastructure Erz Holtzler last evening shared a video with the world and I'm just going to play a small snippet of that if it works. [00:00:55] Kevin: But now, do you want to hear something else that is cool? Because I'm hoping that you will let me geek out a little bit on a piece of hardware that probably none of you have ever heard of. Because my other job is building cloud infrastructure and its goal really is to be invisible so that you can all have developer experiences without having to worry about sort of what's underneath. And so I'll show you today a little piece of hardware that is called the optical circuit switch. Because you see in our data centers, we obviously have thousands or hundreds of thousands of servers and they're all connected with fiber to thousands of switches and these switches in turn are connected to each other. And so when we install new racks with new switches, these connections need to be rewired because those new switches have to be linked in to the mesh of the existing switches and that's where the optical circuit switch come in. Imagine doing this rewiring of all these connections in a complicated pattern manually. Right? It's really impossible to do almost. So we built an optical switch from scratch to do all of this in a programmable way. And here's how it works. So you see, you have all these hundreds of fibers coming in, each thinner than a strand of hair and carrying a laser beam with the data. And all of these beams have to be perfectly aligned so they hit this array of tiny mirrors here. And by tiny, I mean tiny because this is a greatly magnified picture of this chip with this micro mirror array. And in reality, each of these mirrors is less than a millimeter wide. And they can be all be adjusted independently by applying some voltage to it. And so as I said, the laser light comes in on one side. And now we need to aim those mirrors so precisely that the laser beam perfectly hits the end of an outgoing fiber strand. And so that the data can go to its destination on the other end. And that's why it's really amazing that it works at all. Imagine having to hit something that is thinner than a strand of hair head on with a laser going through mirrors. But it does work. And it works really well. Thank you. And it's been a workhorse really in our data centers in 2015. And so I'm guessing you never really thought about that problem. And in fact, you don't have to, because that is our goal for infrastructure, to be invisible, to allow you to build awesome developer experiences, and never have to worry about any of this. [00:03:58] Speaker 1: Okay, I'll keep going. So again, that was shared last night. If this MEMS thing doesn't work out, maybe I'll have an acting or hand model career. So again, I'm Kevin. I lead MEMS and all of the optical switching that we use in Google data centers. Today, I will be speaking about one version and one application of that. You saw the animated version of this. It's much prettier than the still version. And I think most of you know what an optical switch is. It's been discussed for a very long time in the industry. And what it does is it basically steers photons, right? You send photons in and any of the input fibers, and I can send it to any of the output fibers. I don't look at the data. It just bounces off of highly reflective gold mirrors here. And this is what we've created. It's the Google OCS, our homegrown. We call it Palomar. And in the diagram before, I only showed three inputs, three outputs. It really has 136 inputs and 136 outputs. So that means there's 272 fiber connectors at the front of this box. It's a fairly large box, almost the size of the top of this podium here. And we made it in-house. Every single component in this was designed or defined by Google engineers. And since we're in San Diego, our namesake was actually the Palomar Observatory. We had a number of team members that loved telescopes, and so we did that just north of here. And the interesting trivia fact, if someone were to take the time, you could place about 25 million of our individual MEMS mirrors on top of that very large reflector. Okay. So obviously, we're not as big as that reflector right now. So what have we accomplished here? As you heard Erz describe there, we have produced the world's first large-scale deployment of optical switches. We have tens of thousands of switches that have been manufactured, and they are used in all Google data centers worldwide right now. Okay. That means that we have millions of MEMS mirrors working hard every single moment of every single day. Okay. And we believe that we have built more large, obviously not the small ones, lots of people built small ones, large optical switches than all other companies combined in the history of the world. When I give lab tours and talks internally and things like that, I like to tell people that, as far as I'm aware, we are the OCS industry right now. Okay. And in order to understand what we're doing with it, we've got to step back a little bit. So let's go back in time. And so here is the Jupiter architecture as we described it about 10 years ago. And in that architecture, we have this top-level spine that connects all of the aggregation blocks and also connects to what's called the fabric border router. The fabric border router either talks between campuses and data centers or to the rest of the world, right? So if you want to watch a YouTube video, it eventually has to go outside to the fabric border router. And beneath that are the top of racks, which is -- and here I'm just showing machines, but it could be different types of things, different services, storage, et cetera. Okay. And that architecture has powered Google's entire business for a very long time. And these data centers have to run 24/7. And they provide basically the heart of all of -- any application of Google service that you run. Gmail, photo, search, translate, ads, cloud, which is a big one obviously now. If you look up directions on your maps, it is ultimately going through a Google data server -- Google data center, sorry. And eventually, that data would be bouncing off of MEMS mirrors. Okay? So let's keep going here. So there is an unfortunate downside of the spine-based architecture. And that means -- and what it is, is that it makes evolving and upgrading them quite expensive. Okay? So we always want to have the latest and greatest in the data centers. You want to go to 40, 100, 200, and people are already going beyond that speed right now. And sorry, these are gigabits per second. And in addition, when you bring up a data center, if you notice carefully in those videos there, we -- not all of the banks were full. So sometimes, like, you bring up a partial data center because you don't have, you know, all the services ready or you want to spend all that money. So you bring up the shell. Okay? And the spine is almost always put in at the start -- it's like the backbone, the nerve center of the data center. And it's deployed when you bring up the building. Okay? And -- but what happens is that as new technologies come on or you want to upgrade your data center -- like in here in this instance, here in this diagram, I've put a 100 gig block in, but I actually can't talk at 100 gig to the other services that are within the data center. So now I'm actually slowed down. I'm actually prevented from using the latest technology in that data center. And what about just rip it all out and just put a new one in at 100 gig? You can. Okay? But it's very expensive. Okay? At the scale that we operate, the number of data centers, the number of optics that we do, it is a very expensive thing to just rip everything out, put all new things in. But it's also very disruptive to services, right? If half of the building is full, I can't just tell YouTube, sorry, we're going to shut you down for two months, right? It's a very expensive thing. So what have we done over that time frame? So what we've done is instead of having a spine-based architecture, we now replace everything with optical switches. So at the top of every single data center, at the heart of Google's business now are OCSs. And we can now, in addition to have an easier path to upgrade, we can also allow software to handle topology, routing variation, application-specific needs in real time, all handled by software. You don't want humans involved in this, okay? And it also allows us to do incremental upgrades much easier. And again, letting software handle the details so that it's a hit list upgrade or an expansion so that it's non-disruptive to the existing services that are running in the data center. So what does this give us? It gives us a very large reduction in the capex. It's very significant. It's a very large amount, but also gives us a large amount of power reduction. There's a lot of -- sorry. And I'm only talking about the networking layer here. Sorry. I'm not talking about compute, obviously. It allows us a much faster, touch-free path to upgrading them. It allows full support for a heterogeneous optical transceiver environment because the OCS doesn't care. It's just mirrors, right? I don't care how you encode it as long as you stay within a certain wavelength band. And it supports live data expansions and refactorizations, as I described there. The downside or the challenge or the drawback is that it is more complex, right? You have to deal with all that topology, and you need routing control. You need a lot of software engineers. Google has a lot of software engineers, so, yeah, they figured it out. So here's a schematic of a data center floor. I'm just showing, you know, just simply, you know, we have the server halls, and you saw nice videos of that, too, there. But at the bottom now, we've added this thing called the Apollo zones, okay? And these are these big cages where you stick the OCSs because if somebody bumps into them or does other things, you don't want that to happen to the network. And here's a photo of one-eighth of an Apollo zone, and you saw some pictures of that in the nice video that was shared. And in this picture, there are 32 Palomars right now. So there are up to 256 Palomars per data center floor, and I've been to a bunch of data centers, and I love walking down the halls one at a time, and I just see endless rows of Palomar OCSs, which is nice, okay? So what does it take to build something like this at scale? I will absolutely admit, we are not the first to make an OCS. A lot of people have tried making them, and I kind of see it akin to, you know, why would Tesla bother making their car? They weren't the first to make a battery-powered vehicle. Lots of people have done this. But we've brought it to scale, and it requires not just having, obviously, the box itself. You have to hit a lot of things. You obviously have to hit the performance, okay? You obviously have to hit the cost, right? If it's too expensive, it doesn't make any sense to do this. It has to have high quality, reliability, and it has to be manufacturable. I don't care if you have the most beautiful box. If you can't make, you know, thousands and thousands of these things, it's not useful to me, right? And we actually started working with commercially off-the-shelf vendors to start with, and what we quickly found is that they were unable to meet our needs for a variety of reasons here, performance, cost, quality, reliability, and manufacturability. And so we made the decision to bring this in-house. I was hired to basically pull together the team to make this and turn it into reality, okay? But I do want to, you know, acknowledge that there are other ways to do this. But in addition to the factors that I showed on the previous slide and, you know, some that I'm showing here on this very brief or small table here, when you take into account all factors, MEMS-based optical switches satisfy all the requirements for large-scale deployment and, again, in an actively-switched data center environment. If you're never switching and all you want to do is do sparing, yeah, go, sure, you can use a robotic system, right? And robotic systems have matured a lot over the years. They can go to very high port counts right now. They have incredible loss, right, because you're just butting fibers up together. But they're slow, unfortunately, right? So it takes about 40 to 60 seconds to move a single fiber, right? And obviously, you know, if you want to have a dynamic environment, that's a non-starter, right? What if I want to move 100 fibers? What if I need to move 8,000 fibers, right? And I don't want to do that all in one second. And I don't want humans involved. Let the software take care of it. You can't do a robotic solution. There is a company that still does Piezo, and there's a number of people that have done that. They are very expensive. And there are some new technologies that are coming out that I'm watching very carefully as well. But again, they tend to be very low radix and have very, very high insertion loss, which is one of our most important parameters, as I'll show in a slide coming up here. So let's look a little bit more at the Palomar switch. It's a beautiful thing. At the one side of it, you have the fiber management. And that's, again, all of the 272 optical fibers that eventually would get connected from the other side. We have the optical core. The most important part of it, obviously, is the MEMS. There's a lot of other stuff in there. And then we have all of the printed circuit boards and the software. And that's what communicates with the outside world. And in this case, it's the OCS controller system, okay? If we look at the back half of the chassis itself, again, this chassis was designed for reliability and serviceability. So the high-voltage driver boards that control all of the MEMS mirrors, they're hot-swappable. I don't have to power down the box. I can pull a board in, slam a new one in, okay? All the mirrors will then reconnect, okay? Fan boards, the 3+1 redundancy. Fan board can go down. I don't care. Power supplies, designed with 1+1 redundancy. Again, the box will never come down even if I yank out a full power supply. Of course, it will go down if I take out the brain. But, you know, there is the CPU board there as well, okay? And let's start looking a little bit at the front hood of it. So I realize there's a lot of Cal people here. So you notice something about it? You notice something about the colors of the fiber? Yeah, it was, so, you know, when we're developing this, I said, you know, as the tech lead, I said, "No, it has to be that color because it's very important for the performance of the product." And people believe me. So anyway, so we have the high-voltage cables there, and that, again, connects to the MEMS. We have the optical core, which is a little hidden right now. I'll show you a little bit more of that in a sec. That core is obviously put on a big shock system because you can't have this thing damaged during transit. You'd be surprised at how abused these things get, and then the beautiful fiber connectors. If I take off the cover of the core here and start exposing a little bit more, you can see some more of the details, and I'll show a little bit of that in a different slide right after this. But you have the fiber collimator arrays, and again, that's the inputs and the outputs, right? So that's the outside devices that all want to communicate here. You have two MEMS mirror arrays, and I'll show you the pictures of that. And you have these dichroic splitters and combiners. And then we have a camera and injection modules, and those are part of our servo system. So in real time, I can watch every single mirror in real time to know where they're pointing, because sometimes they do misbehave, so we know what to do with them and control them. So this is a schematic view of what you just saw. Hopefully, it's a little bit easier to understand. And again, on the left and the right, you can see the fiber collimator arrays. And I'm showing one laser beam right now, which is a data path laser beam. And so it comes in into the optical core, bounces off of one of the optics, bounces off a MEMS array, another optic, the other MEMS array, another optic, and ultimately it gets routed to an output fiber. And here I'm just kind of showing close to a straight path, but obviously you can envision that, you know, the MEMS mirrors could be tilted and it could be any different configuration. In reality, of course, when you're actually running this in the data center, you could have hundreds of beams crisscrossing through this core, right? Because you have to have one for every output, but here I'm only just showing one. And they could be, you know, at various angles depending on what you need to do. And again, the camera and injection modules there, which provide out-of-band light here at 850 nanometers, which is far away from the telecom wavelengths, which allows it to pass through the various optics that you see in the core. And it doesn't obviously interfere with the data that you actually want to look at, okay? So you already saw a beautiful movie of the MEMS mirrors. So here's another picture of them. Obviously, they're gold-coated for high reflectivity. Optical loss is incredibly important to us. And each mirror can rotate in two axes to provide the deep beam steering function. And as you saw in the video, again, they're all less than a millimeter in size. So to me, that's not a big deal, as it's not a big deal to the rest of the world. They go, ooh, right? It's a millimeter, right? And those MEMS die are obviously put into hermetically sealed packages for reliability and the lifetime of the product. Here I'm just showing one of those die. It's been obviously pick-and-placed, wire-bonded, and hermetically sealed. And then on the right is a thermal image because it turned out to be very nice and pretty. And you can see actually a data beam on the center mirror there. And obviously, it shows very hot because those laser beams are quite high power. And you can just see the inner and outer comb drives that allow this mirror to rotate. And this is an electrostatic comb drive actuator, okay? So I've mentioned it, you know, here in different parts of the talk. Optical loss is an incredibly important parameter to us. Here I'm showing just one example of optical loss for all possible combinations. So that's 136 times 136. So 18,496 different possible permutations all have to be aligned, optimized, and peaked up for this product to function properly. And you can see for this one, the tail ended at about 1.5 dB. And this does include all front panel connector losses as well. Our manufacturing spec is 2. So we easily hit our manufacturing spec. And the mean is below dB, which for those that have worked on these types of things, it's an amazing number to be able to hit a dB consistently at this scale with this number of combinations. And for our own internal use, we do use three, however, for a lifetime. And that includes aging, shock, some mechanical shifting and movement of the system. But it also allows us to also include some budget for thermal effects as well, which obviously happens within the data centers, too. So what are some lessons learned? And I think you've heard some of these things in some of the other talks. You don't want a big team. You actually want a small, nimble, and experienced team that can make decisions. If you have too many people, you simply just get bogged down in meetings, okay? And I like to hire what I call T-shaped people. Not that they're big and brawny, but that they are very knowledgeable and deep in a focus area. But they also have broad experience and knowledge. Because for a product like this, you need software to be able to interface with hardware, right? If I make a design change on the MEMS, I'm impacting the optics, the mechanics, the software. I'm even possibly impacting the hardware layer of this product and device. So it's very important to have that type of thinking if you're going to, you know, attempt something like that on your own. You need to have a clear and well-defined benefit with goals. We did. There's a very large financial benefit to Google for what we did for this particular application. I hope in future talks I can talk about other applications that we also have for optical switching technology. The other benefit that we had in bringing this in-house is that we now have full-stack control, okay? So I have control over the silicon because I drew the dye. All the way up through the SDN layer, the software-defined networking layer, okay? And there's many instances where you need people to be able to cross boundaries that may be more difficult if you were outsourcing some of the aspects of this full integration within the data center, okay? And then the last thing I want to talk about is vendors and partners. It's very important that you pick the right people to work with. There are people in this room. I'll say a little bit more about that. You know who you are. Thank you very much for your support. And we do, as a company, develop long-term relationships with our trusted partners. I do want to give a particular shout-out to Elisa and her team. When I joined Google, I was the lone MEMS engineer in a massive company. I had no lab, no equipment, nothing. And her team definitely helped accelerate the development of what you're seeing here today, okay? So with that, I want to say thank you. But I also want to pause for a sec, okay? So when I give these talks internally, I like to start and I go, and I ask people, okay, how many people own a MEMS device? And typically, only about a third of the people raise their hand, right? Because they don't know that it's in their phone or their car kind of thing. I'm not going to ask you that because I know everyone would raise their hand in this conference. But I'm going to ask you now, how many people use MEMS when you use the Internet? There you go. Okay, so you have a new question you can ask people here. And then I do need to acknowledge, obviously, I'm here speaking today. But this is the work of hundreds of people, okay? And here I'm only showing a small number of people that contributed to this thing that we've created here within Google's data centers. There's obviously the OCS product development team that I led. There's also the data center optics team and the data center software and ops team. And this isn't everyone here. But you can see we have a very diverse team that represents the world because Google's products are for the world, okay? They're for everyone. We want everyone to use and enjoy them. With that, thank you. [00:23:20] Kevin: All right, are there any questions here? Can you talk a bit about how long it took when you arrived to when you had prototype to when you actually began that deployment process? Yeah. I didn't quite get that from you. [00:23:51] Speaker 1: Yeah, you may have heard ERS say something at the very, very end. But we've started deploying these in 2015. So from the moment I joined to the moment we started actually putting them within data centers was a little over two years. That's important. Okay, yeah. Well, you know, again, I had a lot of help, right? There's a lot of, you know, I recognize the people here. And then don't forget a lot of people. There's a lot of knowledge out there, right? You know, everyone, obviously myself, I've stood, I'm standing on a lot of people's shoulders to get here, right? And so there's a lot of stuff that made that possible. Yeah. [00:24:25] Kevin: I have one other question. Yeah. Yeah. So when we were all doing this, at one point it was trying to go optics through the entire signal chain, et cetera. Yeah. You still go to electronics? Yes. Where is that at the actual servers? [00:24:44] Speaker 1: So right now it's at the aggregation blocks and at the fabric border routers. [00:24:49] Speaker ?: Yeah. [00:24:50] Speaker 1: That's what we can talk about today. Okay. Next time. Yeah. Great. [00:24:59] Speaker 4: Impressive to have millions of installed mirrors. That's great. Yeah. And actually I love your concept of T-shaped experience on your team members. I've never heard that one before. Yeah. It makes a lot of sense. I did have one question. You talked in your competitive analysis about optical switching based on MEMS as contrasted to other technologies like robotics, et cetera. Yeah. But there have been other efforts at optical switches that were MEMS based as well. Yep. Could you elaborate a little bit on what you've done that's fundamentally different than companies like Caliant and some of the others? [00:25:40] Speaker 1: Sure. So let's see here. I don't want to give specifics on those companies, but we did obviously evaluate those types of companies that were available. And they did have deficiencies, unfortunately, that prevented them from being deployed at a very large scale. I will say that for our OCS, as you saw there, it was designed, I'd say, from reliability and the MEMS as the start of it. So the MEMS itself is very robust. We have a very good track record of them moving and surviving within the fleet for a very long time. And once you have that as a base and you build the system around it, it's possible to create something like this. Thank you. [00:26:24] Kevin: So you started deploying seven years ago. So why announce now? [00:26:31] Speaker 1: We felt it was time. Yeah. I wish it was sooner, but yeah. So I have a question. [00:26:41] Kevin: Yeah. Are all the data centers like this now? Yeah. [00:26:46] Speaker 1: This is used worldwide in all of the data centers right now. [00:26:49] Kevin: So you finished it before you announced it. Yeah. Yeah. Yeah. [00:26:55] Speaker 4: Yeah. All right. [00:26:57] Kevin: Is that it for the questions? We're not done yet. Okay. Well, let's give Kevin a hand. Okay. [00:27:12] Speaker ?: Okay. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah.