Google Technical Program Manager Mock Interview: Data Centers

[00:00:00] Speaker 1: can you describe the type of failure scenarios that you might see in a data center hey everyone i'm super excited to be here today with adam to discuss a tpm mock interview we're going to be talking about data centers today but before we jump into it adam as usual could you [00:00:20] Adam: share a little bit more about yourself absolutely yeah i have experience in data center consolidation and infrastructures based on my experience working active duty in the air force also migrations in accenture capacity planning at google and most recently as a consumer of aws at airbnb awesome [00:00:38] Speaker 1: well let's jump right into it so um the first question that i just want to ask you right off the bat is can you describe the type of failure scenarios that you might see in a data center great question [00:00:50] Adam: there's a lot of different ways that we see failures all the time in data centers and you know i think it seems like a pretty far away concept but these are these days are hosting data for all the applications that we use and love every day the first i'd like to mention is just component and server level failures so just think about what actually a server is it's a machine with a cpu ram disk and actually running whatever application we're trying to access there are failure rates of those rams and cpu and at google scale when i was working at youtube in the hundreds of thousands or even millions of servers we'd see failures every day in fact there's a dedicated team card hardware operators who are riding around on scooters and swapping out components on a daily basis another somewhat common failure scenario was site power outages this can happen you know it's not so much on a daily basis but if there's a failure in the power grid local fire or disaster going on or even sometimes cases with diesel generators you know people who are familiar with data centers also know that they have backup power in the form of batteries and flywheels so definitely a lot of different things that can go wrong those type of power sources though are primarily used to move from when the power goes out and then trying to get the diesel generators on and i have to mention with site power outages being very real think about the seasonal hurricane season in the us specifically the south east and that every year we've sort of seen exercise of moving our jobs and loads from one site to another sort of trying to be ready and anticipate for this and then of course there's the least popular but probably uh you know the number one reason why we see uh downtime or failures which is changes in software or configurations uh you know unfortunately this can happen if uh source app uh someone puts out something there wasn't proper key way or testing done on it or if you have uh the situation where you're with a third party cloud provider an employee over there could make a policy change revoking uh revoking all of the access on accident so these type of things definitely happen totally um thanks for coming [00:03:04] Speaker 1: those scenarios um i kind of want to change gears in the interview just a little bit to ask you a follow-up question that's a little bit more like an estimation question um and so sometimes in the interviews will sort of uh push you around in different ways um but the question that i have for you today is um you know could you estimate something like how much power or maybe how many wind turbines that it might take to power the global data center footprint of a company like google [00:03:27] Adam: got it uh so just so i understand correctly and i'll give you a little idea of how i'm going to approach this i'll first try and estimate how much power the data centers consume and then how much they generate and you're sort of looking for a number of sort of instantaneously or today how much power we'd use that's right yep okay great um yeah so i would say just on the number of data centers alone there's 20 and there's actually publicly available information for google uh 12 are in the us five are in europe two in asia and one is in south america um off the top of my head i remember that data centers consume about 15 megawatts or more of power but i want to spend a little bit of time and validate that is that okay with you sounds good all right yeah so a server consumes about 200 watts of power and when you have multiple servers in a rack a rack is just like a cabinet in the data center uh about 50 servers uh can go into this rack so if you multiply those two numbers out that ends up being about 10 000 watts or 10 kilowatts per rack uh so now i have to think how many racks there are in a data center uh can depend on the size of the data center but i'm going to use somewhere around 2 000 racks just based on my experience of seeing having seen this number and if i multiply those throughout that ends up being about 20 000 kilowatts or if i go and translate over the units 20 megawatts um so this is how much a single data center would take but it is google and so we have to think that things are a little bit bigger i'd like to add a scale up factor here of about 2.5 just because there's multiple data center facilities on each of these given campuses that we're trying to estimate so i'll use 2.5 multiplier on top of my 20 megawatts that would tell me that each site is using about 50 megawatts and so if i go back i told you that there were 20 different sites uh each consuming 50 megawatts that wound up being about a thousand megawatts if i translate over that's one gigawatt of power um yeah so now that i have the power consumption of the data centers i'll move into the second part where you mentioned the wind turbines and power generation um like off the top of my head i actually don't know how much power wind turbine can create but i remember from my graduate research that solar power is about one by one meter can generate one kilowatt of power so i'm just going to assume at this point that one wind turbine can do about one kilowatt and so because i estimated one gigawatt that ends up being uh one million wind turbines to power google's data center footprints got it that's a lot [00:06:13] Speaker 1: of wind turbines um do we feel like that was an overestimate or an underestimate or what would you change if you had more time to kind of analyze this answer yeah yeah definitely that's that's uh that's a big [00:06:25] Adam: number um but also using it and operating at google scale i think this is pretty appropriate um i think out of the framework that i laid out you know we can sort of go and change some variables so if a wind turbine didn't use or couldn't generate one kilowatt or if there were slightly different power calculations for servers or racks we can go back and adjust these things but i feel pretty confident and good with my [00:06:49] Speaker 1: number cool yeah that's awesome um well we can kind of like uh debrief or or you know pause the interview i would just love to hear yeah anything else you wanted to share with our audience or any other high level thoughts on how that interview was for you uh yeah i mean this is a challenging question [00:07:06] Adam: and i certainly think you know not having worked in this role before i probably wouldn't have been able to answer it so i didn't learn how to do this overnight uh definitely took practice and working with these different domain areas uh really the takeaway for folks is you know this question involved a lot of power and consumption also just understanding the space and size of a data center but that might not be your area of expertise and that's okay when you receive these type of questions it's important to focus on the process and sort of be able to regenerate and recreate the scenario that you did to create your number again um so yeah i mean i also i want to harp on you know really introducing those different factors and being able to defend them and using this process you can probably go forward and tackle any question because the type of interview questions you get they'll be looking for you and how assessing how you tackle ambiguity totally um well i thought [00:08:02] Speaker 1: you did a great job answering and um very thoughtful and you went through it in very good detail obviously in more time digging deeper into some of those things around how you might have changed your answer or trade-offs or scenarios where you might change it um are always really valuable but overall i thought you did an excellent job um and it's always a pleasure having you here adam so um thank you again for being on the show to to provide your answer all right thank you very much thanks so much for watching don't forget to hit the like and subscribe buttons below to let us know that this video is valuable for you and of course check out hundreds more videos just like this at try exponent.com thanks for watching and good luck on your upcoming interview