Data Centres – The Electrical Power System

[00:00:00] Orla Burke: So thank you to everybody who's joined us today for this one hour CPD presentation by Brendan Durban. This event is on behalf of the Electrical Division of Engineers Ireland and my name is Orla Burke, I'm chair of the Electrical Division. So we are so thrilled that you could join us today. Just like to remind everybody to keep an eye out on the events page of the Engineers Ireland website for upcoming events in the Electrical Division. And also if anybody out there has any topics of particular interest that they'd like us to arrange a presentation on, a webinar, a virtual site visit, please feel free to get in touch. And my details are on the Engineers Ireland website or I'm sure you can contact Engineers Ireland directly and they'll put you in touch with me. So without further ado, back to today's presentation. The title, as I mentioned, is Data Centres, the Electrical Power System, and we're delighted to welcome Brendan Durban today to present. Brendan conducts quite a lot of training with Engineers Ireland and he'll give you an overview of that later. And Brendan is also the Secretary of the Electrical Division, so he's double jobbing. So we'd just like to thank Brendan in advance. So just to give you a bit of an overview, Brendan is a Chartered Engineer, he's got over 40 years experience in all aspects of mechanical and electrical building services, including design, installation, commissioning and maintenance. After completing an electrical apprenticeship in 1982, he went on to study engineering in DIT. He has worked in M&E consultancy at senior engineer director level since 1990, and in 1999, he started his own M&E consultancy, Durban Engineering Consultants. His company provided M&E consultancy and project management services to a diverse range of clients in both the public and private sectors. In 2013, Brendan completed an MSc in energy management with TU Dublin, and his thesis researched the improvement of energy efficiency in existing data centres. Brendan, as I said, conducts a number of CPD training courses, and you can visit www.besttraining.ie for further information, or keep an eye on the Engineers Ireland website. Thank you again, Brendan, in advance, and over to you. [00:02:14] Brendan Durban: Brendan O' Thanks very much for that introduction, Arla, nicely summed up. Okay, folks, thank everybody for, we have a great tour on out here today, so thanks to everybody for taking the time out to this presentation. Brendan O' A very important topic it's actually headlines in the Irish Times this morning. Brendan O' data centres and the whole issue with connectivity onto the grid and grid congestion and all these issues. Brendan O' So what we're going to look at today is the electrical system, literally from the pylon into the back of the server rack. Brendan O' So what we're looking at here, so what we're looking at here in this image actually is inside a typical data hall and there are the server racks and within them you can see there's rows and rows of servers stacked up which consume lots of power. Brendan O' So each of those racks could be typically drawing 20, 30 kilowatts of power and that converts into heat and that heat has to be dissipated. Brendan O' So that's where all the power goes in the data centre so we're going to look today at the journey from the, from the pylon right into the back of the server cabinets. Brendan O' So and all I mentioned earlier on that I am I've retired and it's kind of semi retirement because I've taken up this role of trainer over the last two to three years. Brendan O' And I've developed a number of courses and the reason is I'm kind of passionate about engineering, particularly electrical engineering that was my field. Brendan O' And and I feel I will spend 40 years in the industry it's worth sharing that knowledge there's there's a need for it. Brendan O' And I like doing it so and it's kind of not really like work search so I initially developed one main training course on data centers and I kind of. Brendan O' was going to leave it at that because there was a big demand for that and then I developed the medium voltage course which does. Brendan O' A big need in that area there's a big a knowledge gap, I suppose, in the building service industry and. Brendan O' And I developed a course in that and then, of course, largely it followed through on to develop a low voltage distribution course as well. Brendan O' And then I followed off to fill to fill the suite on the data center, I said, you know, uninterruptible power supply systems are UPS as they're better known. Brendan O' And then I have a course on that and then one on standby generators and that's really the whole those five courses is the real data center suite of courses and. Brendan O' incidentally, I've highlighted in red and just so I wouldn't forget it and I nearly did or that has mentioned it that the next day for the data center courses on the 1st of December next Wednesday week. Brendan O' So really, really popular course is regularly booked out a fully booked there it's very, very popular it covers and all aspects. Brendan O' So if you want to remind you later at the end you can book that directly with engineers aren't do the course I do or fundamentals of any consultancy which I developed for people coming into the industry. Brendan O' And coming out because people are committed to building service industry from all different career paths now and it's just to. Brendan O' I suppose to praise them on the different aspects of what a consultant actually does. Brendan O' And the most popular course by far is this bottom left on one, the new wiring rules. Brendan O' came in in December in February of this year and so on new installations have to be designed to the new standard here, which is the bottom. Brendan O' left hand image and the book beside it is my own guidebook which I just published so i'm not i'm not here to plug that but just to let you know and that's a guidebook it's a kind of. Brendan O' It kind of demystifies the wiring rules they're very, very difficult to read and understand find your way around them, etc, so the book actually goes through that the entire wiring rules and explains men. Brendan O' In easier language, I suppose with lots of diagrams and charts and things like that. Brendan O' I do the two courses i'm working on those the moment i'll be developing those in the coming weeks that the standards for emergency light and fire alarm are also going to be updated and changed significantly so i'll be. Brendan O' developing courses on these which hopefully i'll be running through engineers are and again next year. Brendan O' So without further ado i'll jump straight into today's topic, which is the electrical power system in data centers so this little graphic is actually going to. Brendan O' Some of what we're going to be talking about today so obviously we take power from the transmission grid and it comes to HP switch care high voltage switch which which is usually owned by the ESP. Brendan O' And we transform down to the transformer here from high voltage down to medium voltage now this whole aspect of it, the connection to the transmission system, the high voltage switch gear. Brendan O' And the high voltage transformer it's usually done as a specialist contract. Brendan O' outside of the normal building service contracts for the normal building service engineer to start designing the electrical system in the in the data Center. Brendan O' doesn't very often get involved in this and and I won't be either because and it's not my area of expertise it's really from here down from the envy switch career. Brendan O' downwards that the electrical engineer on the data Center gets involved in. Brendan O' So, first thing is the m&m v switch gear or medium voltage switch gear medium voltage is above 1000 volts up to 35,000 volts phone in Ireland, it covers our 10 kv system and our 20 kv system. Brendan O' And it's either one of those two voltages you'd be distributing around the site. Brendan O' So then, when you get closer to the data centers the buildings where the data is being processed, etc, you have to transform back down obviously a safer level of voltage which is 400 volts. Brendan O' And we transform down to these mv lv transformers. Brendan O' And you supply into a main board here, so the transformers usually quite close to the main board. Brendan O' And then you have a generator feed now, so you obviously have to have a backup generator because and if in if the grid fails, the show has to keep going in a data Center it's processing all our. Brendan O' All our data or Facebook pages WhatsApp messages and of course much more important things like banking transactions and all the other sorts of things that are running in the background, because everybody runs their business on the cloud now. Brendan O' So that's the diesel generator provides the kind of long term backup if there's a long outage but there's also a ups system now the ups is a battery backup system stands for on interruptible power supply. Brendan O' It basically is a battery backup system that provides a no break backup to the power system, so if the power fails the ups continues to deliver power with even a microsecond of a break just continues to feed. Brendan O' So that's the power to the power to the central load, which is, of course, the servers and then the battery the generator starts starts up and after about 10 minutes to sorry about 15 seconds, I should say the generators online and it takes over and the ups batteries start to recharge again. Brendan O' So this is what we're looking at today this with the main focus here from the medium voltage switchgear down. Brendan O' And the right hand graphic is just showing it there big big transmission pile in there, whatever that would be a 220 kv or 110 kv. Brendan O' And the Paris is going right back into the back of the rack, this is the the servers the back of the server rack. Brendan O' And you can see the power coming in here the red blue leads are the cables going up onto the actual servers and so we're going to talk you through all of that. Brendan O' Right and the main components in the power system that we'd be looking at there so medium voltage switchgear that's what it looks like so you can see there on the top left hand image there's there's five cubicles we call them. Brendan O' And each of those cubicles is a circle breaker so the equivalent of the little trip switch in your house what you call it an mcb or the trips which some people call it. Brendan O' So that's what each of those are but obviously they're handling vast amounts of power so it's much more complex much bigger etc. Brendan O' And then the transformers which convert the power from high voltage down to lower voltage that's what we're seeing there and we'll be looking at that. Brendan O' And the generators of the generators that's the typical generator there, you can see the diesel engine is in the middle it's just the same engine that you have in a big truck or anything like that. Brendan O' and radiator on the end of it, and then it's driving the alternator on the left hand side, the alternator is just literally a synchronous motor if you like and when it's powered by an external source and give them some field current it generates voltage. Brendan O' And that's the generator and then the ups system here is. Brendan O' This is what we call a static ups system it's stationary there's no moving parts other than the cooling fans in it. Brendan O' So it converts DC power to AC so it takes battery power and converts it to AC and gives you a kind of a short term backup while the generator is starting up. Brendan O' And and then the low voltage distribution system, so you can see that board that's a typical data center board that you can see the length of that that's probably 30 meters long that board. Brendan O' And probably, you know, probably half a million quid to work with switch gear that that's it that's the type of thing as someone quoted once me power they send it does not come cheap it certainly doesn't so that's what we're going to be talking about anyway. Brendan O' So what's the big issue data centers getting a lot of bad news, a lot of bad press at the moment everyone's blame them on our crisis. Brendan O' And the greater shortage of power and everything and and obviously they have to have they have contributed to it because they're they're close on I think one one gigawatt of. Brendan O' power heading that direction at the moment, so our maximum demand in the grid in Ireland is about say five and a half gigawatts or something like that. Brendan O' And at the moment approximately one gigawatt of our demand is due to data centers so and some people are saying that the problem I think they're. Brendan O' really are going to be part of the solution because and generators are going to be able, because they have an inherent backup power systems they're able to come off the grid at a moment's notice and. Brendan O' And that's the whole issue that was in the urge to the article in the Irish Times this morning they're saying that. Brendan O' data centers going forward when they get their their contract from air grid or ESP whichever it is. Brendan O' it's going to stipulate that they must have their own emergency backup power system so there's no surprise not because they are all day centers have emergency backup power systems, but the difference is now they're going to tell them, you can have the power, but if we have. Brendan O' a demand issue and our. Brendan O' You know there's an emergency power issue we're going to have to take you offline and and they're going to have to sign up to that agreement and I can't see that being a problem, and I think they should be incentivized and even the older data centers are here for a long time. Brendan O' incentivizing them to come off the grid at peak times and and mitigate that risk that we're facing into at the moment. Brendan O' So what i've shown here and just to explain for the whole issue is that the level of power data data center takes. Brendan O' If you look in the blue table up here the top left it shows you different typical buildings offices schools residential complexes hospitals. Brendan O' And you can see their power density is quoted and usually in Watts per square meter and the what this the square meters obviously refers to the net letable space on a school, you know the classroom space it doesn't include. Brendan O' plant room space or anything like that so it's the net letable space the building, so you can see for a fully fully air conditioned. Brendan O' office building it's quoted around 82 more it'd be more typically around 16 now because air conditioning processes have changed and they've gone for natural ventilation a lot so. Brendan O' typical office might be taking 60 watts per square meter of. Brendan O' demand down to something like hotel might be only 25 or a church or something like that bank small thing like that 15 watts you know there's hardly any load in it at all. Brendan O' But if you compare if you look at that i've just told those into a sorry the data Center one Center shown down the end low medium and high density. Brendan O' 500 Watts per square me or up to 2000 kilowatts per square meter have you told them to the graph, you can see see see the difference so high intensity day data Center is 2000 kilowatts that's two megawatts and. Brendan O' That big part to to two kilowatts per square me or two megawatts two kilowatts per square meter. Brendan O' To put that in perspective, if you can we can all imagine an electric heater you can see them power city or wherever Harvey Norman you know what it's like and plug in electric heat and you can feel the heat off. Brendan O' Imagine just having 20 or 30 20 of them in the one rack. Brendan O' And then thousands of racks in the one building that's that's what your face is so yeah and that's how you end up with typical data Center has thousands of kilowatts which we we call megawatts, of course. Brendan O' So i've done a little comparison down here different buildings, so a medium sized office. Brendan O' Okay, when a net level area of 10,000 square meters or only includes the office space to circulation routes, etc, it doesn't include the plant rooms. Brendan O' And the data Center that we're looking at them we're looking at the white space on a day Center the white space means the area where the data is actually. Brendan O' processed it's like the production line in the factory for all the racks and the service are so it doesn't include the electrical plant rooms and the ventilation plant rooms, etc. Brendan O' So to them compare them like with like 10,000 square meters in each case so 10,000 square meters will be a medium sized office. Brendan O' Like when the offices see along the north wall there and drops you know a banking headquarters that type of thing 10,000 square meters 60 watts per square meters gives you 600 kilowatts. Brendan O' or not point six megawatts to get the language right so we're talking megawatts right not point six. Brendan O' The same flow flow print for a high intensity data Center is 20 megawatts and that's typically what we're talking with. Brendan O' With data centers 20 megawatts so you can't just connect 20 megawatts on get onto the ESP and say this and i'm building the data Center can you give me 20 megawatts. Brendan O' And they make a connection on to the pathway that it doesn't work like that when you get to 20 megawatts. Brendan O' You have to take power directly off the transmission grid so that's the big difference, whereas anything at not point six megawatts you're coming off the medium voltage network 20 megawatts your transmission level so you're coming up connecting at 110 kv or 20 kv 110 or 220 kv sorry. Brendan O' So that's that's what we're talking about so the typical data Center. Brendan O' An overview now, these are very simplified schematics obviously and it's only so much you can show in a PowerPoint presentation board. Brendan O' You can see what happens here you have the you have the pylons here, so this is either 110 kv or 220 kv network and they literally break into the line. Brendan O' And bring it into a new they build a new substation dedicated to the data Center now when I say they the data Center operator. Brendan O' Obviously pays for all of this and very often develops the substation themselves what's shown in the green box, but it's the ownership of the ESP. Brendan O' So what you see inside the green boxes connection of one pylon comes down through a circle breaker that's the box with the X in it and the other switches then are called what they call disconnects. Brendan O' high voltage disconnects to be able to isolate the circle breaker. Brendan O' So the circle breaker on each side and then both supplies then are coupled through this arrangement here, so it gives the ESP flexibility to do different switching operations. Brendan O' So again, as I say that's not normally part of the building services consultants job and certainly i've never got involved in anything that level i'm just showing at a very simplified level at this stage. Brendan O' So the building service consultant normally starts here on the medium voltage side so you have the transformers here. Brendan O' converting power transforming power down from 220 kV or 110 kV down to 20 kV. Brendan O' So those transfer are typically 50 megawatts is typically what you might see 50 megawatts of power each day handle, which is enormous you know, this is the size of a large town, you know and. Brendan O' And we'll be taking that sort of power and then and the supply comes out medium voltage and you commit to an A board and a B board and we're going to explain a little bit about resilience and redundancy. Brendan O' Everything is duplicated in data centers if the A system fails, you have a B system to back up. Brendan O' that's typically done like this, you have an input circle breaker and you have a number of feeders going out feeding out to your other transformers around the site. Brendan O' And the two boards are coupled together here, so if, if you want to take one of the transformers out of service, you can join the two boards together. Brendan O' etc and lots of lots of different configurations so that's the basic high level overview of the power system and we're going to look at these very various components keep drilling drilling down all the way to the back of the rack. Brendan O' So if you see this diagram i'm just shown as an example there's four supplies on the A side and then four supplies in the B side so in the next diagram you're going to you're going to pick those up. Brendan O' And what happens with them so you see there's a one be one a to be two etc, so this is just one there's multiple ways of doing it configuring the power system, but this will be fairly typical, this is showing and. Brendan O' A data Center with four data holes and each data hall has three transformers feed and you can see there just say they hold food on the top. Brendan O' That's three transformers feeding into it and each of those transformers are typically something around three MVA three megawatts. Brendan O' So that means there's nine megawatts of power going into that data hole right and all that's in it. Brendan O' is racks racks after rack after rack just full of service processing all our data and storing our data and processing it on our. Brendan O' As I say, what's that what's that message is our net netflix movies our bank accounts everything we do smart vehicles going forward smart cities. Brendan O' All that's what they're all there for the processing all that data and and saving a storing it on etc. Brendan O' So, in this case you have three transformers so there's about nine megawatts of capacity going in there, but the servers don't use all the power so nine megawatts. Brendan O' If nine megawatts is going in this product approximately probably six to seven megawatts would be used for processing data and the remainder of the power. Brendan O' is used to cool the plant to cool down the service and there's different ways to do that we're not going to talk about that today, but you can have air cooling chilled water. Brendan O' refrigerant cool there's all different ways of doing it like obviously cover that in the main course of today Center course to do. Brendan O' So each of the data halls have three transformers so they're kind of really a mirror image of one another. Brendan O' And then you see there's a reserve supply in the middle, so the reserve supply is is like your plus one your spare system they have here. Brendan O' And that spare system then can take the place of any of the other transformers to the left or the right, so if one of the transformers. Brendan O' has to come out service this reserve supply can be switched in I haven't shown on the schematic how it's switched in we look at that later it'd be too complex to show on this diagram. Brendan O' And, and then the other thing you have in a data hall of course it's a it's not just all about service, the service have to connect to the outside world. Brendan O' And that's why most all day sense, you know, network rooms pretty outside fibers come in and they go in and link to all the. Brendan O' The various data halls and that's run through the network buildings and, of course, you have admin then you've meet rooms, toilets offices. Brendan O' canteen all the normal things you have never do the building in a data Center as well, because they're they're not all just their. Brendan O' On man, most of them have a permanent staff in them, a small amount of staff in them, and they have to follow that normal other building functions. Brendan O' So and so let's look at some of the stuff down that appears so the end when we looked at the high voltage connection, first of all. Brendan O' there's two options and you can see the top image here that's what we call air insulated switchgear. Brendan O' And it's it's called air insulated switchgear because literally the conductors are insulated from one another by air, you know they're typically separated by. Brendan O' The phase one from phase two to phase three, etc. Brendan O' is separated in air typically maybe one and a half, two meters apart, as you see there, so the power comes in and the pylons drops down and you can see all these. Brendan O' insulators, etc, and these are things like current transformers circuit breakers and disconnect switches and things like that, so it takes a vast amount of space at this type of. Brendan O' switchgear arrangement so that's what we call air insulated or it's called as for short. Brendan O' And then gas insulated is the same thing we're being with high voltage but instead of insulating by air. Brendan O' So the conductors are all contained in these enclosures, which are filled with SF6 gas, which of course has a bad name from an environmental point of view, if it leaks it is. Brendan O' It has a very, very high negative impact on the environment in terms of global warming and ozone depletion and all those things. Brendan O' And so for that reason has to be very carefully monitored, but the advantage with the. Brendan O' gas insulated is it's extremely compact and probably somewhere around and one third the space that an air insulated switches switchgear installation will take off. Brendan O' So again, this is usually built by the data Center operator in Facebook and plenty and which probably a lot of people know about. Brendan O' it's out there, just to give you an idea that I worked on the first. Brendan O' phase of that which was three buildings and the the first connection was 120 megawatts that was for phases one, two and three. Brendan O' they're now on phase the building phase five I think or phase six if i'm not mistaking so and that's they have a massive began. Brendan O' air insulated switchgear like that and, you know, at this stage to probably have 200 megawatts of capacity on the site so it's just phenomenal. Brendan O' And smaller data centers might go for this air insulated switchgear arrangement, but in either case it's built by the data Center operator and then handed back to Eddie to to air grid, as opposed to the ESP. Brendan O' So that's really the two options when it comes to the high voltage side, then transforming down and i'm showing two different foot with four different types of transformer here. Brendan O' And the top one is where you transform from the high voltage down to medium voltage so high voltage just reminds us either in our end is either 220 kV or 110 there's also 38 kV on our system. Brendan O' And it's I don't think there's any data centers connected to 38 I think most of them are connected to either 110 and the data set the Facebook data Center is 220 kV. Brendan O' So that's what you have there typically that would be as part of an air insulated switchgear arrangement, you can see the bushings on the transformer here, so the the taller ones are the higher voltage so comes in here to 20,000 volts. Brendan O' Just think of that thousand times higher than the voltage in your house, and then it goes out the other outside at 20 kV. Brendan O' And the one down below then is another data Center I worked on this is a plant shock and this is 110 kV down to 10 kV. Brendan O' So it's a smaller transformer this one is 20 megawatts the one above it is 63 megawatts and so that's a typical and one that's fed from an underground underground supply from the ESP a remote ESP substation so it's kind of different arrangement. Brendan O' So they're HV to MV transformers the two on the right end are MV to LV so transform from 20 kV or 10 kV down to 400 volts so 400 volts 400 230. Brendan O' Is the standard voltage we use in and commercial industrial buildings etc in your house, as you know it's 230 volt single phase. Brendan O' For industrial and commercial buildings to use three phase and it's 400 to 30 so that's what these transformers transform it down to the final safe voltage that you can distribute it around the building etc. Brendan O' And the top one is typically about about three megawatts capacity and the one down the bottom is essentially megawatts that's a copy and paste error, it should be about probably about one megawatt or 1.6 MVA whatever. Brendan O' So and so translate that into a simple and floor plan what it actually looks like in the data Center. Brendan O' So again i'm i'm showing data center with four data holes I literally made this up out of my head it doesn't represent any particular data Center you probably never see a data Center like this. Brendan O' But to explain how all the where all the different equipment goes I just use as a simple little block diagram. Brendan O' So the outer dashed line here represents the perimeter fence of the data Center so when you arrive up to these places you can't just. Brendan O' throw your leg over the fence and walk up to the data Center highly highly secure environments for obvious reasons. Brendan O' And so they're usually have a very high protective fence around the perimeter and then, as you come in, you see the four green boxes. Brendan O' In the Center they represent the data holes and the kind of cream color at the top security admin meeting rooms, etc. Brendan O' And then on the outer side, then you'll see there's a kind of two blue boxes they're what they call the electrical plant rooms and. Brendan O' Now i'm not showing any cooling here where the cooling plant room because the cooling could be on the roof, or, you know, you could have a different configuration to cooling could be around the perimeter of the building. Brendan O' Or it could be on the roof so we're not getting into that today, this is just a. Brendan O' Notional schematic showing where everything would go. Brendan O' So what we're going to look at is is, first of all, the top left hand corner and top right hand corner. Brendan O' You see hvmv that's where your transformers where they grow in a compound so these are massive big transformers probably 20 megawatt each or 20 mv 80 is the proper term. Brendan O' And and the low cage to keep them apart from one letter in case that proverbial bomb hit one it wouldn't damage the other etc. Brendan O' And then somewhere else, which I haven't shown the schematic you have the esp hv substation this this substation that brings the supply in from the pylons and it's not showing the diagram here, because it could be anywhere, it could be any configuration, some of the sites. Brendan O' The substation is actually remotely sp bring cables from their substation into the site so that's what i'm showing here see if the two transformers the a transformer and the B transformer. Brendan O' transform hv to mv and then you have your mv switchboard shown in red you have your a board and your b board you'll start getting the field for very shortly. Brendan O' Everything is duplicated or has a spare reserve system backup. Brendan O' So what we're going to look at now is a typical data hall i'm going to take data hall one, for example, and look at what switch gears associated with that. Brendan O' So you can see, to the left of data hall one you have three gray boxes, followed by three green boxes, etc. Brendan O' And what they represent is the gray box represents the main lv switchboard. Brendan O' And if you look out from the main lv switchboard you'll see it's fed with a generator. Brendan O' And the green little green box beside it represents the medium voltage to low voltage transformer. Brendan O' So the medium voltage low voltage transformer has to be right beside the main board very, very close because the cables coming off it are absolutely massive so you try to keep that as close as possible to the switchboard. Brendan O' And likewise with the generator. Brendan O' The red box outside then is the the mv switchgear which feeds and the transformers these green boxes. Brendan O' are supplied at 20 kv typically from the red switchboard here and that module just repeats itself all the way around. Brendan O' So, in this case, if you want the basic bin building block is about a tree mv a transformer and a tree mv a generator feeding into where a board. Brendan O' And that board and fit torn feeds down into the ups system. Brendan O' And the ups system then in turn feeds power directly to the server will not directly there's a whole. Brendan O' series of other equipment to feed through before it gets to the service but that's where the the servers draw their power from. Brendan O' So the other stuff that would be fed from the board we haven't shown here is the cooling plant. Brendan O' And things like lighting small power all of those other things that a building needs to function. Brendan O' But where our focus today is on on the server so that's all i'm really showing here in it. Brendan O' So you can see that that really is a schematic representation of what we have a plan version of that schematic really that's what I was trying to create that in plan form. Brendan O' Okay for data halls and each data hall has tree transformers tree generators and then in the middle, you have a reserve. Brendan O' So if any one of these transformers a generator fails, the system is designed so they can pick up power they can take power off the reserve system here. Brendan O' And we look at that in a few slides I think. Brendan O' Next one I didn't realize coming up so quickly Okay, so in this case i'm showing three. Brendan O' boards here then a reserve then another tree boards in the schematic i'm only shown two boards and a reserve just for simplicity and, as I say, if I put the six in you wouldn't be able to read the diagram. Brendan O' So this is typically what happens schematically now we're down at the medium voltage level now and getting in from medium voltage down to low voltage or we're down a good bit down through the power system at this stage. Brendan O' So you have your transformer convert medium voltage to low voltage you have your generators feeding into the board. Brendan O' Now, if we look what happens in each board each board feeds down and feeds through this thing called an ats and i'll explain about that in a minute it stands for automatic transfer switch. Brendan O' But the power comes through that and then feeds the ups system which feeds the servers. Brendan O' L and sp stands for lighting a small power so all the lighting of the building small power small fans and things like that. Brendan O' And then the make me it refers to make a cooling system so they're the three main loads really have fed off each board in each day to all ups lighting a small power and the cooling system, whatever that is chill water fans, whatever it is. Brendan O' Now you see each of those three loads is fed to the ats that are referred to ats to stands for automatic transfer switch and an automatic transfer switch just allows you to transfer the power from one source of power to another. Brendan O' Now, in this case, the red is the primary power coming off the main board Okay, so you can see the next system has a red board as well. Brendan O' And the blue is the reserve so the blue comes off the reserve system here. Brendan O' And allows it means if you want to come in and plan say I want to take this main board out of service and work on it. Brendan O' Well, I don't know I don't own my service on that off that board to sit down so what you do is you transfer the power over onto the reserve line. Brendan O' And then you work on mdb one your services, then you work on mdb two a month later and you service it and mdb three and four etc, and of course it also works in the event of a fault, if the system. Brendan O' power failed, the first thing we do is try and start its generator if the generator didn't work then will then it would transfer onto the reserve line. Brendan O' Okay, now, when we get further down you will see you can to the ups so the ups. Brendan O' which we'll talk about in a few minutes ups is your battery backup system that converts battery power into AC power and it's a no break supply arrangement. Brendan O' And that provides power down to your ups distribution boards and out to the load, so you see i'm shown and a supplies that rack then be supplies rack typically. Brendan O' And so it means that every rack has a nice supply going into it and a B supply. Brendan O' And then every server connects to an A and a B supply so you're bringing that resilience all the way down from the right from the HP network. Brendan O' Right down to the back of the rack that is a dual pathway to every single server if one supply fails in pickup it automatically picks up and supply off the other one. Brendan O' But to make that work on the ups, we end up having to have a reserve ups distribution system as well, so the top blue. Brendan O' line that represents the power distribution system for mains power and the bottom blue one coming down this way represents the ups reserve power system. Brendan O' And you see now they go through a similar switch but in this case it's not an ats it's colon s ts it's that which means static transfer switch. Brendan O' So it's it's static it changes off an electronic transfer system which the transfer from one to the other is in the order of air milliseconds, maybe very three or four milliseconds to transfers, whereas we're in ats it could be. Brendan O' You know 20 or 30 milliseconds you couldn't have that with critical load it would see it as an outage and your service would crash. Brendan O' So that's why if you're doing this arrangement, and this is not the only way it does hundreds of different ways, this is just one example i'm giving. Brendan O' And the critical load is that to draw it supply off this ups or this one and. Brendan O' So that's that's a typical range when we call that like a tier three data Center so tier three is usually in what they call an n plus one arrangement, so this n plus one, which we again will cover in a few slides. Brendan O' means your need plus one, so this data hall here is supplied by two boards it's a need and you have one spare system and this case the spare system is shared by another adjacent data hall. Brendan O' With mdb three and forward on it so that's that's quite a common solution quite a common architecture and it's probably in the most cost effective solution and a lot of the data centers use that are variations of it. Brendan O' Right so we're going to start looking at a few of the other components and the system now so and generators obviously play a huge part. Brendan O' In and data centers and as I said that we were informed in the Irish Times this morning that it's now going to be mandatory for the course. Brendan O' It's taken as read every data Center has backup generators anyway, and they weren't we're waiting on air grid to advise them to put them in. Brendan O' The big differences, they can know air grid will be able to buy law be able to force them to come off the grid and go on to their standby power, whereas at the moment. Brendan O' They have no obligation to do that and. Brendan O' But I think personally I think I think they will start engaging with air grid and evading of what they call the demand side management where. Brendan O' heavy energy users can come off the grid and assist in and mitigate against risks of power failure due to overloading on the grid. Brendan O' And they centers you know that have been incentives in place as we have had for years we've had a thing called the. Brendan O' The winter demand reduction incentives been around for decades where they encouraged big consumers to come off the grid between and either reduce their demand or transfer their. Brendan O' demand over onto generator between five o'clock and seven o'clock in the winter months November December January February. Brendan O' And by doing that air grid we're dividing that big peak that occurs when we all arrive home work we stick on the oven on the kettle and you sit down watch coronation street and you're from the kettle again or whatever you're up to. Brendan O' And if the demand spikes around that time between five and seven during the winter months and electric heaters and all that type of thing. Brendan O' contribute to it, so they encourage people to come off the grid at that stage and and that's kind of gone now and they've introduced the man side management. Brendan O' And I think they're going to have to work with the data centers and encourage them incentivize them to start using the generators at peak periods. Brendan O' So there's typically what the generator looks like it's and it's literally just a big diesel engine it's a V. Brendan O' V form engine, as you can see there, you can see, and this one is forever is six cylinders on one side six and the other so it's a 12 cylinder engine. Brendan O' And just industrial diesel engine and this little section in the middle is the alternator that's where the. Brendan O' AC current is generated and this big box at the end is where the conductors literally come up off the alternator. Brendan O' into the circuit breaker and then you have the control panel here which controls same way as the dashboard on your car, you know, it has your oil pressure. Brendan O' The coolant pressure water coolant pressure and the revs and everything of the engine so that's all done there and the other big thing at the end is the radiator which keeps the engine and. Brendan O' cooled and also cools the turbo air for it for combustion cools down the air for combustion as well. Brendan O' And that's a very popular made caterpillar we all know caterpillar engines around for a long, long time. Brendan O' And here's one that be made in iron for for decades FG Wilson are made up and learn in and outside Belfast and but they're now owned by caterpillar so they're effectively the same company. Brendan O' So some of the considerations if if you're a designer of a data Center you've designed the power system, some of the things you need to consider are. Brendan O' Do I go for diesel or a gas engine that's becoming more and more common if particularly with the new data centers now they're going to have to run for hours. Brendan O' off the grid with their own engine. Brendan O' diesels from an environmental point of view gas engine is better than a diesel engine, but there's a lot of disadvantages with the gas engine. Brendan O' And so he's going to generate where it will base load prime or standby rate is very important. Brendan O' The load transfer arrangement, are you doing a load transfer are you synchronizing with the grid and all the generator power and control cabling how that's done everything arrangements. Brendan O' The generate enclosure do you build dedicated plant rooms, you put it on the roof, you put it down at a ground level, etc. Brendan O' ventilation system they take an enormous amount of air which isn't shown here the diagram so for for to cool that radiator down when you're driving your car you're driving against the wind, so the. Brendan O' The wind is blowing through your radio and cooling it down for you, the generator sitting in a container or in a room so that the fan has to be driven by the engine and has pulls in. Brendan O' fast volumes of air which creates lots of noise, so you have to put it through an attenuator to reduce the sound pressure, etc. Brendan O' And so there's lots of considerations taken into account. Brendan O' And the exhaust system and you can see these data centers now they're building them all over the place and they're instantly recognized by the generator flues. Brendan O' The big polished stainless steel flues you see them popping up over the ridge of the building and they look like warehouses and all of a sudden you see these. Brendan O' shiny chrome flues upside the building and it's a dead dead giveaway that it's a data center when you see that. Brendan O' So where you put the exhaust system, how you get up to the building and then most importantly fuel storage for your your diesel generator. Brendan O' has to have typically 36 48 hours storage of diesel, which is a massive amount of diesel to store on the site. Brendan O' So, and it can be bulk stored in a central tank and like in a tank farmer pumped around. Brendan O' Which is rarely done not recommended, obviously, because the the whole distribution system becomes a single point of failure. Brendan O' So the alternative is now they're building them with the each generator has a tank underneath a base tank they're called in the industry belly tanks but it's just basically a big base tank. Brendan O' sitting here and it just draws the fuel directly off the tank below it. Brendan O' In fact there's the image on the left is an actual confidential DC site it's so confidential I actually don't know myself which one it was somebody shared the picture with me. Brendan O' But you can see them there they're the generators in the containers, so that these are normal shipping containers and are used for roll on roll off containers and you can see the flues coming up here. Brendan O' Chrome flues coming up and they're going up this stack here that's a lattice tower like it's a triangulated lattice tower. Brendan O' So, in this case has one flue going up in each of the three sides of the triangle. Brendan O' Okay, way above the building line there and then the next three generators goes on to the next flue stack and so on. Brendan O' And then you'll probably find that that's on one side, just as I showed you my diagram and then the mirror image of that over the other side of the data Center. Brendan O' And this one, I think, is a CGI but it's you get the drift it's shown there and the generators the flue system going up here. Brendan O' And they see this big space down here that's actually the belly tank on it now so that's the container to generate in the container and all of this space below is fuel storage to give you typically 48 hours storage. Brendan O' And the green box here is your your field field cabinet where the fuel comes in and. Brendan O' That's your field cabinet and then you have also what they call fuel polishing system there because the fuel. Brendan O' has to be kept healthy and what happens in in tanks is you get condensation of water and it leads to microbial growth. Brendan O' In the fuel and clogs up just turns into sludge in the end of the tank and then your generator starts up sucking up the sludge. Brendan O' and blocks the filters in the generator so to avoid that you put in and a fuel polishing system and it gets rid of all that sludge so it doesn't cause engine failure later on. Brendan O' And next next step down it then is the ups system so and so just show you what we're looking at here the diagram that's actually a small little data Center I did that 10 years ago for aircon and that is the ups room. Brendan O' In it so they're they're the ups modules on the left hand side so each one of those was as far as remember was 400 kv 400 kva. Brendan O' So there's four of them there so three plus one one of them would have been a spare so your capacity is three by 400 kva. Brendan O' Which is 1200 kva and then you take power factor everything into account and it gives you about about a megawatt of power when you when you take everything into account. Brendan O' And that's all the switch gear there's quite a lot of switch gear goes into just feeding those four modules feeding the modules taking the power back off them and then sending the power out to the servers. Brendan O' And that's that's the next part of our journey we're going to look at so. Brendan O' In terms ups what do you look at reliability and redundancy is important in all aspects of the data Center but it really comes to comes to the floor in you're looking at the ups system, but you go for. Brendan O' n plus one or two n level of redundancy static systems versus rotary so the system in the top diagram is a static system it's electronic components which convert to. Brendan O' power to back to AC and there's and the batteries which aren't shown in the diagram because they're stored in a separate room. Brendan O' And the battery system is your short term storage in a diesel rotary system which is color drops is that what they're called diesel rotary ups system. Brendan O' that's what you see down the bottom diagram so it's a generator one end diesel generated as just as we saw in the last diagram and then you have a flywheel now i'm not going to get into explain how this works it's very, very complex piece of. Brendan O' machinery, but it's basically a kinetic energy store okay so there's two rotors one rotating 1500 rpm and there's another one running a 3000 rpm. Brendan O' And that store that kinetic energy store it gives you about 10 seconds 15 seconds, maybe of energy storage to allow the diesel engine to start up so normally. Brendan O' That thing is running away sorry the motor generator party should have explained that at the start the motor generator here is running away. Brendan O' i'm driving this flight wheel spinning around and that power has gone into the servers in the normal way when the power fails. Brendan O' The flywheel continues to run it continues to drive the motor generator set which now turns into a generator and the generator then delivers the power. Brendan O' To the to the load, so the flywheel as I say is about 10 seconds of 10 to 15 seconds of energy storage in it. Brendan O' And in the meanwhile, the generator starts up a clutch mechanism closes here and links the diesel set back onto the flywheel back onto the alternator and. Brendan O' Bob's your uncle and that's how that works so they're not very popular, so I wouldn't be worrying too much about them 99% of the data Center market. Brendan O' If not 100% at the moment is served by static and these were popular in the 90s and the 82 thousands were very popular choice, but the technology has really moved on with static system, so they are the main player really. Brendan O' So when it gets to static systems and there's lots of order consistent consideration with the static system there's transformer type systems face transformer less systems. Brendan O' There's modular versus mono block and we will show you a slide not a minute how you operated if you operated online, which means you're you're sending all your power to the director for an inverter. Brendan O' You've certain losses in the system most manufacturers offer what to call a high efficiency or eco mode where you can go into bypass and just feed the load directly don't go to the inverter. Brendan O' Obviously there's a risk associated with that, but the savings and typical. Brendan O' hyper scale data Center like you're talking millions and millions of euros just an energy cost per annum millions and millions of euros just by operating in the eco mode. Brendan O' that's how important this and obviously the associated reduction in carbon emissions there's a lot to be looked at there. Brendan O' And next thing is static and maintenance bypass how you bypass the ups etc and low power factor and harmonics etc battery systems this traditional battery is see lead acid lithium ion is now an option and it's really taken. Brendan O' gaining popularity and the battery analysis and care system switch room. Brendan O' generator compatibility and so on there's a lot to take in and there's I'm just looking at the clock as well of a few slides to go and we're approaching a. Brendan O' question time so i'll just move on anyway with that this diagram I mentioned about resilience and it just shows typically here if you look top left and you have. Brendan O' A load of 200 kva as you see in the Center so the 200 kva represents say a data with a 200 kva load, which is not realistic, it would be 2000 kva maybe. Brendan O' And you decide to use a 100 kva module with an end system you just put into bear two modules okay so you've two modules if one of those fails you've lost the load. Brendan O' So what most data Center solutions, the minimum level of resilience is n plus one. Brendan O' So they meet the need to put in one spare module so in this case you need three 102 of them meet the. Brendan O' need and then you want to spare one so that's where the plus that's what the plus one means. Brendan O' The problem with that arrangement is you still only have a single power path into the load so if this the switch gear associated with this ups system failed. Brendan O' And or the main cable whatever on it, the bus bar system in this board failed, you would lose the entire ups system. Brendan O' So that's why the next one is probably the most popular thing with critical loads and ups systems are often done in a two end configuration, which means. Brendan O' You have an a system and a B system and both systems are capable of meeting your your needs. Brendan O' And a step above that that's using some of the data centers is to n plus one so they have an a system and a B system and in both systems. Brendan O' They have a spare module as well, so we can see starting off with a basic need of just 200 100 kva modules would would meet your 200 kva demand. Brendan O' You end up putting six in so there's a massive amount of additional and plan to put in data centers to meet that this resilience element of the design. Brendan O' And the other thing I suggested is a monoblock versus modular so the monoblock is what you see in the right hand side and it's sometimes called. Brendan O' monolith meaning the same thing really it's a big block ups so that, for example, on the right, could be a 400 kva ups system. Brendan O' The one on the left could also be a 400 kva ups system which is made up of individual modules so you can see there there's. Brendan O' One, two, three, four, five, five modules so each of those modules will be let's say i'll do the other way around let's say it's there 50 kva which is very typical. Brendan O' So that'd be five modules to 250 K at 50 kva giving you a total capacity of 250 and the system on the right is 250 kva. Brendan O' So this system on the left is modular way of doing it now is far better it's much more energy efficient, you can buy the modules and build it up as time goes on. Brendan O' And their event availability time is much higher, which means basically if one of these unit fails you literally just whip it out like any piece of equipment it's plug and play you whip it out and you replace with another one. Brendan O' If if this unit fails, you have to switch the whole unit down to start pulling parts out like getting into. Brendan O' repairing your engine in the car so it's a fairly, fairly big job so the downtime of these can be much, much higher than this system and, as a result, the availability is is lower. Brendan O' And so, for lots of reasons, the industry is moving off to this and this modular design arrangement now. Brendan O' And then, in terms of switch gears to the switch gear for the ups system is typically like this you this system is shown three ups system so it's shown in the Gray boxes so each ups system has an. Brendan O' rectifier shown AC coming in one side DC out the other, then the battery connects into with there. Brendan O' And then the inverter does the opposite director for converts the DC back to AC and sends the onto the load. Brendan O' And do the little box that you show inside the ups there with the two AC lines and if that represents what they call the static bypass switch. Brendan O' So if the inverter if the system gets into trouble or it gets overloaded and the inverter can't handle the load. Brendan O' It transfers over onto this static bypass path here and allows the power to go this way. Brendan O' Okay, so there's a lot more than Apple to end that's that's basically what its function is there that if there's a problem with this power path it transfers over to here. Brendan O' And, and this system here to switch gear is on three separate ups modules and you can see each one has an input breaker on the top. Brendan O' An output breaker and then item 10 has a complete bypass system on it here and then this bus bar down the end that you see. Brendan O' All these lines here they're going out to your not directly to your service that we're going out to this switch gear that in turn feature your service and our server and that's where we're coming to in the next slide. Brendan O' So this is typically I found this slide and I was delighted to find it because it's the exact language I was looking for. Brendan O' As the language I don't like using myself so and but it's what's used here in the industry in Ireland because we're so. Brendan O' influenced by what happens in America, most of the big data center operators here are American, as you know, Microsoft Google Facebook Amazon are all American companies on and many of the. Brendan O' COLO data centers are as well, they're American operators so and you have the ups here and what we're looking at is how to get power from the ups to the server. Brendan O' So you don't just bring big heavy cables you have to go through a series of switch gear systems to get there. Brendan O' So, in America, they talk about PDUs which means power distribution units and RPPs remote power panels, etc. Brendan O' The reason they do that way in America because they have two different voltages that low voltage. Brendan O' They have 482.77 is one voltage system and then they have a lower voltage again for smaller power, which is 208.120. Brendan O' We don't have that in Ireland and and and and really so the PDU which has a transformer in it. Brendan O' we're using that term in Ireland but it's not really a PDU because we don't have a transformer in it so and. Brendan O' Again, I won't get bogged down to that but that's the typical arrangement you come to ups to some sort of a switch gear system or a bus bar system. Brendan O' And then power is delivered to the server right so i'll show you in the in the next slide here typically what happens so in Ireland, the most common solution is to run the power from the ups system. Brendan O' And just to go back show you where we're talking about from the circle breakers here right and nine each one of those would feed a bus bar system. Brendan O' And the bus bar system looks something like this sitting above the server racks here so you'll see again. Brendan O' A and the B system, and you can see the A supply comes in this end you see the box here. Brendan O' The A supply comes in here, but the B supply goes in the opposite end that's again to avoid the cables coming on the same path. Brendan O' To improve resilience and then the racks on the opposite side of the pod, they have an A and a B supply as well, and there's some other you can snide or have a system. Brendan O' power bus by E&I open Donegal really successful company just left me probably saw on the paper there a few weeks ago. Brendan O' And a small got path already started off smaller company 20 years ago, and he sold the veritor for 2 billion. Brendan O' Two billion euros three weeks ago, and I take my hat off to the man, fair play to him starting off from small beginnings and that's just one of the many products that he does. Brendan O' And there's another one, another similar arrangement there, and you can see this is quite a common approach as well. Brendan O' The A supply being in red and the blue supply in in the B supply in blue, red and blue is often used for the A and B supplies. Brendan O' And finally into the back of the server we're nearly here and slightly over time, so the power comes down from the bus bar and it feeds into these. Brendan O' power strips, you can see them here, one is in blue again and one is in red in the back of the server rack okay and they are just like the multi block you have in your desk with a strip of sockets. Brendan O' it's an intelligent version of that Okay, so all the sockets are monitored and everything that power can be measured through them and then you take a red lead and the blue lead up to every server so every single server has a. Brendan O' An A supply and a B supply to it and they can come in single phase or three phase and in 16 amp or 32 amp giving your range or a power range for your cabinet from anything 3.7 kva up to 22 and I have to say 22 kva is getting more and more common 22 kva in Iraq. Brendan O' So basically that's it folks I think, thank you very much.