Critical Power: Electrical systems and data center efficiency

[00:00:00] Speaker 1: Welcome to the Consulting Specifying Engineer Webcast, Critical Power: Electrical Systems and Data Center Efficiency, sponsored by ASCO. I'm your moderator, Amara Rozges, with Consulting Specifying Engineer, Peer Power and CFE Media. Here are some tips for today's webcast. If you're having trouble with your slides or audio, please refresh your browser or click the Refresh Media button directly under the presenter's photo. You can control the volume settings of this webcast by adjusting the volume on your computer speakers. If you're having technical problems, click on the question mark at the top right corner of your screen to bring up a list of system checks to try before contacting an online technician. If you do need a technician, type a message into the "ask a question" box and someone will respond as soon as possible in the "answered questions" box. Type questions for today's speakers in the "ask a question" box on the left side of your screen. The Q&A portion will start after the prepared presentation in about 45 minutes. If you're on Twitter, tweet your questions to us at #CSEDataCenterEfficiency. Today's webcast is being recorded and you will receive an email with the link to the archive in about a week. To download the presentation slides, use the event resources on the left side of your screen. For those of you who are interested in receiving a Learning Unit credit for this event, you'll need to pass a 10-question exam. To take the Learning Unit exam and to download your AIA CES certificate, use the "Learning Unit Exam" tab option at the top of your screen. The exam will open in a new browser window. You can complete the exam after the webcast, however, the link will break when the webcast signs off. The exam will be posted on the Consulting Specifying Engineer webcast with the on-demand version of this event. In keeping with the American Institute of Architects Continuing Education System Policy, please take a few moments to read the Quality Assurance slide. Thank you. Now, please take a moment to review the course description for today's webcast. And here's a list of learning objectives for today's event. We'll cover these in today's presentation. Now, we will hear from our sponsor of this event, ASCO. At the conclusion of the video, you may hear a few seconds of silence to account for varied internet speeds. Please stay on the line as we wait for all participants to rejoin the audio. Thank you. [00:03:33] Speaker 2: Load banks are a vital component for every backup power system because they provide necessary and accurate power loading to ensure the reliability of your power. That's why facility executives and other professionals need the most dependable load bank solutions. The combination of ASCO, Avtron, and Froment connects all the pieces, so your facilities always have power. The combination of unparalleled experience, leading-edge technologies, and the most responsive global support in the industry are complemented by the broadest load bank portfolio that can be integrated with Sigma load bank controls. To learn more and download a load bank white paper, visit our website or call 800-800-ASCO. [00:04:25] Speaker 1: Thank you, ASCO, for your generous support of today's event. Now, let's meet our presenters, Brian Renner and Ken Katsmita. Brian Renner has more than 20 years of experience in management and engineering for new and existing facilities. As an associate at Smith Group JJR in Chicago, he specializes in electrical power, commercial, industrial, government, and mission-critical facilities, with a focus on data centers and laboratory and healthcare buildings. Ken Katsmita is an engineering design principal at Jacobs in Philadelphia. For 20 years, he has been responsible for engineering, designing, and commissioning power distribution systems for mission-critical facilities. Ken's project experience includes data centers, specialized R&D buildings, and large-scale technology facilities containing medium-voltage distribution. Both speakers are members of the Consulting Specifying Engineer Editorial Advisory Board, and it's a pleasure to work with them both today. Brian, take it away. [00:05:44] Speaker 3: Thank you, Amara, and welcome, everyone, to our presentation on electrical systems and data center efficiency. So, data centers are a major energy user in the United States and worldwide. In the U.S., it's estimated that in 2013, they required over 34 power plants, providing over 500 megawatts of each plant, or 91 billion kilowatt hours in 2013. It's estimated by 2020, another 17 power plants and a usage of 139 billion kilowatt hours will be required. In the past, we've used watts per square foot as a metric for how much energy data centers use. It was not unusual to hear 100 watts per square foot used for data centers. But kilowatts per rack is now the most common energy metric that's being used in data centers, with upwards of 30 to 50 kilowatts per rack not uncommon in data centers. So, with all this energy use, cooling is also becoming challenging for these data centers. Water cooling is becoming and making a comeback. It used to be used a lot in supercomputing environments. And really, with all the cell phones and the usage of live streaming videos, we're seeing a great increase in live streaming videos taking up a significant portion of internet traffic during peak times. So, let's talk about what we're going to cover today. And we wish we could get into each one of these in depth, but we will be providing key information. We're going to discuss codes and standards. We're going to talk about reliability classifications and availability. We're going to talk about energy metrics, including PUE, mechanical energy approaches, measurement and verification. We're going to talk about electrical equipment efficiencies, distribution strategies, and even DC power. Some of the codes we will touch on, and I encourage everyone listening to obtain these standards or look them up. We'll be reviewing uptime institutes, tier topologies, or tier standards at least, with regards to availability. We'll touch on TIA 942A, the current telecommunications infrastructure standard for data centers. We'll talk about IEEE standard 3006, on reliability of data centers. We're going to talk about the energy code, such as energy standards, such as ASHRAE 90.1, and a very key draft, 90.4, which is going to really be an interesting standard. We'll talk about, again, ASHRAE on the thermal guidelines, and how that's affecting data centers, and electrical, and energy. And we'll talk about the green grid, with regards to PUE. So let's talk about reliability standards for a little bit. One of the things that is confusing, if you're not familiar, is that reliability is often used interchangeably with terms like availability and redundancy. But there are really key differences, which we're going to touch on a little bit today. With regards to, and I use the term reliability, the most common standard has been tiers. And the uptime has ownership over this tier rating. The ANSI TIA 942 had also used tiers in their standard to discuss availability and redundancy. But they've changed that in version A due to uptime's kind of ownership of the word term or tier. And so when we talk about tiers and reliability, or really actually availability, the uptime institute has four levels of tier. Tier 1 being the least, providing the least availability and redundancy up to tier 4 with the highest level of redundancy. And within those tiers, there's a general discussion of the levels of redundancy, using N, N+1, and 2N, the number of distribution paths as we get down to the data equipment racks, either an A, an A or B, or an A+B path. Concurrent maintainability, the uptime institute talks about, particularly in Tier 3 and Tier 4, the ability to take down components, say generator or a UPS, and being able to maintain that piece of equipment without interrupting or disturbing the power to the equipment. It also gets into fault tolerance, compartmentalization, and cooling requirements. What I want to stress, and we're going to talk a little bit about IEEE 3000.6, is that, really, these tier levels by uptime have certain limitations. One of them is that they're really talking about redundancy and availability and not necessary reliability. But if we use the term "reliability" generically for tiers, also you have to understand that this affects energy efficiency because of equipment operating at part load is not as efficient as equipment working on full load. And when we talked about ASHRAE 90.4, the proposed draft, they actually take into account reliability or availability and levels of redundancy in their energy-efficient requirements. So, with regards to the actual technical term of reliability, IEEE standard 3006.7 covers actual calculations and determinations for reliability. The IEEE standard 3006.7 used to be a chapter in the IEEE gold book. The IEEE gold book has been around for years and covered the reliability data obtained from various government agencies with regards to failure rates of equipment and hours of downtime. And it includes both, there was a chapter that was introduced in the last version of the gold book on mission critical or 24x7 facilities. That's been broken off, those chapters have been broken up in the gold book and now are individual standards. So, the chapter on mission critical is now the 3,000.6, but that book had a lot of good data on switchgear, on generators, reliability of different types of wiring and busways. And it really has some good information and I encourage everyone to seek out the latest chapters which have been broken into various standards. But the thing to understand is that there's difference between availability and reliability. When you see the nines, five nines, four nines, what that's really referring to is availability. It does not count the number of failures, so a five nine or four nine facility could have one long outage or it could have a series of 10 minute outages. You don't know. So, reliability really is how long something is going to function before you have an outage of any type of disruption to the continuous operation of a system. So, IEEE 3006.7 presents a lot of detailed calculations on how to actually determine reliability of both electrical and mechanical systems. So, moving on from reliability and we bring this up just so you understand the basic terms and as they apply to energy, let's talk about the energy standards. Probably you've heard of ASHRAE 90.1 Beyond data centers, ASHRAE 90.1 has been used to benchmark and to do compliance methods for energy for a wide variety of facilities. But within, in the 2013 version of ASHRAE 90.1, they actually introduced data centers and they started talking about PUE and we'll talk about PUE in a minute as an energy metric for data centers. And they gave PUE, minimum PUE to be achieved in data centers by climate zone. Typically, those PUEs, and again we'll talk about PUE, would vary between 1.3 and 1.6 as a minimum standard for you to achieve. The thing you need to know about 90.1 2013 is it's only been adopted in a few states. It's listed here as two, but there's actually several states there in green and you can go to the DOE, you can Google this to see which states have adopted which versions of ASHRAE. So, this is the one standard that I think is going to really be a game changer and is probably the most important standard that will be produced shortly with regards to data centers and energy. And that's ASHRAE standard 90.4 P. It's, it's, it's had a second draft actually just recently in, I believe January, the third draft has been out for review and you can Google this and achieve or get a copy of this draft of the standard. This is really going to be a game changer. This is going to still work with 90.1 but it's going to bring a whole new set of energy metrics in beyond PUE. It's going to bring in mechanical loss components. It's going to bring in electrical loss components and it's going to break those electrical distribution segments into incoming service, UPS segments, ITE or data equipment and it's going to start to put compliance methods in which specifically require mechanical components and electrical components to meet certain efficiency standards. So, we're going to get away from the PUE which again I'll talk about in a little bit here but this is also going to apply to both new data centers and expanded and renovated data centers. It's, it's, it's really, I encourage everyone to get the, get a copy of this and take a look at it but when this comes out this is really going to change how we're measuring energy efficiency of data centers. But let's talk a little bit about PUE because until ASHRAE 90.4 comes out, PUE is still being used in the, in industry a lot. It was developed by the green grid. It is a useful metric. It's most referred to metric in data center efficiency and power utilization effectiveness or PUE is defined as the ratio of the total facility energy. that is the energy that is the energy that's going in let's assume it's just in a data center itself and doesn't have an auxiliary component such as offices and storage. It's the total energy going into that facility for mechanical, electrical, IT divided by just the equipment or just the energy being used for the IT equipment. And the PUE values for data centers can, can vary substantially from 1.06 up to 2.0 or higher for older data centers. Generally a PUE of about 1.4, 1.6 and below is considered more efficient. So the lower the PUE the more efficient, the more energy is going just to the IT equipment versus the, versus being wasted in the electrical and mechanical systems. And we'll talk about those efficiencies and how to achieve efficiencies in the mechanical and electrical so more energy goes to the IT equipment which is the goal of improving PUE. But there are limitations to PUE. First of all, PUE varies by location. So you have to understand that different areas of the country you can take advantage of, and we're going to talk about mechanical, they call it free cooling, economization. It's a ratio, it's not the total energy. So a facility with different PUEs, even though you might say it has a better PUE or better ratio or efficiency, it might actually use more energy. And some of this comes into the fact that the IT equipment might be inefficient. You could raise the temperature in a data center by lowering the amount of air conditioning you're using, and the fans on the IT equipment actually step up to pick up that air. So you're shifting energy to less efficient fans on the actual servers themselves, which results, may result in a higher total energy usage, but your PUE improves because you're shifting everything to the IT equipment. So there's some issues out there. There's seasonal variations, PUE changes. Part load also comes into PUE. You have to understand a lot of data centers don't come on full load. So your PUE can be a lot worse when it's operating at part load. And then the other thing about PUE is you have to make sure that you're monitoring this and measuring it properly, and we'll talk about that later on with the EPMS system slides. A little bit more about PUE. We're going to show you, there's a diagram here showing you usage in a data center that has a PUE of 2.0. And you've got your IT equipment, you've got your electrical and your mechanical. This is kind of an old school wheel. Your mechanical systems could include in these older type systems chillers, crack units, air handlers. Your electrical loads would be things like switchgear, transformers, PDU, UPSs, lighting, and this is the energy that's going to those components. And sometimes it's waste energy as, for instance, in transformers. So I want to very briefly talk to you about mechanical efficiencies. And we'll just step through this. It's just you understand what our mechanical counterparts are doing. There's a number of ways that they're improving their efficiency. Includes raising the temperature, aisle containment, economizer or free cooling. And we're also going to talk a little bit about part load efficiency. Again, I'm just going to step through these quickly. One of the current methods is to raise the temperature in the data center. In days gone past, data centers typically operated at 68 to 72, known as a set point. These are quite cold. If you've been in the older data centers, you walk in, it's very cool in those data centers. ASHRAE now has recommendations under that standard TC 9.9 to raise those up to as high as 80 degrees Fahrenheit or higher. There's a 4% energy savings for each degree upward. But you have to be careful in your analysis as you raise it too high. It may not necessarily be the case. So data centers are exploring set points that are moving towards 80 degrees Fahrenheit in the data center, making it much warmer. And this also, as you do the inlet temperature higher, the discharge on the other side of the servers or data equipment also is getting hotter. So your hot aisles are getting hotter. So one of the ways that they're controlling this is with containment. It could be hot aisle, it could be cold aisle, but typically hot aisle containment. They're putting actually constructing enclosures on the discharge backside of these racks that collects the heat and channels it away to be cooled rather than cooling the whole space. You can also use this kind of containment for UPSs that use batteries. We know that batteries are very sensitive to temperature fluctuations, but the UPSs themselves, the electronics, can operate at a higher temperature. So you might use the batteries in a contained area or room to keep those at a proper temperature for long life. And the UPS itself might be allowed to be in an area that can go hotter. One of the things you need to be aware of, though, is when you get into these hot aisle containments, it gets very hot in those enclosures. So if you've got lighting, fire alarm, and other devices inside those hot aisle containment, you could be exposing them to temperatures that are beyond the normal specifications of typical electrical equipment. Free cooling. Basically, our mechanical counterparts look at weather data, meteorological data for a given area where a data center is going to be used over a one year period. And they use all that temperature and humidity data to basically collect and determine the best type of cooling using ASHRAE's guidelines for temperature and humidity. The other thing, we talked about part load efficiency. This is, you need to work with your mechanical very closely. Look at what they're doing in terms of part load efficiency, what they're using to control demands, and make sure that they're right sizing and scaling the mechanical systems properly. Because, again, that's a key part to efficiencies is making sure you've got the part load and which mechanical systems are operating at which time. So I'd like to turn this over to talk more about the electrical efficiency that it can. [00:23:29] Speaker 4: Thanks, Brian. As Brian was talking about, one of the key metrics in measuring efficiency in data centers is PUE, Power Utilization Effectiveness. And it's basically the total power over the IT of power equipment. And when people talk or look to lower their PUE, they generally tend to look towards the cooling side of it. What consumes energy, the cooling side. But the electrical system also consumes energy and it consumes it in the form of losses. Those losses are basically due to inefficiencies in the equipment. On the distribution system, it's the heat given off by the equipment. That's kind of what I want to focus on a little bit is that power side of it. So, again, when you look at the data center energy consumption, in this case, unlike the one Brian said, this is a little more, you know, this is a data center. It's got a 1.3 PUE. They've utilized all those things Brian talked about in reducing the mechanical system, containment, increasing the heat. And they've lowered that mechanical losses down so it has a better ratio. But when you look at it, there's still a big piece, which is electrical distribution losses, about 12%. And what does that really mean? Well, if you look at a 2,000 KW data center with an annual cost of 10 cents a kilowatt hour, that comes out to about $280,000 a year just spent on wasted electrical energy. So, there's just a significant amount of money and that's kind of what we want to focus on. So, when you look at the legacy electrical infrastructure, you have the utility or the generator coming in. It goes through a medium voltage transformer, switch gear UPS, goes into the data center as a PDU, IT power supply, and then to the server. So, the four key components I think have about the highest loss is you have your medium voltage to low voltage transformer. So, you've got no load and core losses. You have a UPS, which has rectifier and inverter losses, may also have a transformer. The PDU, again, another transformer with no load and core losses. And then you're looking at your power supplies, which, again, is a rectifier and some transformation. So, those are the four key pieces that have the most, you know, the highest losses. And this is kind of just an example by showing you if you increase just each one of those by a percent or two, what that means as far as energy savings. So, increasing the transformer by a percent, the UPS by a couple percentage, you can quickly add up to about $117,000. And that's on that two megawatt data center. So, just by doing a little bit can add up to substantial savings. The first thing we want to look at is the transformer. Now, in 2005, the NEMA TP1 guideline came out. It was basically a guide for determining efficiencies in distribution transformers. And, basically, what this did was set the minimum requirements. So, a legacy transform, say it was 150 kVA, was probably around 97%. With the TP1 efficiency, that transformer had to be 98.3 or higher. So, again, it's the minimum requirements. You can go out and get much higher efficiency transformers. Like an ultra-high efficiency transformer can get you up into the 99% range. Now, in 2016, this year, there actually is an amendment to that to increase that minimum rating up even higher. So, a 98.5 would go to a 98.9. Your medium voltage transformers, which are around 99%, would be at 99.3. So, again, they're pushing that efficiency up in transformers. The next thing we're going to look at is UPS efficiencies. So, this is kind of a standard conventional double conversion static UPS. Power comes into the rectifier, goes AC to DC, charges the battery, comes out through the inverter, converts it back from DC to AC. And during those conversions, you get losses in the system. This is a typical curve of what you would see for a double conversion static UPS efficiency. You can see if you're in the 90 to 100% range of load, you're operating about 94% efficient. If you're operating as you go lower on the curve towards the 25% range, you see it starts to quickly drop off. You're around 87 and quickly drops down as lower you get. So, again, as Brian was talking about, you really want to operate these UPSs in the higher range so you're running at more efficiency. Some of the factors when you're looking at UPS efficiency, one, transformer. If you go with a transformer UPS, transformer-less UPS, it'll be a little more efficient. You lose those losses. Another thing, as Brian had mentioned, is the reliability of the system. When you have an N-UPS or a single UPS, you tend to operate it probably in the 60-70% range, so higher on that curve. When you have an N+1 system, maybe you have a catcher UPS. That catcher sits mostly unloaded for most of the time, so that's going to affect your efficiency. Or a 2N, you know, fully redundant system where neither system is ever above 50%, so you're down in the 30-40% range, which is lower on that curve. So that can have a major impact on your efficiency. So what you really want to do is kind of right-size that UPS. Again, you don't really want to overbuild if your load isn't there yet. Kind of grow UPS as the building grows. And you see that a lot now with UPS manufacturers trying to create modular UPS, where a single 1,100 UPS module might have four 275 KVA UPSs built within it. And what that allows them to do is that if they're running a UPS maybe at a lower load, maybe you're running at 20%, they can actually put two of those modules in standby and run the other two at a much higher efficiency. So it improves your efficiency on the total unit. Some other things to look at, other UPS technologies, rotary systems, flywheels. They still have the AC to DC component, which is some losses. So they tend to run a little higher on the efficiency. Once that big rotating mass gets moving, it takes a lot less energy to keep it moving. So you'll see a slight increase in efficiency. It's not a big increase, but there is a slight increase that you can look at. The next piece you'll hear a lot about is eco mode or economy mode in the UPS. The traditional eco mode, they basically took the power and ran it through the bypass. So it bypasses the inverter and the rectifier. So you don't have those losses as it goes through the rectifier and inverter. You see that usually about 99% efficient because you are going in bypass. But there are some negative effects that people are a little hesitant. It gets to that resiliency versus efficiency or robustness. One of the things is you are running basically unconditioned. So any fluctuations in voltage or frequency are going to be seen by the load. Another one is transfer time. So the time it requires the UPS to first detect that we've lost power. Then it needs to energize the inverter. And then it needs to transfer to battery. So even though that occurs within the ITIC curve and you won't lose power to the server, it can have effect on things that are downstream. Sometimes you might see a transformer go into saturation and they have a huge inrush. You could see static transfer switches if they're in the system transfer because it's all an outage. So you just need to be careful. It can affect downstream stuff. Another thing is thermal shock. UPS is called upon when you lose power. So that's when you want it to operate. So at that moment, having it in eco mode, you're basically energizing the inverter, transferring 100% of the load, so a big step onto the inverter, all at the time you're trying to protect yourself from an outage. So there is a little bit of a thermal shock that could affect the system. Another thing is harmonics. The rectifier inverter does create sort of a filter from any harmonics from downstream. So when you run it in bypass, that's going to go straight to the utility or generator. And then the last one is fault discrimination. It's basically that UPS looks to put the system in bypass when there's a fault condition. If you're operating in bypass, the UPS has a hard time sometimes determining is it a fault, is it a loss of power, and it might actually transfer it back to the inverter when it is a fault, and then transfer it back to the static bypass. So it will extend that out at that arc or the damaged downstream equipment. So it's something you need to look at. So what manufacturers were starting to do to try and get away from some of those challenges, they went to what they call an advanced or a high efficiency eco mode. And basically what that is is they keep the inverter engaged. So the power still throws through the bypass, but the inverter is actually energized. And what that does is it helps cut down on the transfer time. So it seems more like it's conditioned power. Any fluctuation in voltage will actually go through the inverter. It's a seamless transfer. And that also helps with the harmonics. So they're starting to improve that eco mode. But there's still some things you need to look at if you want to implement it. And because you do go through the inverter, you will see a little bit efficiency loss. So it's really about 98%. This was a case we were actually commissioning a data center, and we had the opportunity to put the UPSs through different modes of operation. And it kind of just goes through different PUE levels that occur. If you go through both A and B systems and double conversion, if I do double conversion on one and kind of use that multi-mode on the other where I'm shutting units off, it just gives you, as you go through, you can see as you then go down to both eco mode on both sides, you're down to about a 1.3 from 1.56. So this was basically a six-module system. It was scalable. It was in a 2N arrangement. And we had it about 50% load. So it just kind of gives you an idea of what you can expect from different modes of operation. The IT power supply is the next piece. It, again, has some losses. There's a rectifier from AC to DC, some transformation. You usually see about a 92% to 90% efficiency, depending on whether it's a 380 volt DC, 240 volt AC. So as you go through, you'll start to see an increase. But they're all around 92 to 93, if not a large percentage of different IT power supplies. The one thing that is interesting on IT power supplies is if you look at the PUE, the IT power supply is actually in the denominator. So one of the things we say, if you really want to decrease your PUE, just go out and get inefficient servers and that will help you with your PUE. Obviously it won't help you overall, but it's a way to, it just kind of shows you that the PUE is a matrix, but you can play around with it to make it to your advantage. So when you're looking at that, obviously selecting equipment, efficient equipment, or replacing legacy equipment will be more efficient. It can help you reduce electrical losses and save money. But what if we eliminated that inefficient equipment altogether? What if we just got rid of it? And that's kind of the idea behind the higher voltage distribution. When you look at the legacy system, you have a PDU to transfer from 480 to 208. So there's losses involved with that PDU. So what if you could just go from a higher voltage GPS straight to the IT power supply? And that's kind of what's behind the higher voltage AC architecture. You're eliminating that PDU. So why 415 volts? If you look at North America, you have 208 phase-to-phase, 120 phase-to-ground. In Europe, it's 415 phase-to-phase, 240 phase-to-ground. Now when you look at the IT power supplies, they're actually single-phase devices that can accommodate a range anywhere from 100 to 240. So they were basically built to work in the United States and Europe together. They didn't want to make separate servers. So the idea is why not push that IT power supply up to the higher side, up to the 240, and use an already established European distribution system to gain efficiencies. So some of the advantages of a 415 volt distribution. One, you're going to gain efficiency. You're looking at about a 5 to 7% reduction. It's because you eliminated PDU losses. The IT power supplies are more efficient, and you also reduce the cooling system. You don't have to cool the PDUs. Another advantage, you're much smaller conductors. So for that same 30 amp circuit where you used to get 3.6 kilowatts, I can get 7.2 kilowatts out of that same circuit at 240, so a lot less copper. You gain white space. You pick up two cabinets. The next is reduced maintenance costs. And then one big advantage is the fact that the 240 system is already there. There's already distribution equipment. Plug receptacles are all available. You're not introducing anything new that's not already standard. The next type system is a DC system. Basically the same thing when you're looking at the legacy architecture. You've got a rectifier AC to DC. Inverter DC to AC. You've got a transformer into PDU AC to AC. So you've got a lot of transformation and each time that creates losses and you lose efficiency. So the idea behind the DC is why don't we just go from DC straight from the rectifier straight to the DC power supply. And basically eliminate the double conversion EPS. Eliminate the PDU. And distribute straight DC to the server cabinet. And I know a lot of people in the industry will say well nobody uses DC. It's not a common practice. But if you look in the industry, it actually has been used for a while. Telecom industry uses DC power. Traction power, light rail in the cities all use DC power. If you look at any of your electronics in your house, the power cord has a DC, AC to DC converter in it. Batteries are all DC powered. And one of the big things I think that's happening now is both Etsy and e-emergence have standardized on 380 volt DC. Where you used to see a lot of people using different ranges of DC. They now have guidelines, standards all based around 380 volt DC. So manufacturers are starting to produce stuff at 380 volt DC. And it's starting to become a little more popular. Some of the advantages of DC power. Again, you're going to get some energy efficiency. About 8 to 10% reduction. You're going to eliminate the UPS losses, the PDU losses. Again, IT has become more efficient. You're losing the cooling system. Some of the other advantages. There's just less components. So things are more reliable. Fewer pieces to fail. Smaller footprints. Another big advantage of DC is the scalability of DC. Since you're not using a sine wave and it's more of a flat line, you can basically plug and play DC. You don't have to synchronize. So the scalability factor of DC becomes very important. And another thing is just green power to integrate with renewable energy sources. All your photovoltaics, your fuel cells, wind power are all DC powered. So you can plug them into a system pretty regularly. The one thing you have to watch on the 4, I forgot to mention, on the 415, one of the negative things you really have to watch is the fall current. You've taken away that transformer, which uses the fall currents. You want to keep them as smaller systems. So you just really need to keep an eye and make sure you're using the correct rated equipment. With that, I'm going to turn it back over to Brian to talk about power measurements and how we measure efficiency. Thanks, Ken. [00:39:05] Speaker 3: So, how do we tie this all together? How do we measure and verify the efficiency of a data center? Really, this is where the electrical power management system, or EPMS as it's sometimes called, in data centers. There are a lot of reasons to install an EPMS system. I mean, we're going to talk, obviously we're talking about energy here. You want to verify and track the PUE. But there are a lot of other reasons for having it. You want to decrease downtime events. And you decrease downtime events by checking the loading of various systems. If there's power quality or fault issues, depending on the type of metering you've got installed, it can bring you information about what's happening in various components in your system. And the other reason, too, is that whether you're in a co-location facility or another facility that might have mixed departments or mixed users within the same company or organization, there's a big push to start billing those users individually for the energy that they're using in their individual racks. So, what does an EPMS system look like? Basically, you're taking your one-line system and you're deciding what type of meters and where to put them in. Now, the Green Grid, which is the organization that puts together the PUE standard, talks about the levels or, let's say, level of metering that you would want to put in for PUE. And they talk about level one being basic and only having a few meters, one at the facility, another one at the discharge of the UPS. So, you can calculate the IT, you can subtract the IT energy away from the total facility to calculate your PUE. Level two is intermediate. You might get it down at both the IT and just at the PDU, if you have PDU's distribution and at the utility. Level three might be at various levels. It also talks about the levels of interval of metering. PUE, obviously, should be done on an annual basis. That's the most accurate way to do PUE. You don't want to do instantaneous or track your PUE on a daily or monthly basis because there are seasonal variations and annual. So, on the basic level, you might only have meters that take that information monthly. But if you get to more advanced, it might be 15 minutes or less. Now, these are just for the Green Grid's PUE. There's all sorts of reasons why you want to have more sophisticated metering, continuous metering throughout the various levels in your data center. But you want to, the metering is really going all the way down to the PDU strip, the power strips that are going inside the individual cabinets. In that case, it might have a meter and you just need to interface that metering unit into your computer system. Same with some of the generators and UPSs have very sophisticated metering within those pieces of equipment and you just need to interface those. In other cases, you might want to put some very high-level power quality metering at the mains of your equipment, your main gear to get waveform capture, see what the utility is doing in terms of power quality to the site. That's always a big issue is how reliable and clean the utility power is. So there's a lot of reasons why you want to have the power measurement system that exceeds even just the Green Grid. So I think what we've done here is to highlight, and one of the key things you got out of that PUE diagram between 2.0 and 1.3 that Ken and I put together, is really the mechanical guys have gotten very efficient. They've really been driving this efficiency standard, but as the mechanical gets more and more efficient, the opportunities for efficiency start shifting more and more to electrical. So there's going to be some great ideas here for you to start improving the electrical efficiency in the data centers, how to track it, and what standards are going to be used to verify and comply with that. So with that, I'll turn it over to Amara. [00:43:20] Speaker 1: All right. Excellent. Thank you both so much. Here is a list of resources and references from the content that was covered during the presentation. You can link to this information when you download the PDF of the presentation under the Event Resources tab. And thank you, Brian, and thank you, Ken, for that first-rate presentation. Now our presenters will answer questions about this topic. Type your question in the Ask a Question box on the screen and indicate which speaker you would like to answer your question by typing his name before the question. If you're on Twitter, please tweet your question to #CSEDataCenterEfficiency. We'll get to as many questions as time allows. Additional information will be posted online at www.csemag.com with the archived version of this webcast. Don't forget to take the Learning Unit exam and to download your AIA CES Learning Unit Certificate. Use the Learning Unit Exam tab option at the top of your screen. The exam will open in a new browser window. You can complete the exam after the webcast. However, the link will break when the webcast signs off. The exam will be posted on the CSE website at www.csemag.com with the on-demand version of this event. So now we're ready to take some questions from the audience. Brian, we're going to kick this off with you. Brian, please define some redundancy design trends. Are data centers opting for renewable energy supplies? Can you talk about that a little bit? [00:45:12] Speaker 3: Sure. There are two parts of this question. With regards to redundancy availability and the uptime tier level, it's really industry dependent. If you're in the financial world, the banking world, the insurance world, you're still seeing tier 3, tier 4, very redundant, reliable data centers. In some of the enterprise, particularly in Internet-based companies, you're seeing a change from the multiple redundant large data centers to smaller efficient data centers located throughout the country or world where they can just switch over in an instant because they don't have the same latency issues that the financial or banking industries might have. If you're doing an Internet search, you're not going to notice a microsecond switch over to another facility. So in those instances, those data centers are getting very efficient because they don't have a lot of redundant components. And they can switch to a smaller data center. And they can also track or follow energy rates across the world in terms of nighttime and daytime. In terms of the renewable energy, it's not necessarily affecting efficiency or redundancy, but what you're seeing is a lot of the major corporate players investing in sustainable and renewable energy just as part of a good sustainability plan. So some of the large companies, Apple, Google, you see them investing in wind and solar farms to put back in the energy that they're taking. So yeah, good question. [00:46:51] Speaker 1: Okay, excellent answer. Thank you. And Brian, one more question for you. This is a really good one. Is there a better metric than PUE, power usage effectiveness? [00:47:02] Speaker 3: Yes. We talked a little bit about the limitations of PUE and Ken brought it up as well about how it's possible to shift energy hogs to the IT equipment and IT equipment efficiencies have not been part of the PUE equation. So we're seeing efficiency being applied. So we're seeing efficiency being applied and I talked about 90.4 ASHRAE proposed pushing the metrics and the evaluation down to individual components within the electrical and mechanical systems and really getting those as efficiency if possible. You're also looking at a renewed focus on the IT equipment themselves, which is not part of the standard but is part of other organizations and standards to make those more efficient. So electrical equipment losses, mechanical equipment losses that are identified in ASHRAE 90.4 are good metrics, but there are lots of other metrics as well. And let's face it, in the end of it, it's the energy bill you're getting for the facility. We can talk about ratios and efficiencies, but if you're measuring the energy with an EPMS or you're getting your bill every month, you're knowing what you're paying for energy. Yeah, and I agree with Brian. [00:48:07] Speaker 4: I mean, PUE kind of becomes, you know, it's what you put in it is what you're going to get out of it. You'll see a lot of times people have a 1.1 PUE and in parentheses they'll say it's a mechanical PUE where they just completely ignore the electrical inefficiencies. So it's something you really do have to be careful about how you're doing it. And again, at the end of the day, it's for your facility to see how you can better your facility and how you can increase, you know, reduce your rates. So I think that's what it comes down to. [00:48:36] Speaker 1: Okay, yeah, good point, Ken. Thank you for that input. And Ken, I'm going to send the next question over to you. Will all components required for uninterruptible power supply systems be considered? So for example, UPS efficiency figures are quoted in terms of the inverter efficiency for static systems, but nothing is considered for the batteries and battery room or the flywheels if used. Can you talk about that a little bit? [00:49:04] Speaker 4: Yeah, I think, you know, when you look at the UPS, the inverter and rectifier, the transformation through there is probably the biggest losses. So that's why they kind of pinpoint that as your efficiency. And that's why the Eco mode when you're trying to bypass that gives you such a better efficiency. You do when you have batteries versus a flywheel. Obviously, you know, the flywheel, once that big mass gets going, it takes a lot less energy. So you get a little bit more efficiency. But again, when you look at the overall losses, that is small in comparison to the inverter rectifier piece. The other efficiencies you'll see is more or less the flywheel has less heat. You know, there's other types of redundancy. You know, you can, the flywheel doesn't have to be 74 degrees like the battery system does. You can have it at a higher temperature, so you gain efficiency there. It's different types of efficiencies. You won't see it strictly as losses, but it kind of, you know, comes all together when you look at it as a whole. So, yes, there are different pieces of efficiency you have to look at there. [00:50:04] Speaker 3: Yeah, we're seeing a lot of clients also bringing up the sustainability component of batteries. Batteries can be a fairly dirty component to have to dispose of and in their manufacturing. So we're seeing a lot of requests to try and get as batteries in their maintenance headache to try and get them out as much as possible. Yep, I agree. [00:50:27] Speaker 1: Okay, great. Ken, we're going to stick with you. Next question. Robustness versus efficiency of data center power distribution. Can you elaborate on that, please? [00:50:39] Speaker 4: Yeah, I mean, I guess they're kind of two different items, but, you know, they both kind of play together. When you look at, you know, what Brian and I had talked about, when you look at robustness or reliability, you put in dual systems, now those systems are operating, you know, lower on the efficiency curve because they're not, they're less loaded. So you need to look at that. There's also, again, a lot of people look, oh, I want to use eco mode because it's more efficient, but yet it kind of cuts down on what you would call robustness of the system. You're putting raw power into the servers versus condition power. There's other things that can go wrong. So you really want to, you know, just because you're making it more efficient, you don't want to hurt yourself on the reliability because the whole idea of the UPS is to protect yourself. So you really do want to be careful as far as, you know, how you build the system, try to build it more modular, try to keep the efficiencies by, you know, operating the equipment where it wants to operate in that sort of system. I think I hope that answered the question. [00:51:39] Speaker 1: Yeah, you got it. Thanks so much, Ken. All right, Brian, next one's for you. Life safety versus UPS STS optional standby power. Can you explain this in more detail? [00:51:55] Speaker 3: Sure. It's a good question. And I've spoken on this before through CSE. We tend to use the generic term emergency, emergency generator, emergency system, emergency UPS, simply because it's a broad-based term. But really under the codes, particularly the NEC, there is a significant difference between emergency life safety systems and optional standby. In most cases, the generators and UPS that we're using in data centers are standby or even optional standby, actually, instead of legally required standby. They're optional systems. So it's important when you're dealing with your code officials to use the right terms. And we all use the word emergency generator, but really in most of the cases, data centers are not having significant life safety systems other than lighting, fire alarm, and other systems, which can be either protected with their own integral inverter or battery systems rather than being put on the UPS or the generator. But if they are put on the UPS or the generator, you do have to follow code. You do have to separate out that distribution separately from your data center distribution. Okay. Excellent. [00:53:13] Speaker 1: Ken, the next one is for you. What is the difference between a network transformer and a distribution transformer? Are they used interchangeably? Network transformer, usually when you see, like, transformers that are set up to back each other, like a lot of times, utilities will have multiple sets of network transformers that allow, you know, if a loss of one transformer picks up another, they're kind of a transformer that's, I hate to use the same word, but networked together, where distribution is more straightforward. [00:53:39] Speaker 4: They're individual straightforward, in and out. The transformer itself is interchangeable, it's just the way it's configured into the system, I believe, is the difference. [00:54:05] Speaker 1: Very good. Thank you. Brian, I'm going to start this question with you, and then, Ken, if you have information to kick in, that would be terrific. This person's asking, with a 415 volt UPS and distribution system, how do you handle available fault current? Brian, can you kick that one off? Sure. [00:54:26] Speaker 3: This is a great question, and one that came up when we started using 415 more in data centers. You are seeing higher levels of fault current at the 415, typically, I say typically, but we're using overhead busways as a distribution means above the racks at 415. And not only do you have higher instances of fault currents at these, because you're going direct from a 415 volt substation or switchgear with larger available faults, you've kind of got a direct path and you're using a large busway, but you're also running into issues where we have in the past of the PDU strips, the connections at the individual racks, and whether that equipment can handle the higher fault levels, and trying to make sure that that's properly coordinated. Because sometimes we don't even get involved as the electrical engineer into the specification of the racks and PDUs, so it's important to advise our clients on what they might have to interface with. [00:55:31] Speaker 4: Yeah, and I agree with Brian. You do see that a lot. It's even hard to find a rating on the PDU strip, so that's something you definitely have to be careful of. One of the things we try to do is instead of the big multi-module huge systems, you try to look at smaller systems, smaller module systems, so smaller transformers, utilize the fan rating if you can for emergency modes, try to keep that on a multi-module system. The problem is the fault current, when there's a fault, it goes to bypass. You have a 4000 amp bypass, which is not giving you any impedance to cut the fault current down. So if you can keep the UPS modules into smaller systems, smaller individual systems, you can really cut that fault current down, so it's something you do want to be aware of. And again, just make sure the ratings of the equipment that you're using meet the fault current, because it will have a higher fault current. [00:56:21] Speaker 1: All right, thank you both for your input on that. If you do have a question for Ken Kutsmita or Brian Renner, please type your question into the Ask a Question box on your screen. Questions that we don't get to today will be answered online, and you'll receive information about when that information and the details are available, along with the archived version of this webcast. All right, Ken, another question for you. Are there different personnel safety concerns with DC power as opposed to AC power? [00:56:58] Speaker 4: Yeah, no, that's a good question too. I mean, there are specific grounding requirements you'll have to use. So, you know, check IEEE when you use a DC power system. I think the biggest concern, I used to work for the railroad a long time ago, and the biggest concern was always DC power would grab hold of you and won't let go. So there's no zero crossing in the wave with a DC curve. So I think that's the biggest concern that people must be aware of. It's just a little bit different, you know, power. So having people that are qualified to work on it does become an issue, because there's not many electricians out there that have worked with DC power. So that is a concern. That's probably the biggest one that I'm aware of. There is now fault current, you know, arc flash for DC too. So that's something to be aware of. It's a little different calculation, but it is available now and it needs to be done. So, yeah, there definitely is some safety concern that you have to be aware of. [00:57:57] Speaker 1: Okay. Good to know. All right. We have time for one last question. And this is kind of a wrap-up question. If you could both give just a couple of key points to wrap this one up. Brian, I'll start with you. What's a low-hanging fruit opportunity to make the data center more efficient? [00:58:16] Speaker 3: Well, I think Ken brought up some great ideas and maybe I'll just point out one and he can jump in with one or two other. But I think in my standpoint, the opportunity to go to 415 to get rid of transformation in the data center has been one of the better approaches just to get rid of those losses. [00:58:39] Speaker 4: Yeah, I agree. I think, you know, because the 415, the system, there's already power supplies, receptacles, panels that are already rated for that. It just makes perfect sense. The other, just simply transformers, you can get much more efficient transformers. They're generally just replacing an existing transformer with, you know, if you're in a legacy system or if you're trying to go to a new system, just look towards more higher efficient transformers. I think that's a big plus. You know, when I was talking about eco mode, what we see, you know, a lot of people are afraid of the eco mode, which I understand because of the negatives. What we do see of people that have a 2N system or maybe an N+1 catcher type system, they put the catcher in an eco mode because it's basically sitting there unloaded. There may be one half of a 2N in eco mode where they don't put both halves in so they're at least protected from one source. So you just have to be careful what you want to do and how you want to do it, but it's definitely there. Great. [00:59:32] Speaker 1: Thank you. Well, we will close today by thanking our great speakers, Brian Renner and Ken Kutsmita, for kindly sharing their time and knowledge. And I'd also like to extend a special thank you to our sponsor, ASCO, for supporting today's webcast. And now that we are just about done, we want your feedback. A short survey will pop up on your screen as soon as this webcast event ends. questions.