Webinar Recording: Next Generations – Data Center Cooling Technologies

[00:00:00] Farad Musa: hello everyone again thank you for joining us with pyramids chapter i'm farad musa the chapter technology transfer committee chair for pyramids chapter of course i'd like to welcome dr omar abdelaziz first and thank him for giving us this opportunity to learn from his valuable experience and also i would like to welcome all the co-hosting chapters alexandria falcon jordan lebanese and saudi arabia they're all co-hosting chapters with us today but dr omar if you would allow me before we start i would like to talk briefly about what usher does for whoever doesn't know just a very brief introduction ashi stands for the american society of heating refrigerating and air conditioning engineers it's an american professional association seeking to advance hvac and refrigeration systems design and construction it has over than 50 000 members in more than 130 count countries worldwide from different fields as building service engineers architects mechanical contractors building owners manufacturers and others that are concerned with the hvac and and refrigerants industry uh actually also is a non-profitable organization uh in which most of the people who are working on it are volunteers uh it's well known for publishing hvac technical standards as most people know but it also funds research projects offers continuing education programs and aims to improve building services engineering energy efficiency indoor air quality and sustainable development and uh the past few years sustainability has been the theme for every ashley president uh and i will share with you in the chat box the links for the the ashley website for whoever wants to learn more about it and also the we have the ashley awards and ashley certificates that uh that you might want to learn about it as well because they are very beneficial i will turn uh their links as well and the membership link as well for if you want to know about the benefits of the membership uh and please uh like and follow our page on facebook to be updated with our coming programs and events [00:02:30] Omar Abdelaziz: uh before i start uh i would like to thank uh the uh hosting chapters uh for inviting me to speak on this interesting topic of the next generation data center cooling technologies this is a very timely uh topic uh i also would like to thank all the audience who are here today coming from different parts of the world uh i see many arabic speaking uh people but i also see many uh non-arabic native speakers so i will uh give my talk today in english however if you would like you can uh enable caption in arabic so um at the bottom of the of of the screen where you have the zoom you can click on more and then you click on captions and then you can select the language that you you want your caption in and it will easily translate what i say in the language that you prefer all right um now i probably can uh start so my name is omar abdelaziz i am an associate professor of mechanical engineering at the american university in cairo and i am the student activity chair at pyramids chapter and i do other activities as well uh uh internationally today we will cover uh data center cooling first we would like to know why uh we need to cover this so looking at the growth and the energy demand what are the key performance indicators and then what are the conventional uh cooling technologies this is important just to build the basis right everyone hopefully know what is the conventional air-based cooling technologies in the center and then we move forward and talk about the advanced cooling technologies like direct chip and immersion cooling technologies and other emerging to technologies and conclusion so the iea has published a very interesting report called uh data center and ai or energy and ai and it predicted uh the global installed data center capacity in gigawatts and it's uh the number is alarming as you can see here uh you see there was very little growth between 2005 and 2016 the growth was not that big right but since 2016 something happened which is the advent of ai and with the advent of ai the growth has been almost exponential and you see this at the enterprise level at the co-location level and the hyperscale level but the most the high increase is in the co-location and in the hyperscale data centers and the difference between enterprise co-location and hyperscale is that enterprise data centers are data centers made primarily for companies or for telecom companies and and things like that so it is more controlled data centers and you can see that the growth is not that big although recently we have seen a little bit of growth because i because again ai is increasing the demand for i.t in companies co-location are data centers that companies built not for their own use but to rent out data center spaces within their data center so for example i am a new company uh in the it space and i want to uh put uh 10 racks right do i where do i put the 10 racks do i just put them in my in my attic probably they will burn out so i go to this co-location data center and i rent a space to put my racks so these are co-location and again the demand for co-location that center is growing especially in countries where there is a lot of growth in in it companies especially small and medium enterprises hyperscale is like the the big name companies that work in it that either use it for their own use or for cloud compute now if we look at the energy used so this is the gigawatt if we use if you look now how much energy this is being these are used you will see that the growth it went from 150 terawatt hours in 2005 to now almost 500 terawatt hours in 2025 this is 3x increase in just 20 years that's a lot a significant increase and again we see the same trend almost no nothing no much increase in the first 10 years however there is a significant growth in the last 10 years now what i like about the curve in the bottom here is how much cooling is consuming right look here cooling in the past was a significant chunk of the uh of the equation right so the green is cooling the server is the light blue so basically cooling was almost as much as the light blue right that means that that you have to spend a lot of energy to uh just running the data center now look with the advent of all of the cooling technologies that are available and we will discuss some of these today you will see that the cooling component it's roughly the same the the green piece it it doesn't grow much but you see that the light blue piece grew a lot right that means what that means that for the same amount of it we are using much less cooling and we will see another chart that directly shows that also it's important to note here that u.s china and europe contribute to about 85 percent of the global demand in that centers so why cooling matters because it equipment like servers and networking gear uh they produce significant amount of heat and i'm not sure if any of you have been near any of these or have been into any uh of the data centers you will hear the fans whizzing and you know the hot aisle can get pretty hot however air cooling becomes very ineffective when that density exceed 30 kilowatt you know there are some technologies to get us around 50 kilowatt but that's about it uh cooling systems can account to 30 to 50 percent of the facility energy use so when we use um air cool designs the power use effectiveness is very low and the efficient efficient cooling can significantly reduce the cost and increase the the reliability because one of the major impact the things that impact the reliability of our it equipment is how hot they get and if for any reason air does not get into our data center they get too hot and the rack would shut down so now with hot with high density ai equipment as you can see with uh with some of the recent uh chip manufacturers right how can you cool these with simple air cooled equipment you will not be able to do that so there needs to be new technologies before we proceed we need to understand what are the performance metrics that we have to use in order to easily compare and benchmark between different uh data centers uh also these help us to track the performance of year over year and also try to find how we can improve our performance the one and foremost used uh performance metric is called the power use effectiveness or power usage effectiveness pue for short and then there is water usage effectiveness wue pue and we are kind of the most used uh uh kpis in data centers and then there are some recent uh kpis uh that are are defined recently uh like um the coe or carbon usage effectiveness this is with the new uh and the new impact of sustainability uh the dci dcie or data center infrastructure efficiency energy reuse effectiveness and cooling efficiency ratio um maybe i need to increase this a little bit just to show what's happening here uh if you have a data center this is the it load 66 percent cooling 27 percent there are some electrical equipment other one percent lighting one percent these together would make the power usage effectiveness 1.5 so the power usage effectiveness effectiveness is just by the total energy used divided by the it uh usage ideally you want this to be 1.0 but is it possible probably not because you still need a little bit of lighting a little bit of other a little bit of electrical equipment because you have ups and and such and you need to cool the equipment right uh people know that 1.5 is typical however uh we are now going down to 1.2 and even lower and uh it can be influenced significantly by the ambient conditions so if you are operating a dead center in abu dhabi it's going to be much higher poe than operating at that center in copenhagen or in a place that has very low ambient conditions so in 2023 uh and or this organization here uh did a survey of 558 data centers and they grouped them by the capacity so small data centers all the way to uh hyperscale data centers and they found out that the average uh poe range from 1.69 for small data centers to 1.44 to large data centers but even the 1.44 this is the stock average this is not the new technology because now the new technology is about 1.2 if you go into one of the major companies like google and you look at the data center uh tracking website you will find that they are claiming less than 1.1 already and this is a significant improvement in energy efficiency the other thing is water use effect water usage effectiveness which is how much water we are using over the year divided by how much i the it used in kilowatt hour and basically it indicates the water used for cooling water uh cooling towers and things like that humidification and so on and so forth and we want the w ue to be as low as possible uh however in dry areas you need to humidify uh in some areas you use water water for dark evaporative cooling and that would increase w ue but reduce the pu e right uh the w ue is especially important when you are in an arid area because in an arid area you don't want to use water uh a study was done by lauren's berkeley national lab in 2016. they found out that the average w ue in the u.s was 1.8 liters per kilowatt hour uh and they estimate that if you have 100 megawatt data center what what that would mean is that you will end up with consuming 1.1 million gallons daily that's significant amount of water however if you look at how much water is needed at the power plant and that was the crux of this figure here right looking at the difference between how much water you consume on site versus how much water you consume at the power plant and you can see that the consumption on site is much much smaller than the consumption at the power plant and people usually forget about this um this interesting point because they care about just reducing water on site maybe that would increase pu e but if you increase pu e you are going to increase the water that is consumed at the power plant so the other metrics for co e for example it's basically the co2 emission emissions ratio divided by the it energy consumption the dci is the research broker of the pu e it's the it energy over the total facility energy uh ere is uh very important especially in uh areas where you can use the waste heat from the data center from the data center in the buildings uh around you or other industrial process heating uh using uh high temperature heat pumps or so on so with ere this is a new concept i'm not gonna talk too much about it in today's lecture but the idea here is to use the waste heat from the data center and try to up that heat to usable temperature that can be used to heat the dwellings heat pools provide the preheating for boilers and so on and then there is the cer which is the measure of cooling uh system effectiveness again another interesting uh concept or metric that tells us what is the quality of the cooling system do you have a good cooling system or a poor cooling system all right so data center so just to understand what is the anatomy how does the data center look like if you have an enterprise data center or if you have a co-location data center um typically a hyperscale data center would look similar to an enterprise data center um maybe the some of the architecture would be a little bit different uh like the difference between an uh an enterprise and the hyperscale is the distribution of the data hole in a hyperscale they will try to make the data hole symmetric and they push the electric room on the periphery somehow um so it's a it's kind of a a similar in in sorts but in this enterprise design what they do is they put all the air handlers on the uh perimeter they have the data hole in the middle here and all the electrical equipment with the batteries in the middle right there is the 12 kilovolt room here so this is the switch gear uh you have storage rack for testing right and here you have the uh point uh the uh the point the point of uh uh presence and point of entry for all the people who operate your uh equipment and this is the admin building uh of course you have a mechanical yard where you have your chiller that basically supplies all the chilled water for your air handling equipment if we look at the collocation now collocation is a little bit different it's a different animal and the reason why collocation is different is because it operates as you are trying to almost expand you are renting out the space so here you have computer room a and you can see they are renting out this space or renting out this space or renting out this space so you can rent out one block at a time and uh typically they are designed such that the raised floor can be reconfigured for anyone's or any renter's um rack placement configuration and here you can see that that uh the computer room air handling units are placed in the middle uh and they usually provide some spare units to like just in case in the future they need to increase the capacity uh also they have to always design for extension or expansion in the future all right all right this figure is very important for any um data center design engineer because you need to know like how cool or how much you want to how much you want to pull your your building do you want to provide it at 25 and 50 percent uh what is the comfort conditions right it's not comfort conditions it's more of uh uh appropriate performance conditions and you can see that you can operate according to ashley pc 9.9 which is the data center pulling uh technical committee and ashley 90.4 which is the standard for data center so here you have uh the recommended zone a1 zone a2 zone a3 zone and a4 zone so the difference between a recommended and a4 depend on many things mostly related to the reliability of the equipment and the supplier right the the rack supplier what does the rack supplier means but you can see that you would operate in a1 like very comfortably a1 here between 15 degrees c and 32 degrees c so you don't need to keep the room keep the data center at 25 you can keep the data center at 30 or 31 or 32 without compromising the performance of course everyone tries to design here 2012 or somewhere here why because this is safe performance but no you don't need that you can operate all the way up here right so if you operate if you design here you are actually still within the safe region because you are still below the a2 well below the a3 well below the a4 and uh there is if you design with proper factor of safety you will always maintain your performance in the a1 window the conventional design is simple uh we have data centers these you know racks i tried to design them with uh ai but it was for for design so we have racks here uh you have raised floor you have the air cold air blown from the raised floor entering the data centers and from the back what is happening is the hot air leaves and goes into the crack the crack pushes the cold air down in the distribution and in the underfloor cooling and and and so on so the the key components here is that you have a crack or a crack these white boxes in the back right and if you have a crack computer room air handling unit then you need a chilled water loop with a chiller if you have a crack computer room air conditioning then you have a dx system that provides the um refrigerant at the right conditions these systems have been widely deployed people know how to use them how to design them and i don't need to dwell on them too much what is new here is the direct evaporative cooling can we use direct evaporative cooling and i attended a course and actually uh course uh i think it was maybe a almost a year ago and i was surprised that fortune 500 companies uh in operate or relied exclusively on uh direct evaporative cooling in arizona very hot climate georgia very humid climate alabama very humid climate nevada very added climate so added and hot climate arizona and nevada again humid climate georgia and alabama so you can design your data center to be called using direct evaporative cooling and uh perform really well so you can see here this is the recommend the black here is the recommended zone right and the plots here in this chart is just uh for um uh you know the phoenix uh uh ski harbor international airport weather so basically the hours of the year where the point is plotted on the psychometric chart so uh you know what what where is the worst case condition here the worst case is where you have the hottest temperature right and this is the hot the worst case right this is kind of the worst uh or maybe this is the worst case probably this is the worst case right because this would have the highest ambient condition and high highest wet bulb temperature right so if you do what uh direct evaporative cooling you go all the way here right and then you you allow the room to uh to cool down to again because this uh are in um ip not si but when we say 31 so 85 so you can easily go from here to 81 and allow the room to cool again so there is no problem because it's a sensible heating uh or the sensible or the load is since the heat factor of one so if you supply the con the room with conditions here right typically they stay at 80 percent so you supply it at these conditions right then you have a lot of air flow rate you push a lot of air flow rate but you still would stay within the a1 region any questions here please feel free to ask questions if you have i'll look quickly here in the qa box if there are any questions um so the source of the chart uh well this is the ie the source of this chart sorry the source of this chart is ie the international energy agency and the link is provided in the uh at the end of the references the comparison between crack and crack in terms of electricity consumption and lead uh crack usually so i forgot to put this uh chart but usually crack would have much higher energy consumption compared compared to crack but crack is used with smaller um system so if you don't have a big uh data center you don't you cannot justify the use of uh cooling of a computer room around here you probably uh it's sufficient to use a computer room air conditioning so that's why in this chart here when you have small capacity the poe is high because typically these rely on crack and not crack um so hype is asking about how wet bulb dry bulb relative humidity and you aligned and how select range of temperature and humidity set points i'm not sure i know which slide you are asking can you say which slide is this are you talking about this slide so hi all right so until as i clarifies the question i will see later um in we 1 million gallon per day is used to cool down 100 megawatt is that a problem if that is recycled if that is recycled that should not be a problem but if it's not recycled it cannot be recycled because the water is typically evaporated into the air so you cannot recycle that and that's the problem uh how does the humidity affect it equipment this is a very important question so uh one of the companies that worked with this they found out that if the supply temperature uh uh supply humidity is less than 65 98 of the time there is no problem whatsoever but if it's more than 65 percent two percent of the time it can be a problem so basically what they do is they uh and that is add that inlet to the um the equipment right so you have to you typically if you see here right we we there is a mixing happening so you you provide it at 80 percent and then you mix so it becomes 65 percent so usually as long as it's 65 percent right you operate at 65 percent you are there is no problem uh and uh the the the limit or the demarcation is two for two two percent of the time higher than 65 you are uh you you you can be in trouble but two percent if you are 98 percent of the time or more below 65 you are okay all right moving on so this is a very important slide because uh here people typically use the uh design ambient conditions based on this wet bulb and the mean coincident bump uh dry bulb but when we use data centers we cannot rely on these conditions because these conditions are for a typical year data centers we typically use the either 10 year worst extreme conditions 20 year extreme condition typically if you are if i am designing a center that is gonna stay uh and and work forever uh i would i would design it i would design it with 50 years extreme conditions so in in cairo egypt it would be 47 dry bulb and 28.8 wet bulb temperature all right one of the new innovation is the pumped refrigerant economizer because with pumped refrigerant economizer you can significantly reduce the poe here you can see what you do is you have a compressor uh and you have a pump and typically a condenser and and you have your evaporator in your crack right so in this crack you add um a pump that at the time when the out when the ambient conditions is cold enough to not need the compressor you actually pump the the refrigerant or create a pumped loop and by creating the pumped loop you reduce the power as you can see here if you are in a pumped loop the compressor consumes 0.6 kilowatt if you are in a compressor loop the compressor consumes 8.5 kilowatt so that the difference between a pump loop and a non-pump loop is significant 0.6 versus 8.5 and you can see here that with a full compressor the poe is 1.28 with a partial economizer phase is 1.17 and with a full economizer phase this is gonna be even lower than that probably 1.1 another new innovation is the passive rear door cooling and here what we do is we place a heat exchanger uh just a cooling coil at the end at the back end of the rack and by doing so the hot air leaving the rack gets tempered or pool and before it leaves so your hot air is not hot pile anymore and you can pack your your um your racks more closer and this is helping uh making the data centers more dense of course here uh it relies on the surface of the fan so you do not need an additional fan here it can use the chilled water or even the condenser water because the air leaving the rack can be at 45 degrees or it can be at 50 degrees or more so if you put here air coming from the condenser water at 30 degrees you can temper enough the good thing here is it provides a good redundancy because if the car is out of service this can provide significant cooling and now it can provide up to 20 kilowatt additional per rack so if the rack can provide 30 and this is 20 kilowatt then you can go up to 50 kilowatt per rack there is some challenges because of the dew point control because you don't want to have any condensate on this coil and also now you need to create the rack ceiling because you don't want this hot air to to be spilled outside so you want this door to be sealed well on the door there is also the active rear door door cooling so it's similar to the one um we talked about however here we use fans on the heat exchanger and by using fans on the heat exchanger uh these are hot hot swaps swappable fan and by using these fans you can improve the heat exchanger significantly uh again we can use either the chilled water or the condenser water uh or even a refresher for cooling so any anything between 15 and 21 degrees c uh now with this design we can go up to 75 kilowatt per rack so the previous design we can go up to 50 kilowatt per rack here we can go up to uh 75 kilowatt per rack again the same challenges viewpoint control uh fan con uh the fan control also like how uh past you you want to control uh or you want to run your fan and what is the water valve or what is the opening of the valve that is controlling the coolant another important innovation is that chillers with free cooling uh this is used more in um in the second part of the presentation which is the direct chip or liquid cooling however i am seeing this now getting into air pool data center as well and what you have here is you have a chiller that can operate either in a just a chiller operation or chiller plus free cooling or completely free cooling operation and that depends on what is the ambient condition so if the ambient condition is low enough then you do not need to run your compressor and all you need to do to provide cooling cooling for your water is to basically create having this run around coil with this indirect heat exchanger to cool your chilled water loop so this can provide cooling especially for systems like what we will talk about later because with um with the liquid cooled data centers the chilled water temperature is much higher than air cooled that said with free cooling what we are doing here is we are actually have running both the compressor but also running this um economizer and by doing so you can improve the overall performance of your chiller significantly all right all right so overall the limitation of air cooling as we said earlier is what is the maximum capacity 50 kilowatt 75 kilowatt per rack with the active back door cooling air has low thermal capacity so you need a lot of airflow so your fans are big and noisy uh when you are operating in high ambient temperatures there are sustainability concerns uh you cannot use waste heat right the the heat that is waste that is uh wasted from the it equipment you cannot cannot be utilized and we need a new technology so liquid cooling uh the first thing is we are increasing the density with the server uh um using gpus ai accelerators and and and we can go up to 200 kilowatt per rack and we need to achieve our our sustainable sustainability goals the first technology i want to talk about is what is called direct to chip cooling and in direct to chip cooling basically you have a rack and inside this rack you provide uh the cooling water through uh the cdu and the advantage is that it has better heat transfer lower inlet temperature less need for large fans and the cold plates use non-conductive fluids and um basically you can apply this and scale it in the current data centers right do a retrofit uh to move your current capacity so if you have a data center that is designed for 15 megahertz uh it if you retrofit your air cool direct equipment with uh drive to chip rack equipment you can double your it capacity from 15 to 30 megawatts so just by doing this kind of retrofit you can double your it capacity of course you need to make sure that you have the power from your um infrastructure to uh power your equipment so this is just a very interesting uh cartoon showing the the way it looks like and basically you have your uh your uh your it equipment inside the rack and these are what we call the servers with the cold plate and you have the coolant distribution unit and you have your outdoor heat rejection equipment so your cooling distribution unit will uh cool this this coolant right using the outdoor or the chilling equipment and send this coolant back into the pad the poe for a single phase direct to chip is around 1.13 and for two phases around 1.07 the key components are you have to have cold plate that are mounted on the chip the coolant distribution unit or cdu which includes a pump a reservoir loop and the heat exchanger and the secondary loop so this is our secondary loop so this part here is what we call the secondary loop and this is what we call the cdu which has inside the the pump the loop and the reservoir and then here we have our uh our cold plates over the servers uh the challenges is that there are some need to do leak risk management because the the refrigerant or the coolant wood is can leak so we have to be careful about that there is additional plumbing and also the server design maintenance is additional and coolant selection is important uh you want to make sure it's dielectric it's not going to be conductive you want to make sure that the coolant is having uh good thermal conductivity and so on so what is the temperature of the water of of the water that you use so we have here heat rejection equipment so what is the water temperature and uh according to actually they created this guideline so you can provide coolant like here this water that is entering this blue uh blue uh line right entering into your cdu can be at 17 degrees c 27 32 40 45 or greater than 45 because we don't we are we are actually extracting the heat with the with the cold plate directly there is no air so you can afford to use at higher temperature so you can go even at higher than 45 so if you are providing at 17 degrees c you use chiller but now the chiller the chilled water temperature is 17 degrees c so your chiller is operating at much higher efficiency you can also use cooling towers only because it's 32 to 40 degrees or you can even use a dry cooler you don't need a cooling tower you can just use a dry cooler and then you can have supplementing supplementer uh cooling equipment to improve your efficiency so if you are using a chiller when the temperature outdoor is very cold you can use a water side economizer uh if you if you are using a cooling tower if it's not sufficient you might need to use chiller or uh you might need to use um district heating system and as a you know the and use it as a preheating so basically this would be your waste heat recovery technology for district heating plants now it's important to note that as we move from low temperature to high temperature what that means is that we are going to consume more water so that if you if i'm sending water at 17 degrees c this pipe is small if i'm sending water at 45 degrees c this pipe has to be large because i'm sending more water right because the capacity of the cdu is the same let's say this rack is is 100 kilowatt or 120 kilowatt right so this cdu has to process 120 kilowatt and the 120 equal m dot cp delta t so the delta t is the water and exit temperature minus the water inlet temperature and the exit water temperature cannot be hotter than what the exit from the um the freshen or the cold plate can be so there are some limitations we have to be careful about that now do we need uh a coolant distribution unit or not that's very that's a very important question and the benefits of having a cdu is that it isolates the water quality problems so if there is uh your chilled water plant or your cooling water plant has particulate by biological matters and so on and so forth you have you don't have to worry about this getting into your cold plate also you have a good thermal isolation so your water volume that is cooling uh your um your it equipment is more controlled your your leaks are more control controlled your maintenance is more controlled fluid properties are more controlled controlled you know someone who can come and say oh i want to put this additive to improve thermal conductivity if you do it on the small amount of water for your cdu that's much easier than doing it on your plant and then finally there is this clear demarcation between the customer and the it manufacturer responsibility the main challenges is that with the cdu you have to pay more so there is an additional cost and it also makes the system more conflict complicated and there is an additional maintenance measure because you are adding this equipment with the pump with the heat exchanger sometimes there might be some reliability issues of the additional components like the pump inside the cdu and finally there is a penalty in terms of energy efficiency because you are adding this heat exchanger which results in an approach temperature inside uh between the coolant or the customer coolant what the chilled water that i'm creating and uh the cold plate as well as the pump energy this is a typical uh server design and in this server you can see this is the main board this is the ram this is the cpu and you can see here that i'm not getting all my cooling using the cold plate but typically with direct to chip 80 percent of the cooling is achieved 80 percent of the cooling is achieved using the liquid and 20 percent is achieved using air so basically up to 55 degrees c goes in and what leaves is up to 60 degrees c so if you remember from ashley we said 17 uh 27 30 35 and so on up to 55 so the maximum 55 and the maximum it leaves is 60 you cannot have it leave more than 60. this is very important uh and just to give you a a visual this is what looks like uh an air pool uh data center air pool track versus uh a cold plate track so you can put too many like here you see how big this is you see how slick this is so it actually you are packing more it power more it power in a much smaller uh footprint now what are the benefits and challenges of direct to chip you have higher thermal efficiency which improve higher compute density you have potential lower fan air flow you also have better temperature uniformity which means you have better reliability for your components also with the waste heat because now water is leaving at up to 60 degrees c so 60 degrees c water can be used as a waste heat source the challenge is you have to pay extra for your for your equipment the architectural changes to the server and drag design there is an issue with fluid management because you have to use dielectric fluid and you want to make sure that there is proper leak detection and also the integration with the existing pooled uh infrastructure and i've seen uh systems where we are slowly slowly uh retrofitting d2c with uh or retrofitting all air pooled equipment with d2c in the same data cell uh so what we understand from this is that d2c is a strong stepping stone stone towards full immersion but it will require hybrid integration approach so it's not the best but it is kind of a good stepping stone all right what's what is a a new technology a really good technology so we we like last time i was at the ashley uh expo uh in january this was making head uh head spinning heads right which is single phase liquid immersion cool all you do is you place your uh servers inside open liquid uh uh boxes like this right so you have a coolant here and this coolant is cooled using a cdu your coolant distribution unit right and this coolant distribution unit cools without uh with the chilled water or water from your facility now the design of this server uh box is interesting because you want to make sure there is no corrosion there is no erosion and so on it contains the dielectric fluid and the dielectric fluid can go in and out and uh absorb all the heat from the servers then there is the next technology which is two-phase liquid immersion so here we said it's enough to keep the liquid in single phase so it does not evaporate all right now we said what will happen if this liquid boil actually if the liquid boils then we will improve the performance but if the liquid boils we cannot keep it open we need to keep it closed right so now we have what we say two-phase immersion cooling again using dielectric fluid that boils it has to be contained in a contained equipment like this more more expensive equipment more expensive servers or racks and what happens is if you look here you have your server the heat would boil the dielectric fluid and the boiled fluid go up rise and condense on this heat exchanger and drop back again and the cycle continues so there are two benefits the first benefit is that you are using boiling heat transfer coefficient which is much higher than um condensing than um natural convection heat transfer in single phase uh liquid immersion the second thing is you are using uh the the latent heat of condensation so you can achieve much higher capacities and i have seen up to like i have catalog data for more than 200 kilowatt per rack so the figure this one on top here is 100 kilowatt per rack but i i have at hand 250 kilowatt per rack so the challenge is you have infrastructure for handling the vapor how are you doing it and you can see that this becomes very challenging uh also the fluid selection is important possible component compatibility can be an issue so you cannot just use any server so you have to work with the server uh supplier to make sure that the server is compatible with your system however this is gaining traction uh in high-end dc or dead center cooling applications all right just to summarize what is the difference between single versus two-phase immersion so if you look at single phase immersion you have a box here and you have your servers inside the box right so what you do is you you have your single phase you have your single phase dielectric cooler it goes into your cdu it cools down using your heat exchanger right this is a pump here and then it goes back into your server rack so you have your cdu and then you have your coolant your plant cooling system right and your your plant cooling system can be as we said chiller cooling tower or dry cooler so you have this thing in the middle which would result in additional heat external delta t and additional pump power with two phase it's much easier because all you have is you have your server here this is a hermetic design and what happens is you have your uh chilled water or cold water or cooling water goes into your condenser leaves back so all you need is a pump and a heat exchanger with your plant and here the refrigerant or the dielectric fluid that boils go up condense on the heat exchanger and goes back goes back down so it's a much simpler design so while this server is expensive the entire or the balance of system is going to be cheaper because you don't need all the uh additional components like the cdu so if i if i want to compare between direct to chip and immersion uh if you look at the coolant contact in the direct to chip you need to have a cold plate on the chip surface and when you add the cold plate that means that you need to have an interface resistance and the that means that your coolant temperature would be at a higher temp needs to be at lower temperature to achieve the same uh chip cooling level whereas with immersion cooling the entire server and the components are immersed in the the fluid so you can achieve the same cooling with much higher coolant temperature the infrastructure change in d2c is moderate you need to probably change the rack and add some piping with immersion uh cooling you actually have more radical change and it's more of a green field design than a retrofit design uh the heat removal capacity for d2c is high but again limited to how much the cold plate can handle whereas for immersion is too is very high so the highest uh d2c i have seen as a rack was around 120 kilowatt the highest single phase uh immersion cooling uh rack server rack was around 150 and the highest two phase was 250 250. this is again the numbers that i have uh again um i might be a little bit outdated but this is the numbers that i have uh the airflow requirement for d2c are reduced because i told you you need 20 air cool 80 percent liquid cool whereas for immersion you don't need air at all you just need a little bit of air just to make sure that people do not die inside the data center so you need enough air to provide cooling for that data center air so that people are comfortable comfortable enough in terms of maintenance and service complexity the d2c is moderate whereas immersion is more complex and in terms of uh life cycle consideration d2c has medium complication because it is a a system in progress whereas the liquid the immersion technologies have higher upfront cost but potentially they would have significant savings over the lifetime because you don't have to pay much electricity in terms of use cases uh the d2c are for high density racks and upgrades and hybrids whereas immersion when you are doing high performance computing ai new builds and and ultra high density applications so in terms of environmental benefits um the we are like water and energy reduction is important immersion cooling significantly reduced uh reduces water and energy consumption in hardware operation there is a lower carbon emissions and improved hardware lifespan and um these are again and a reiteration on the um on the implementation challenges uh we are saying that liquid cooling infrastructure uh is important uh is important and it requires new servers uh and the traffic cost can be high uh again there are issues with compatibility uh in terms of service serviceability serviceability immersion tanks may may change access to equipment like it's not easy to swap out and maintain equipment in terms of standards and metrics the traditional poe might not be sufficient uh we need uh we need probably new uh metric or define poe a new way so that we can capture the impact of uh additional uh power used in immersion cooling system uh while we are going to reduce the air cooling fan power probably there is other heat rejection which is the other end uh that we talked about like here let me go back to this figure which is this box here what is the heat rejection here what is what how are we doing this heat rejection so we said in this slide heat rejection can be done using chillers cooling towers or cooling tower and or dry coolers right so we are adding another layer of complexity uh to the system and again there are thermal and plumbing risks and finally there is a great opportunity for waste heat recovery uh i'm running a little bit late but there are hybrid systems so what we call hot water loops for heat use uh and uh this these are basically uh when you are in d2c leaving at 65 you you get this hot water into uh uh a district uh heating plant so you can um uh preheat or provide heating to uh uh to uh system so significant uh use you can also use this for uh source energy for your absorption chillers especially low low temperature absorption chiller also ai a based control and optimization is very important because there was a a huge debate on how to optimize the performance of cdu's and it was found out that you can create uh optimized sequencing using ai and finally uh in term in terms of uh microfluidic cooling rather than using cold plate so again uh since we know that d2c is a stepping stone and would probably be a good option for retrofit and there are there is significant bank of data centers uh so instead of uh using like you know completely story destroying them and going to emerging immersion uh looking at ways to do the retrofit with d2c rather than going to the cold plate which has some limitation going into the micro fluid the micro fluidic cooling would um unlock significant potential for uh improvement because the micro fluidic means that you are going to to pull that the chip directly and by doing so you are going to remove a layer of thermal resistance in terms of uh future outlook uh the micro fluidic or the cooling channels inside the chip packaging which is going to push the thermal management deeper uh two-phase cooling becomes more mainstream as the power density is going to continue to rise uh again the waste heat reuse becomes standard and sustainability metrics will evolve with time so pu e and w e are not going to remain the uh standard other features or other metrics will continue to uh evolve and finally um geography and climate will drive design choices so in hot climates water scarcity energy cost are going to push more towards liquid immersion uh unfortunately i couldn't present today i have a very interesting study that shows um that in different climates the choice between uh air-cooled d2c uh single phase uh two phase uh two phase uh two phase can vary and uh i'll present this in the next ashley so uh stay tuned so if you come to ashley try to attend my lecture so uh again it's a it's that center continues to grow and um one of the key uh lever arms for efficiency and sustainability sustainability is to improve efficiency and also it helps us to improve the density of data center and we can no longer rely on air cooling uh next generation technologies like that that bag to chip or single phase immersion and two phase immersion are going to uh be kind of the mainstream uh people are still apprehensive regarding two phase immersion however uh i have worked on it i've done a lot of research research and i believe it it it can significantly uh provide can provide significant energy saving and also uh life life cycle cost saving and for uh organizations planning the center growth especially ai uh and high performance computing the light liquid immersion cooling is no longer optional it is becoming a strategic mechanism with that i'd like to thank you uh if you have any questions please uh let me know i think we are i see like 34 questions so farad i'm not sure what is the best way of of doing it [01:02:27] Farad Musa: okay so much thank you dr omar for this prosperous uh presentation uh and uh yes we have questions in the q a uh if you would allow me before we answer the questions i would like to thank you again with the introduction uh that i had for you because i lost connection before and i apologize again for this uh we had dr omar abdul aziz dr omar is associate professor of the thermal fluids with over 20 years of experience in r d and project management and he earned his phd in the mechanical engineer from university of maryland college park he specializes in energy efficient building technologies sustainable energy production and alternative heating and cooling systems including lower global warming refreshments he also served as a senior fellow at the u.s department of energies building technologies office for two years as he led the building equipment research group at ornl overseeing a diverse research program with an annual budget of millions of dollars and also he's an active member of several professional organizations including ashley of course and served as a co-chair of the refrigeration air conditioning and heat pump technical options committee rtoc for the united nations ozone secretary uh techno economic assessment panel as internationally recognized expert he has published over the 100 peer-reviewed papers and holds several patents wow this is the introduction you should have had at the first place i'm sorry because uh i know you introduced yourself quickly but uh i took time to summarize this bio so i i have a great bio here i had to tell thank you so much dr omar and now we can have the uh questions if you would like me to [01:04:26] Omar Abdelaziz: go through it or would you like to read it uh i'll go through the questions and i'll read the questions and and try to answer them as much as i can so there was a questions on uh there is a question on the psychometric chart so i will go back to the psychometric chart so this one and this question on the psychometric chart was uh how do we um look at the dew point and um uh wet bulb dry bulb rather humidity and you aligned and how to select the range of temperature and humidity set points so so basically uh as you said as as you see the black box the or the black area here this is what we call the recommended zone so if you operate is within this recommended zone you are fine right and then if you remember there was an earlier slide this one you see a1 is the extended zone a2 is the further extended zone you can operate all the way to a4 but a4 is kind of okay for uh the different level like depends on the equipment that you are using people usually only work in a1 not the recommend like extend beyond the recommended and work in a1 so if we are in a1 you are going to probably be all the way to 82 right or 85 let's call it 85 right so it's going to be like this this is your your zone so let's say i want to be i want to have my conditions here so if i want my conditions to be at 85 and 50 right and i am the outdoor conditions is here so i will cool the i i will pull my equipment i will pull my air i will do i will do wet bulb cooling so i will go i move along a constant wet bulb temperature and then as a supply what it would do inside the room is that it will have sensible heating so it will send it will have sensible heating all the way to 85 so that means that that if i want to calculate the amount of airflow the amount of airflow is going to be uh related to the difference between the point that i'm supplying and 85. so if i am here let's say which is the worst condition right so how does that compare like how much cooling can i so can i provide if i'm here the cooling the lowest the the temperature the wet bulb temperature that i can achieve is 77 right so i only can achieve up to this point right so this point is what go down this is 81. so at this condition i will have to basically push a lot and a lot of air because i want to move from here back to 81 to sorry from 81 or from 81 to 83 so cp m dot cp delta t equals the capacity of the data center and by doing so i would see that the volume flow rate i'll be pushing at this condition which is 1 to 11 hours per year is a lot but if i look at the worst condition that is 2 12 to 22 hours per year which is this one right then the worst case i'm not pushing much air so it has to be designed like do i care about the worst case 1 to 11 hours or the 12 to 22 hours or do i look at that ashley extreme case which is 50 years extreme case which is this one and with this one it looks like i can like the the i can only get to 85 that means that i cannot not meet the a1 condition but i will meet the a2 condition there was another question about the humidity so as what i said let me reiterate that very clearly if your equipment is seeing 65 percent relative humidity or lower 65 relative humidity or lower for 98 percent of the time or more there is no issue whatsoever so basically if you are supplying at 80 percent and then you mix with the room you will definitely be more than be lower than 65 because you have to go like this but here you probably are going to go to uh 70 percent so so sometimes there are some maybe one hour per year two hours per year where there are issues where direct evaporative cooling can be a problem so the issues of 20 years extreme conditions are enough that's why i said no i use the 50 year extreme i don't use the 20 year extreme so in my designs i use the uh where is that so there is 10 years extreme 20 years extreme and 50 years extreme so i use the fifth 50 years extreme not the 20 years extreme the difference between the free cooling and the pumped refrigerant economizer uh the difference is that in free cooling here you are actually using the same refrigerant uh line right so you are not having an additional heat exchanger so your your condenser basically you are disabling your compressor and having your sub sub sub sub cooled refrigerant pushed through your uh your your system and then back into your condenser so by doing so you have a design in which you are having a continue uh the same heat exchanger pipe the same piping you are not separating this pipe this the the pipe of the of the uh uh uh what is it called the the part of the pump from the pipe from the or the from the condenser to the evaporator so it's kind of in in line or integrated economizer whereas if you look here right with the free cooling we are adding a new heat exchanger and a new coolant typically uh uh propelling propelling propelling propelling glycol so it's a diff it's not refrigerant because this is a pump and this is a coolant and you have another heat exchanger so it's a little bit different uh two phase direct to chip uh cooling so in two phase direct to chip the difference between single phase and two phase is that you see the probably it will appear in this slide right so in single phase i have a liquid coming in at 55 and leaving at 60 whereas in two phase i would have a liquid uh coming in at 55 but leaving as refrigerant or a or a or as or as a vapor at 55 so this fluid goes in and boils and by doing so i'm going to uh create uh or use the latent heat of vaporization of the fluid and by doing so there are two improvements the first one is you are using boiling heat transfer coefficient which is higher than the convection heat transfer coefficient the second thing is you use smaller pipes because you need less refrigerant flow so the problem is the dielectric fluids that boil are more expensive than the dielectric fluids that are single phase and the direct fluids that boil are also more difficult to work with because they are volatile and and we have to be careful about how to work with that all right um now i see a question that i'm not understanding why gb 200 servers cannot be used in immersion cooling that i am not sure i understand what is gb 200 can you elaborate mr muhammad bilal all right uh mr mr tofi uh so basically for direct to see to to uh direct uh to chip cooling it's actually fairly simple because these are uh swappable and the cooling uh component here these are uh quick plugs so you have valves you close the valve you have quick plug so you can easily quick plug uh like uh or what we call it quick connects so you quick connect disconnect the cool the inlet and the outlet and you can take the whole server out maintain it put it in swap it if you want that's easy for immersion uh if you look at this figure for a single phase immersion it's easy because this whole thing does not boil so you can you see the handles here so you can and these are safe fluid so you can put your hand on these handles pick your entire server up of course you have to unplug all the uh electricity and all these things this is dielectric fluid so you will not get shocked or anything so it's safe uh i think that the main challenge is with this one so with the two phase the problem is it's uh boiling right so since it's boiling you cannot have it's not hot swappable so we'll have to kind of shut down the operation in order to be able to uh swap the server again uh for for the two phase there there is a way to contain the fluid because you actually have to go into service mode and when you when you go to the service mode so the selection of the dielectric fluid is is such that it is liquid at room temperature it's liquid at room temperature it's liquid at room temperature so since it's liquid at room temperature there are little fumes uh or no fumes and of course operation and uh operators have to wear proper ppe a question on do you think modern cooling techniques such as magneto caloric or elasto caloric systems could be good candidates for cooling data that servers um i i i worked on magneto i'm i'm actually working on a magneto caloric project and have worked on elasto caloric before in the past i i i i like that you enjoy it mr uh abdul maiz but i don't think it's uh uh it's possible that um uh that we can use uh these technologies in that sense there is a question about compressor put in crack unit so this question again it pertains to the design choice of the uh of the manufacturer some preferred the compressor to be uh placed in the same unit with the with the uh inside the crack just to make it more uh reliable some some preferred outdoors it's again a design choice do we get user requirements urs i'm not sure i understand the question mr muhammad mansoor what is urs what user requirement urs for drag manufacturer i'm not trying to understand the question mr muhammad mansoor ibrahim i'm not sure i understand your question do we normally get a user requirement urs for rack manufacturers with recommendation for cooling system required so i'm not sure i understand your question but typically user requirements are for the if like the direct the the system architecture and then uh as a consultant you will have to give them potential providers i'm not sure if they have a pre-qualified list or not so with regard to data centers uh the criteria for indoor air quality uh is mainly as i showed in the um in this chart in this uh segmented chart right so again these are the allowable envelopes right and these are based on levels of pollutants verified by coupon measurements as indicated in this section so so again there is section 9.4 uh section 2.2 in standard 9.4 that would define that unfortunately i don't have that on the top of my list but uh if you follow up with me via my email i'll be more than happy to uh find that for you uh when it when it comes to the design consideration with ashley 50 20 years maximum i think it makes more more sense to use the maximum db plus the average bed bulb for the location itself the maximum wet bulb isn't it because the same two maximum uh temperature would coincide together really so mr yasser i agree with you but when we design for direct evaporative cooling you design for the maximum wet bulb when you design for uh dry coolers you design with the maximum dry bulb i'm not saying use them together it's just that you use the maximum dry bulb when you design for dry coolers use the maximum wet bulb when you design for um for the dark evaporative cooler and uh if you want the like the point on the circumventor chart the best way to do it is to actually uh use um projected data from ie uh the immersion technology actually does not affect the data center redundancy requirements the center required the redundancy requirement is based on how you design the architecture of your data center so the redundancy requirement depends on how many racks that do the same um the same the same thing are there in the center and are they on different electric uh electricity supply lines and so on so it the the new emerging technology does not uh affect that what affects it is that system architecture uh unfortunately i'm not going to provide the pdf of this webinar but i'm going to allow uh ashley tools to share the present to to share the uh video what is the desired temperature humidity for high ambient markets like a so for high ambient markets and like uh um like uae and kuwait my recommendation is to again operate or design somewhere at 27 and let's say 50 50 50 50 50 but i would go all the way to maybe uh this condition which is going to be 32 and 40 percent rh as our design criteria where we can actually go up like this and go to 50 without even without any problem the new ai data center have a different behavior in matter of node fluctuation than typical one can you highlight this issue uh thanks mr muhammad actually this is a very important uh question because uh that's what actually i tried to discuss but i was kind of rushed towards uh and uh so uh where is that so ai uh actually the the the the ai that center can can be idle and then all of a sudden they get a question that they have to run all their models and and and that can basically go high load low load high low load high load low load and so on and by doing so so the it equipment uh would actually um cooling load would fluctuate significantly air cooling cannot easily manage that however uh if you go to the direct to the direct to chip you can kind of track that with a proper proper ai based control optimization so if you have an ai based control optimization that can uh have the pump of your cdu uh operate with variable speed control so that it can like change the fluid of your coolant of your uh cdu coolant uh the current inside the cdu to to basically track this cycle of the of the dead center that would be great now with single phase and two phase immersion this is not going to be an issue right and that is why people love immersion technology because especially two-phase immersion uh because with two-phase immersion what will happen is two-phase bath the back of the immersion coolant itself is create is is is creating kind of a storage facility it's like a storage right so uh if you have a burst of eye of it processing power or what happens is some of the point will bubble right and it would take longer for the coolant to condense that right but eventually it will condense so uh and at times where there is no load you are condensing more of the vapor on top of the uh of the back and and eventually things will balance out through time so that's why two-phase immersion is very attractive for ai applications uh are there available temperature range ranges specific in the ashley guidelines intended for the cold aisle uh supply conditions yes uh actually if you uh if you open ashley 90.4 which is available online uh because in the uh ashley standards online database you will be able to find all these information it's available it's free uh because it's part of the ashley and u.s department of energy commitment for uh building energy pool savings mr under rahman mahmoud uh again the question about dead center air quality and the concern with evaporative cooling based on the experience in the u.s it's not a concern because people do not stay long time inside the data center that center is for machines it's not for human uh and air quality is not is not an issue uh as you as you can see in the anatomy of the data center that i have shown earlier right people do not stay too long in the in that data center right the point of presence is here right this is the point of presence this is the data hall so if these air handling units are that are back evaporative cooler you take outdoor air you humidify it and push it in right uh and you have some somewhere of exhausting the humid hot humid air that's not going to create an issue for uh for um air quality what is the refrigerant module we use the refrigerant module we use uh is i don't know i don't know i don't know i don't understand exactly what you are asking uh late but uh there are different refrigerants uh that are used in the chillers uh like in this chiller yard for example we can use anything like uh the same refrigerants used in typical building design however now with the push towards sustainability uh they prefer to use lower gwp refrigerant lower gwp refrigerants why do we use pump after compressor in case number one i don't understand which one this one the pump is not operating this is uh the gray means that there is no not working green means working so in mode one the pump is not running it's just right here everything is gray means that it's not working so it's just this is the coolant it goes into the first heat exchanger but it's not working and then to the second exchange of course clever people put a three-way valve to avoid the pressure drop in this heat exchanger when they don't need it to operate uh the i cannot just i present the main steps for design queen at the center it's actually a whole uh workshop and design date probably can take half a day or more uh so i unfortunately mr later i will not be able to do it today uh in terms of sustainability requirement there is actually a very good and informative uh uh procurement guide by unit uh sorry by undp uh that was developed to say how do you procure equipment and data for equipment uh for data center what should be the coe what should be the water use effectiveness uh how do you select equipment how do you look at coe and so on and so forth if you are interested i'll be happy to share uh that document with you and you just have my email and i'll be happy to do it okay a1 a2 and a3 and mr sawi is asking about these so this uh uh envelope actually in the uh temperatures uh this what we call the supply it temperature the temperature and the supply it equipment uh so it's the mixture between the supply air temperature and the return air temperature uh some people can think of it as uh you know almost supply air temperature but in um in the in most hot air hot cold air hot air hot hot cold air hot air aisle we think of this as the cold air aisle temperature mr mohammad arfa is saying they remove the compressor itself from the cycle but now there is a wise to add a pump please explain so if you remove the compressor from the cycle that means that you need to still have a way to run your refrigerant so in frankfurt which is a cold weather right uh if you are operating your system at higher temperature you don't need to create a diet expansion and you can basically in a city like frankfurt your temperature is year long below 35 degrees c so you can actually have a pumped refrigerant uh loop so what you do is you pump your refrigerant between the condenser and the evaporator and you you get rid of your compressor completely uh life is asking whether we can use ammonia there was actually a case study done by one of the un implementing agencies to use ammonia as the refrigerant in uh a chiller for a data center and it's possible the problem is this is a very difficult uh refrigerant to to work with of course that centers are nice because you have actually they are a chiller yard of you know 10 000 uh ton or something like this so so especially for hyperscale that's the data center so it's a big that center or a big chiller uh so you can afford to have your maintenance engineers for a ammonia plant however it's not necessarily the best um the most the most the most used refrigerant i understand mr uh mustafa uh basiuni about the indoor air quality and uh it equipment corrosion which is accelerated and that's again my answer so that here yes the acceleration uh there is a equipment corrosion acceleration if you are going to operate with a temperature with a relative humidity higher than 60 percent so if your return return relative humidity is higher than 60 percent you will corrode your equipment but you will not operate at higher than 60 percent that's what i said is that you want to operate at i'm i'm i'm struggling to get my let me see if i can have that is that a way to i use a highlighter one second maybe i can should we do it this way so let's say i i i want my conditions this is my this is my line right this is my operating line right and i'm okay to achieve i want to this is my line and i want to be 60 percent so i said actually 65 right i said 65 so 65 is this line so that means that this is the maximum temperature i can have as my return right so this is my return maximum okay all right so if i am working with the worst condition this is the worst condition in phoenix i work with a direct evaporative cooler so what i do is i i do evaporative cooler right i say i i can only hit 80 percent above 80 percent is not good so i hit 80 percent right with 80 percent i have in this one and i go in the room i go sensible cool sensible heating only so i'm not gonna go here i'm gonna go here which is even lower relative humidity so i'm not going to leave the room with rh of 65 i'm going to leave the room with an rh of let's say 62 does that make sense now again let me try to do it so this is the maximum design temperature okay and we said i don't want to be above 65 so this is 65 where is 65 here right so this is my maximum condition okay so if i go and say i want to see what is the worst case condition so the worst case condition is this one i i humidify this one to 80 percent all right so i can achieve this point if this is my supply air temperature right this supply air if it goes into the space it will go sensible only right so my return air would be this return air so this return air would have a relative humidity less than 65 percent and to know them the flow rate is going to be basically this is delta h h room air this is h supply air or if i know the tp delta t whatever and you want to do it with it uh not on this page doctor okay most green most google dc is green yes and that is why because they are using what we call free cooling or direct evaporative cooling or um you know cooling from underground facilities so they use nature to provide the cooling so geothermal cooling they use uh river cooling they use uh just free cooling from the air either they put them in cold cold places to use free cooling from the air or they use uh dark evaporative cooling all right any questions all right thank you [01:35:10] Farad Musa: thank you dr homer so much thank you for your time and thank you for being with us with pyramids chapter thanks to everyone for joining us for today's webinar and please follow us for more events thank you so much thank you bye have a great night