Cooling AI Data Centers: Three Liquid Cooling Technologies Explained

[00:00:00] Speaker 1: Hello, greetings, and welcome. I'm Brian Posey, and today I want to talk a little bit about liquid cooling in the data center. So, data centers are getting hotter, literally. As AI workloads increase, so does the power that is required to run them. So, modern GPU-based servers, especially those that are used for AI training and high-performance computing, are pushing rack densities far beyond what the facilities were originally designed to handle. So, whereas racks once averaged 5 to 10 kilowatts, it's now common to see racks handle 30 kilowatts, 60 kilowatts, possibly even more. This is where liquid cooling comes into play. And liquid cooling might sound like a cutting-edge innovation, but it's actually been around for years. Anybody who's familiar with high-performance gaming PCs knows that liquid cooling has long been used to keep these systems cool. The idea is that by moving heat more efficiently than air, liquid cooling allows those systems to run faster, quieter, and more reliably under heavy loads. One way of thinking about this is that a modern AI rack is kind of like a high-end gaming PC, but multiplied many times over. So, the big challenge that liquid cooling is meant to address is heat removal. Traditional air cooling systems work by moving cold air through servers and exhausting hot air out the back. But air has its limits. It's relatively poor at carrying heat, and moving enough air to remove all of the heat requires large fans, wide airflow paths, and a lot of energy. Never mind the fact that all of those fans make a whole lot of noise. So, at higher densities, airflow becomes harder to manage, hot spots tend to form a lot more easily, and fan energy consumption rises. So, you're increasing your energy costs simply by the sheer number of fans that are required to move all of that air needed to keep your components cool. And cooling systems can struggle to keep up. And this is particularly problematic as GPUs increase, because GPUs generate heat in addition to what the server would normally generate on its own. So, in order to meet these challenges, data centers are turning to liquid cooling, not because it's new, but because it's now becoming essential. So, let's talk a little bit about why liquid cooling works better. The key advantage to using liquid cooling is simple. Liquid transfers heat way more efficiently than air does. Water, or engineered coolants, can carry thousands of times more heat per unit volume than air does. That means you can use a relatively small amount of liquid to remove the same amount of heat. Heat can also be captured closer to the source, even directly from the chip that's producing the heat. And cooling systems can operate more efficiently overall. Another major benefit is higher temperature operations. Liquid cooling systems can run with warmer coolant temperatures than traditional air chilled systems. And this unlocks two important advantages. First is reduced compressor use, because there's going to be less reliance on energy-intensive chillers. There's also heat reuse potential. So, waste heat can be recaptured and repurposed. For example, you might use it to heat a building, or you could use it for industrial processes. So, in short, liquid cooling doesn't just remove heat better, it changes the economics of cooling. Now, the important thing to note is that there isn't just one technology for liquid cooling. Liquid cooling in the data center is based on a spectrum of technologies, and each is suited to different needs and different use cases. So, we'll talk about three different technologies that are sometimes used to remove heat in the data center. The first is known as a rear-door heat exchanger, or RDHX. So, a rear-door heat exchanger replaces the standard back door of a server rack with a liquid-cooled coil system that removes heat from the exhaust air before it enters the room. A rear-door heat exchanger works a lot like the radiator in your car. So, think about a car for a moment. In a car, the engine generates a lot of heat, and coolant absorbs that heat and flows to the radiator, and then air passes over the radiator coils and carries that heat away, keeping the car from overheating. Now, in a data center, the process is essentially flipped. Servers generate heat, kind of like the engine in a car, and they push hot air out the back. And that hot air passes through a liquid-cooled coil in the rear door. That liquid absorbs the heat and carries it away. So, instead of air-cooling liquid like what you do in a car, liquid is cooling air. So, how does it work step-by-step? Air enters the server from the cold aisle. This is the same way that it's always worked. Then, internal server fans push that air across CPUs, GPUs, memory, and other components. And then, the hot air exits the back of the server. And, instead of just going out into the room, what happens is that that hot air goes into the rack's rear door. And the rear door contains a coil filled with circulating coolant. As the air passes through the coil, heat is transferred from the air to the liquid, and the air temperature drops significantly. The warmed liquid carries the heat away to the facility's cooling system. So, effectively, what's happening is that the rear door heat exchangers are acting as a capture point for heat at the rack level. And this changes the dynamics of the entire room. So, the hot aisles are dramatically cooler than they would otherwise be. And, in some cases, particularly in low-density environments, the hot aisle might not be any warmer than the cool aisle. Also, less heat escapes into the data hall, reducing the strain on the cooling units. Airflow requirements also drop, since heat is removed more effectively. And, in some cases, a well-designed rear door heat exchanger can remove the majority of a rack's heat load. Meaning, the room's air cooling system becomes supplementary, rather than the primary means of keeping this data systems cool. Now, when we're talking about rear door heat exchangers, these devices fall into two main designs, active and passive. So, a passive rear door heat exchanger doesn't have any fans that are built into the door. It relies entirely on server airflow, and it's simple and very efficient. It's best for moderate density systems. Now, when we talk about active rear door heat exchangers, these include fans that are built into the doors. And, they're designed to help pull air through the coils. They can handle higher heat loads, and they provide more consistent performance, regardless of server airflow. So, typically, rear door heat exchangers are going to be used in environments where existing systems need to be retrofit. So, in other words, an organization wants to bring in liquid cooling, but they want to retain their existing air-cooled systems and just retrofit them to use liquid cooling. This is a really simple and relatively low-cost way of bringing liquid cooling into the data center. It's minimally disruptive to existing IT equipment. The operations are familiar, and there's a relatively low risk compared to other forms of liquid cooling that bring liquid into the individual devices. The big limitation with rear door heat exchangers is that they still depend on air movement inside of the server. And, they're also less efficient than direct-to-chip for very high rack densities. So, with that said, let's go ahead and turn our attention to direct-to-chip cold plates. So, direct-to-chip cooling delivers liquid straight to the hottest components inside of a server. This might be the server's CPU, the GPU, maybe even memory. The idea is that there's a liquid-cooled cold plate that sits in direct contact with those components. And, this is very, very similar to the methods used in high-end gaming PCs. If you've got a PC like that with a liquid cooler, then you've already seen a version of this. In a gaming PC, there's a metal block, it's called a water block, that sits directly on the CPU or the GPU. And, liquid flows through that block, absorbing heat from the component. And then, that warmed liquid travels to a radiator, and fans pass air through the radiator where the heat is released. So, direct-to-chip cooling in a data center works on a very similar principle. It's just scaled up and engineered for reliability, serviceability, and continuous operations. So, how does this work step-by-step? Well, the idea is that cold plates are mounted directly onto CPUs, GPUs, or other hot components. And, then, coolant flows into the rack through a facility connection. It enters an in-racked manifold that distributes the liquid to each server. Then, there's a quick disconnect that allows servers to be serviced without draining the system. Liquid flows through each cold plate, absorbing heat directly from the chip's surface. Then, that liquid exits the server and returns to the rack loop. The liquid is routed to a coolant distribution unit, or CDU, and the CDU then transfers the heat to the facility water system, while isolating the IT coolant loop and regulating temperature flow and pressure. So, direct-to-chip is very effective. It greatly reduces the amount of heat released in the room. It allows for much higher rack densities without overwhelming air systems. And, fans still exist, but they do something a little bit different. Typically, fans are used to air-cool components that are under less of a load. This might be things like power supplies, or networking components, or just the residual heat coming off of the server. But, then, liquid cooling is used for the hotter components, the CPUs, the GPUs, the memory, and things like that. But, the majority of the heat, often 70-90% of the heat, is captured for liquid. And, this works really, really well for AI training clusters, GPU-dense environments, and high-performance computing. Now, the third type of liquid cooling that I want to talk about is immersion cooling. And, this is fundamentally different from the other two forms of liquid cooling that I've talked about. Because, instead of bringing the cooling to the server, it literally places the entire server directly into the liquid. Yes, the server is submerged in liquid. Now, we're not talking about water here. We're talking about a special liquid that has been designed to be dielectric, meaning that it doesn't conduct electricity. So, what this means is that your server components can operate safely while they're submerged in liquid. So, there are two main types of immersion cooling. There are single phase and two phase. So, single phase immersion is kind of like taking a hot pan and putting it into a sink full of water. The water absorbs the heat, and the water warms up gradually, but the pan cools very, very quickly, as soon as it's submerged into the water. Now, two phase immersion is kind of like boiling water on the stove. Heat causes the liquid inside of the pan to boil, and that's known as a phase change. And, as that boiling process occurs, that carries energy and heat away from the pan. With single phase immersion, servers are submerged in a bath of dielectric fluid. And, this fluid doesn't boil. Instead, what happens is that pumps circulate the warm fluid through the heat exchanger, and then the heat is transferred to facility water and carried away. So, what's happening physically is that the heat moves from the component to the surrounding liquid, and then ultimately into a cooling loop. So, two phase immersion works very differently. What happens is that the servers are still submerged in liquid, but this liquid is designed to boil at very low temperatures. So, the heat from the chips causes the liquid to boil, and the boiling liquid vaporizes. It turns into a steam, and then the steam comes into contact with a cooling plate that's just above the tank full of liquid. And, when that steam makes contact with the cooling, it condenses back into a liquid, and then the liquid returns back to the cooling tank. And, so that's what causes the heat to be removed. The heat is being removed by a phase change. In other words, that conversion from liquid to steam and back into liquid. So, the big advantages to this type of liquid cooling are minimal need for server fans, lower energy use, no need to move large volumes of air, uniform cooling across all of the components, and extremely efficient heat transfer. And, this means that you can achieve much higher densities of servers. It also reduces acoustic noise, because you don't need all those fans, and it reduces airflow management complexity. However, there are some limitations. For one thing, it requires an entirely new operational model. You can't take your existing rack-based systems, and then just dunk them in a tank of liquid. It doesn't work that way. You're going to need specialized hardware. So, there's going to be hardware compatibility and handling considerations. Also, you have to think about fluid management. And, you also have to think about long-term maintenance and planning, because remember, your servers are submerged under liquid, so they're not quite as easily accessible as they would have been before. There's also a higher initial cost to implement. So, that's something to think about as well. And, just on a side note, it's worth noting that immersion cooling is no longer just for cryptocurrency mining. So, while it gained early attention in that space, this type of cooling is now being piloted, tested, and increasingly adopted for enterprise, AI, and high-performance computing workloads, where density and efficiency are becoming critical constraints. So, the big picture is that liquid cooling isn't a single solution. It's a range of options. Rear-door heat exchangers extend the life of air-cooled environments, and they're comparatively inexpensive to implement. Direct-to-chip systems target the most intense heat sources, and immersion rethinks cooling entirely for maximum density and efficiency. So, the important thing to keep in mind is that liquid cooling is a spectrum, not a single technology, and it's becoming essential for the next generation of data centers. With that said, I'm Brian Posey. Thanks for watching.