A New Era of Data Center Cooling

VIDEO TRANSCRIPT

[Phil Alsop]
Okay, Phononic, you are positioning thermal management as a serious limiting factor for AI performance. So just how close are today’s hyperscale data centers to hitting that thermal ceiling, and what does that actually mean in practical terms for operators? What’s the impact of that?

[Ryan Parker]
For several of the workloads now, we’re not just approaching the thermal ceiling — I think we’re already there. Training is a workload that would specifically be in line for that. We see the next shift as precision, rather than over-cooling for a few hot spots. It makes a lot more sense to directly cool those hot spots, and let the bulk cooling run at a lower sustained level as well.

[Phil Alsop]
And I believe in terms of the technology solution you’re offering, you’re citing, I think it’s 15% potential PUE savings and three times ROI. And what you’re doing is GPU, high bandwidth memory cooling solutions. So to your point there, very targeted cooling. Just tell us a bit more about the solution, and particularly how, therefore, where those gains are coming from and how quickly operators can realize them.

[Ryan Parker]
What’s different than standard cooling is we can respond in milliseconds, and that gives us about a 600 times control gap versus bulk cooling. And what that allows us to do is dial in those various things in a software-defined manner. PUE savings actually come because we can run the cooling water 10 to 15 degrees C warmer than you could do otherwise. That gives you the PUE savings right there. We’re already seeing, on our next-generation parts, about a 14% overall power savings at the data center level by switching to our solutions, just from running the water warmer.

Then, also, when we do that we can set the temperature that the device runs at to stay below all of the throttling thresholds. We call this margin compression — memories refreshing multiple times, you’re starting to slow-clock, and eventually you’re in full throttle. That’s lost performance that people don’t know they have, and we can unlock all of that by never actually running into those thermal excursions at all. So that unlocks a lot of performance, and you actually can fully utilize the asset that you paid for. By running it cooler you also get another benefit, where you can make the device last up to 5.5 times longer, because the Arrhenius effect and thermal cycling are what’s happening to you — you’re keeping those temperatures constant.

So we actually see our solution, especially for HBMs, as a really compelling case, and now we’re starting to proliferate that to co-packaged optics and other parts of the data center. Eventually our vision is anywhere there’s a fan or a hot spot, you could replace it with a thermoelectric with a millisecond response time, to give you precise control — and that’s especially important as racks intensify.

[Phil Alsop]
Yes, I mean, I was going to ask that — you potentially answered most of the question, but I’ll ask anyway. As the industry is rapidly moving towards high-density workloads and therefore higher rack power required, is your approach in terms of how it differs just the targeting — the targeting of the liquid, or of your cooling, as opposed to traditional liquid or air cooling, which tends to blast most of, whether it’s the whole data center or the whole rack? Just what do you do differently to handle those escalating thermal loads?

[Ryan Parker]
Yeah, so our thermoelectric solution deploys exactly on each hot spot. That could be an HBM memory, it might be a voltage regulator, it could be a lot of different parts within the rack — and then we let the bulk cooling do what it’s great at, which is getting all of that heat out of the rack, and let us take care of making sure all the temperature irregularities are controlled so that bulk cooling can run most optimally for itself. So we see it as very complementary with bulk cooling — our solution, anywhere there’s a hot spot, we can go in and address it, and with that software control, we’re only adding power from our solution when you need it to remove the heat, otherwise we can turn it off.

[Phil Alsop]
Okay, and I know that the Gen 2 GPU HBM cooling can deliver up to 75%, I think it is, higher heat dissipation. So just translating that to real-world impact on AI workloads, in terms of sustained performance and avoiding thermal throttling as an issue — what is the difference this makes?

[Ryan Parker]
We’ve just gotten data back in the last week that’s absolutely stunning to me — I’ve never seen anything like it in my 30 years doing this. We talked about those three variables of PUE, performance unlock, and device life. With this newest part for HBM4, we’re actually seeing that you can have all three of those simultaneously — you don’t actually have to trade off. And that’s a massive impact for the data center.

[Phil Alsop]
And more generally, you’ve introduced the concept of a thermal fabric. So just explain to me how data center operators should think about this compared to, whether it’s DCIM or workload orchestration tools — are you replacing those, or is it an additional layer? Just tell us the idea behind it.

[Ryan Parker]
Yeah, we stick to our swim lane on this one. We see those as very complementary, just like bulk cooling — you could kind of think of us as a layer in the middle between the two. On the orchestration side, it’s currently void of any thermal signaling, so you’re placing workloads without actually knowing if thermally you can do that. So this becomes the new fabric alongside compute and networks. Now you can have a thermal fabric and get a lot more precision and a lot more determinism in how you’re scheduling workloads. On the DCIM side, we talked about the 600x lag — it takes about four to six minutes for those loops to react. We can react in milliseconds, so you actually completely avoid thermal excursions, and you’re not constantly over-provisioning the water temperature. So that signaling layer to each of those is incredibly valuable, we think.

[Phil Alsop]
And you mentioned co-packaged optics earlier, which is — I won’t say it’s a pet subject of mine, but it’s a fascinating topic, it’s an emerging architecture. So what role do you think your cooling can play in accelerating its adoption, and are there any particular challenges you see remaining when it comes to scaling up the technology for production environments?

[Ryan Parker]
I think there’s a couple different issues with co-packaged optics that are emerging. The first one is, how do you keep the laser stable — and that’s what our thermoelectrics do. The laser actually sits on top of our thermoelectric, and with that millisecond response, you keep it stable. We’re already doing that all the way from 400 gig to 1.6T, and even have 3.2T designs underway. That’s important, but what also happens is now, when you bring it next to an ASIC, you get parasitics from that, and so it’s even more important that you keep the laser very precise, because that extra heat coming from there — and then you take that device and pull it closer to the GPU, and you start getting the GPU parasitics as well. And so we think our approach of targeting the hot spots and cooling them if you need it lets you get the laser stable without spending any more money. But we can also start to look at what the ASIC needs for cooling, and then we already talked about the HBM, and even the GPU die itself.

What also comes out of that, though, is we can see if you’re having issues with the laser and give you a signal before it fails — it’s taking more power over time to keep the same temperature of the laser, and that starts to become a predictive failure mechanism that we can see in the device. And I think everyone’s fear about co-packaged optics is, how do I know when it’s going to fail? Can I predict that so that I could actually be smarter with my orchestration and not have a big outage?

[Phil Alsop]
That makes sense, and I think you did already answer this one — but you’re working, I think, with most if not all of the major hyperscalers, and I was going to ask, you know, customers prioritizing energy efficiency, performance, and hardware longevity — normally there are trade-offs, but I think you just told me the results are suggesting that they no longer need to worry about those trade-offs, because they can improve all of those things equally well, or at least significantly. Is that right?

[Ryan Parker]
Yes, and this is a bit of a choose-your-own-adventure that we’ve found, depending on who you talk to. Some people are very aggressive on performance — in my previous days at Intel we used to unlock parts for gamers as an example — it would be possible for GPU vendors to create unlocked parts that could run even higher, so you could flex in that direction for even more performance. If you were more interested in making your amortization table meet your actual device life, you can actually run it even colder still. Or what I think probably will also happen is, hey, it’s great that we can run the water temperature 10 degrees warmer — what about 15 or 20, what would that look like? And can we save even more energy, and repurpose that energy for even more compute?

[Phil Alsop]
And maybe just in finishing — I guess there are various constraints or challenges in and around the data center, but thermal physics would seem to be a key one. So with that in mind, what do you think the next generation of AI data center design might look like, or is already looking like? And to what extent will the evolution be driven by cooling innovation versus advances in the silicon, which almost daily we seem to find somebody doing something incredible with the latest iteration of their chips?

[Ryan Parker]
Yeah, it’s funny — for most of my career it was, how do we make silicon better, and then networking and storage would have to react to that as well, but cooling would just do more of it, and it was kind of a great cycle for decades. Now we’re at the place where both power and cooling become primary design constraints, and as densification happens, it only gets worse. So I think the industry shift is really around how do you think of thermal as a constraint, and how do you pair hot-spot cooling with that constraint. I’ll give you an example of some of the discussions we’re having: on a GPU die, HBM is a problem and people are already talking about how they can unlock that, but there are other spots on there too, whether it’s a matrix multiply cache or other places that are getting hot. We can see a world where you’re actually just putting thermoelectrics on each of those hot spots and running them when you need to cool. And I think that mindset shift is pretty significant for people, and it’s a way people are going to actually gain the next layer of differentiation and performance.

[Phil Alsop]
And just your thoughts — I know another potential solution, I’m not saying there isn’t a need for cooling at the component level, but the idea of dropping the whole IT hardware system in a bath of liquid, submersion cooling — is that got a role to play, do you think? Or do you think this very targeted, very efficient cooling that you and others have developed is the way to go? Or can you see the attraction of just dropping everything into one bath of liquid?

[Ryan Parker]
Yeah, I think to me there’s kind of two things. We could extend the current generation with our solution plus bulk cooling, and then as the next generation comes along, you still have hot spots, so you still need some kind of targeted cooling as well. So whether it’s microfluidics or immersion or any of these things, we see those as still complementary to what we’re doing, from our perspective. You can’t get the millisecond-level response time out of immersion, but you can get rid of a lot more heat, which is great — that only makes our solution better, because you have to run it less, add less power, and it makes the whole solution more efficient.

[Phil Alsop]
That’s great. It’s been brilliant to chat, and thanks for sharing so many great insights into what’s going on. I’ve been involved in data centers for 20-odd years, and as you said, you’ve never seen results like that — I’ve never seen a time like it either, in terms of what’s going on. So it’s been fascinating to chat. Thanks for sharing some of these great insights. Appreciate you.

[Ryan Parker]
Of course. Yeah, appreciate the time.