Phononic’s Thermal Fabric Addresses AI-Driven Data Center Challenges

TRANSCRIPT

[Phil Alsop]
Okay, if we can start with something you say as a company, that thermal physics now sets the performance ceiling when it comes to AI infrastructure, or for AI infrastructure. From a semiconductor perspective, how close are today’s GPUs, HBMs and photonic devices to their thermal limits?

[Matt Langman]
Yeah, great question, Phil. Well, what we’re seeing in the industry, and many are seeing in the industry right now, is just the ongoing need for more and more performance. We’re seeing racks move from tens of kilowatts to hundreds of kilowatts to, in the future, one megawatt racks, right?

So more and more power being delivered to provide that performance for all the AI services that many of us are using these days. What we see is as that power gets delivered to the rack and then fans out to all the individual devices, it’s in service to higher and higher powers for GPUs, CPUs, higher performance bandwidth for networking. And with that, what we see is, especially in the case of, say, GPUs, right?

We’re seeing those go from hundreds of watts to kilowatt power ranges, and it’s all in service to that higher performance. And what happens is those GPUs are designed for optimal operating points, right? Optimal temperatures, optimal powers.

But there are use cases and conditions where if you’re not cooling properly, you could start bumping up against those higher temperatures. And what happens is semiconductor devices, they start to protect themselves, right? So they either downclock in frequency or they go into double refresh mode, all in service to not failing, right?

And still providing some level of service. And that’s what we’re seeing today — in order to solve that problem, the industry is essentially over-provisioning, providing great solutions in terms of air and liquid cooling. But what we’re seeing is systems are needing to be over-provisioned to compensate for those scenarios.

And for Phononic, we see the opportunity to really target those hotspots to provide an optimized, intelligent solution.

[Phil Alsop]
OK, and as a company, you’re known for your solid-state cooling technology. So please tell us a little bit as to how this differs from conventional compressor-based or liquid cooling systems, in terms of the integration with semiconductor devices and power electronics.

[Matt Langman]
Yeah, I would invite Brooks to talk a little bit about our thermoelectric capabilities and our technologies. And then we’d also like to talk a little bit about how we see the evolution, not only from the thermoelectric perspective, but looking at controls and firmware.

[Brooks Henderson]
Yeah, great question. So thermoelectric technology, and our thermoelectric technology specifically, is capable of both heating and cooling. So in that way, that’s one way it’s different from traditional, bulk cooling technology, right — many of these semiconductor devices within the data center have an optimum temperature range.

And so we’re able to maintain that temperature range, both on the cooling side and the heating side, with our technology. Additionally, because we are providing a semiconductor device ourselves, it’s capable of doing this — we have inputs and telemetry that providers of traditional bulk cooling are not capable of getting.

So we can get insights, we can track the performance of our semiconductor devices and the core compute, memory, and optics, and leverage that for further insights to our customers.

[Phil Alsop]
OK, and your next-gen GPU HBM cooling solution targets, I think it’s a 75 percent increase when it comes to heat dissipation. So what are the key thermal challenges emerging from HBM4 and advanced, I suppose, 2.5D/3D packaging? And how does your approach address them at the package level?

[Matt Langman]
Yeah, so as you mentioned, we’re currently sampling and shipping our second-gen GPU HBM thermal cooling solutions. As we looked at the JEDEC roadmap and the industry and HBM suppliers’ trends for improving performance of HBM, all in service to those AI workloads — similarly as in GPUs — we see, in general, TDPs for HBM increasing as well. Our first-gen cooling solution targeted around the 20 to 30 watt per square centimeter range in support of HBM3.

And now we’re targeting upwards of 50, 60, 70 watts of cooling for HBM4. And this becomes critical, especially as the industry moves to more and more inference-type workloads. We see those types of workloads really needing more and more memory bandwidth and memory performance, which is, of course, in service to how the industry is moving from HBM3 to HBM4 to HBM5 to provide those levels of service.

And what we’re targeting is, just like Brooks mentioned, we have our innovative solid-state cooling devices, which are utilized on demand. And then, when coupled with the firmware that Brooks mentioned, we can read into telemetry signals, we can read temperatures, and we can adapt the cooling needs for HBM based off of your use case. If you want to hit a certain particular level of temperature and optimize based around a temperature range, we can do that.

If you want to maximize, say, the reliability and useful life, we can optimize for that type of scenario. And if you want to optimize for energy efficiency, we can use the telemetry and firmware signals so that we’re only cooling when you need it, right?

And we’re really providing not only the increase in performance by reducing throttling, which is what happens when these devices get to higher and higher temperature, but then we can also do it in ways that actually optimize the efficiency as well.

[Phil Alsop]
OK, maybe just expanding on that — I mean, we’ve got claims of up to 40 percent greater compute performance and reduced thermal throttling. So to your point there, maybe expand on how directly does thermal management now influence effective silicon utilization and the performance per watt, which I guess is what folks are talking about.

[Matt Langman]
Yeah. So as we’ve done engagements with people across the industry and looked at industry papers, what we’ve seen is when you get to higher and higher temperature regimes with HBM, you start going into protective modes like double refresh, because you need to throttle down and protect the device.

It’s kind of your standard Arrhenius equation — higher temperatures cause wear-out, higher temperatures cause failures. So devices, when they hit those higher temperatures, they need to protect themselves and they throttle down.

So what does that mean, ultimately, for AI workloads? It’s worse performance. So by providing an intelligent, precise cooling solution that targets that HBM directly, we can then optimize cooling based off of those various use cases.

So you reduce your throttling. At the same time, what we’re able to do is look at the system’s level, and now that we’ve identified the hotspot within the AI server, you can do some creative things, such as increasing your liquid temperature, because you’re not over-provisioning.

And then that increase in liquid temperature allows you dramatic PUE savings as well — upwards of 0.15 PUE savings, which is kind of on the order of when we went from air to liquid cooling. So you’re reducing both the throttling, getting better performance, and you have the opportunity to improve your energy efficiency and PUE, all by targeting the hotspot.

[Phil Alsop]
And one other development that’s impacting the industry is moving towards co-packaged optics. So it’d be good to understand how critical is co-design of thermal management with photonics and high-speed electrical interfaces, I suppose particularly at 1.6T and beyond. Just your thoughts there.

[Brooks Henderson]
Yeah, absolutely critical. Everyone’s familiar with Moore’s Law — electronics have been scaling performance reliably for decades. Well, there’s no such thing as Moore’s Law in optics.

So the industry has developed and is deploying these novel approaches to packaging, such as co-packaged optics, so that the optical networking can keep up with the bandwidth requirements of AI training and inferencing that Matt was talking about. These architectures have very real thermal considerations. Sensitive optical components that used to be packaged away in pluggable transceivers, sitting on the faceplate of the switch boxes, are now coming inside of the box.

The lasers that are remaining outside of the box are demanding and drawing higher power and heat loads than ever before. And so both of these applications — the lasers, and the optics that are going on the board inside of the box — require precise cooling. Our platform, we’ve been co-designing with customers in this industry for many years and have tens of millions of our devices in the field today with laser cooling in data centers.

So we’re excited about where the co-packaging trend is going.

[Phil Alsop]
Yeah, and I guess to follow on from that, from the power electronics standpoint, how do engineers need to think about cooling perhaps differently? So maybe as part of the power delivery and efficiency equation rather than just an external system — is that what’s occurring now, or needs to occur?

[Matt Langman]
Yeah, absolutely, you’re spot on, Phil. As we engage with many types of power engineers across the industry and the ecosystem, a lot of the conversations we’re having are around people having, say, a fixed power budget within certain design attributes or boundary conditions.

And when we start discussing what’s possible with our solution — of precise cooling, millisecond response time, intelligent and predictive cooling — it unlocks all sorts of new opportunities for these power engineers, where they can optimize for performance, optimize for a temperature. But they’re doing it in a way that’s super responsive and also saves energy, ultimately, overall. And then they can give that energy back — either just not use it and have an OpEx savings, or they could give that energy back to other parts of their infrastructure and continue to maximize overall performance of the system.

So we absolutely encourage and collaborate with power engineers to think not only about the one thing that they’re challenged with and trying to solve for, but then think more broadly, because as you have a more precise, super-fast-response, intelligent cooling, new use cases and new opportunities start to unlock themselves.

[Phil Alsop]
OK, and the thermal fabric, as I understand, introduces millisecond-level two-way thermal telemetry. So how does this level of control interact with chip-level power management, DVFS, or system-level workload scheduling?

[Matt Langman]
Yeah, great question. As we looked at our solution, we call it our thermal kits. So we have, as we mentioned, thermal kits for GPUs. We’re coming out with thermal kits for co-packaged optics. And these kits are that combination of solid-state thermoelectric cooling plus localized firmware.

And the way those interact is the firmware interacts with the local controllers, reads the telemetry signals of temperature, voltage, and then connects back to the overall system. In the case of, say, GPUs, we’re providing some controls capability, but then eventually we would look to just interact directly with the BMC devices right on the board, right in the system — make it a seamless integration. And then, ultimately, those would connect to Redfish APIs.

And this is where we get to the thermal fabric. Thank you for bringing that up, because what we see as the opportunity is once we have these different devices instrumented with telemetry and firmware, connected to Redfish APIs, you can then unleash new levels of optimization fleet-wide for your data center — not just on virtualization and workload placement, but now as a function of power. So you get even more levels of energy efficiency, power savings, and performance.

[Phil Alsop]
OK, and you talked about, I think, the 3x ROI and significant — you mentioned earlier — but significant PUE gains. So how much of that value is ultimately rooted in improvements at the semiconductor and package level, versus system-level infrastructure changes?

[Matt Langman]
Yeah, and it actually is a combination of both. So what we see is by reducing throttling for those HBM devices — and by the way, our technology is flexible enough that if there’s hotspots on the GPUs themselves or CPUs, the same type of methodology applies. You reduce throttling and you maximize performance. So right away, when we look at the way hyperscalers are looking at ROI and OpEx costs, performance is obviously a key metric, right?

So the more performance over time you get, the better workloads, the better response times, more tokens you can deliver and utilize per system or node — huge ROI from there, clearly, as the industry is embracing more monetization via tokens and token use cases. But then, as we also alluded to, now that we’ve really gone after those hotspots, you can unleash things such as higher liquid temperature, which then gives you that PUE savings because you’re not over-provisioning, you’re not spending as much energy and time getting hyper-aggressive or super-aggressive liquid temps.

You can really balance between the different zones and provide an optimized solution.

[Phil Alsop]
Maybe just in finishing, to look ahead, as I always like to do — do you see thermal management becoming a first-class design parameter alongside power and performance in chip and system design? And if that is the case, how is Phononic positioning itself in that co-optimization trend?

[Matt Langman]
Yeah, well, we’re seeing it more and more, for sure, as the industry does move to hundreds-of-kilowatt racks and getting towards one-megawatt racks. And as Brooks mentioned, as networking performance continues to be an optimization point for AI workloads, no one wants their GPUs or their CPUs and their agentic AI workloads bottlenecked by their network performance. So, of course, that’s driving more and more performance.

We absolutely see things like thermal control for lasers being a critical aspect that the industry is rallying around. And then, of course, being mindful of power and power densities — whether it’s higher and higher performance GPUs, or the roles of other hotspots in the data center, power supplies, voltage regulators, other memory — we see, as we look over time, there’s going to be more and more hotspots.

And it’s not just going to be localized — you’ll see hotspots throughout the rack, and then across racks. And that’s where we see our solutions of having fine-grained, millisecond-response thermal control capabilities that are instrumented and connected, to really unleash those new capabilities in terms of not only cooling and optimization, but also fleet-wide improvements.

So we absolutely see it needing to be a first-class part of the conversation, and we’re seeing the industry do that.

[Phil Alsop]
It’s been great to chat to the pair of you, and thank you so much for sharing some great insights into, obviously, the work that you’re doing at Phononic, but also your thoughts as to how the industry is progressing in this sort of AI-fueled rapid expansion. So Matt and Brooks, I really appreciate your time. Thank you.

[Matt Langman]
Thank you, Phil.