Matt Langman on TechArena: Phononic Unlocking Data Center Performance

TRANSCRIPT

[Allison Klein]
Welcome into the arena. My name is Allison Klein, and today I have a really exciting interview with a good friend, someone I have known for a very long time. He’s SVP and GM of infrastructure at Phononic, Matt Langman is in the house.

How are you doing, Matt?

[Matt Langman]
I’m doing great. Great to see you, Allison.

[Allison Klein]
So, Matt, you know, I said it at the top, but you and I go back very far and we have a shared history in driving the foundations of technology for the data center. But why don’t you just start with an introduction of yourself and what you’re doing at Phononic?

[Matt Langman]
Yeah, thank you for that. And it’s great to cross paths again. I appreciate the time here at Tech Arena.

So yeah, a little bit about myself. I’ve been in the semiconductor microprocessor industry for around 30 years or so and had a wonderful opportunity to work in a range of different market segments of IoT and edge. And the last decade or so has been data center and AI.

Had amazing opportunities to work with cloud service providers and OEMs to build a range of world-class data center microprocessors. In the last year or so, it’s been exciting to join Phononic and work on thermal control solutions for data centers and hotspots in the data center.

[Allison Klein]
We’re going to get all into Phononic during this episode, but I just want to start with a broader question for you, which is, you know, you’ve been working in data centers for a really long time. We’ve never seen a moment like this. Can you put your take on what is happening in AI data centers and how should we as members of the value chain think about that?

[Matt Langman]
Yeah, it’s a very exciting time for many of us in the AI and data center industry. Just watching the insatiable hunger — like all of us have an appetite for AI services, right? It’s unleashing all sorts of new capabilities in terms of our day-to-day work and efficiencies, and all sorts of new capabilities along with that, of course, is just the continued build-out of AI data centers and the CapEx spend to meet that need for all those services and capabilities.

What we’re also seeing as part of that, and I think many others are seeing, is in order to meet those needs of those services, it’s more and more performance, which essentially means more and more power having to be delivered to the infrastructure. We know GPUs, CPUs, and a bunch of other devices in the data center, in order to really meet those levels of performance, meet those capabilities, are having more power delivered. And of course, that power has to go somewhere.

Right now, it’s heat. And there’s all sorts of interesting challenges that the industry is doing to address — how do we address the thermal pain points in the industry in data centers?

[Allison Klein]
That’s so interesting. I was at OCP Barcelona recently. I was doing a panel and one of our former colleagues, Zane Ball, was asked the question of where is the biggest challenge in data centers moving forward?

He’s CTO of OCP, so his opinion kind of matters. And, you know, he said it’s thermals, right?

It’s power and thermal delivery, and how do we manage that? Because that’s the choke point.

So I’m so glad that we’re having this conversation. You talk, within your conversations about your company, about hotspots multiplying in number and intensity. And I think that it’s really interesting to think about, because it’s kind of like chasing bottlenecks.

And, you know, we see you address one problem in a system and it unleashes awareness about others. When you say that heat’s now capping compute, what does that cost an operator in terms of real cost to an organization from a standpoint of performance and economics?

[Matt Langman]
Yeah, it’s a great observation, a great question. And what we’ve seen in terms of hotspots is whether it’s GPUs, CPUs, voltage regulators, DIMMs, SSDs, networking devices — there’s many, many hotspots, as they’re all really trying to optimize for performance and not be the bottleneck in those AI workloads.

And that ultimately is where we see operator pain points, whether it’s networking performance or GPU performance, where you start having to throttle back because you’re getting too hot. Ultimately, that means that you’re not fully recuperating the investments you’ve made to get that high-speed networking performing, to get those GPUs performing at those inference workloads that you’ve been investing in and optimizing your capabilities for. And so when you start having throttling in your processor devices, performance lags, or devices just start going into self-protection mode because you don’t want them to wear out over time.

And then ultimately, once again, it’s a performance issue, which then turns into a performance-per-dollar issue. Because the alternative is that operators now need to essentially over-provision their cooling. So you’re either sacrificing performance or you’re investing a lot of money to keep these devices cool.

Either way, there’s an opportunity we think the industry could do better.

[Allison Klein]
You know, it’s been for a long time that everybody thought that air cooling was the answer. And, you know, we could drive up ambient temperature and humidity allowed inside of a data center to accommodate. But we’re really pushing up against the limits of air and liquid cooling.

There’s a lot of talk about it, a lot of early deployments of it. And there’s conversations about direct-to-chip and immersion, and what’s the right solution. I think that you bring a really interesting perspective on this.

You talk about precision and bulk cooling. Walk us through that and tell me about your view in terms of how you see both from a chip and system level, cooling being optimized for this moment.

[Matt Langman]
Yeah. So where we see deployments these days is liquid cooling is becoming essentially the de facto standard. If you look at Computex that just happened a while ago, all the solutions on the floor are liquid-cooling based.

And what’s happening is, in order to meet those cooling demands to get those high-performance GPUs, CPUs, networking devices out there, you have to get to aggressive liquid temperatures to keep things cool. Our observation is that what’s happening is, because people are really addressing the worst pain points through bulk cooling, they’re over-provisioning. So, like you mentioned, targeting those hotspots with a solid-state, intelligent type of cooling capability allows you to target that hotspot and then use it on demand, so that you’re cooling when you need to.

And therefore, you can relax some of the concerns, such as your liquid temperature. You can raise your liquid temperature because now you don’t have to address the worst-case scenario through your bulk cooling. You now have a combination — kind of a chocolate-and-peanut-butter combination type of cooling solution.

And by doing that — raising liquid temperature and only cooling when you need it — we’re seeing significant ROI, like 3x ROI on your investment by doing it that way. We’re seeing PUE savings on the levels of up to 0.15 PUE, because you’ve increased — yeah, I mean, it’s huge, right?

You’ve increased your liquid temperature, so you’re not running your pumps and chillers as much as you have in the past. You’re also reducing throttling, so you’re getting a performance benefit and an energy savings benefit. We just believe that that’s a very attractive solution for data center operators.

[Allison Klein]
The other thing you talk about is the growth to 1.6T. I want you to break that down and talk about transceivers and thermoelectric coolers and why they’re important in this moment.

[Matt Langman]
Yeah, and you’re right — like I mentioned, nobody wants their networking bottlenecking their AI performance, of course. And we are seeing more and more deployments of 1.6T into networking. Through critical portions of those optic solutions, of course, are the lasers themselves, and you really can’t have any drift in laser capability, degradation in laser capability. And the predominant mechanism to ensure laser consistency is through thermal cooling. So because of the form factor constraints, because of the energy efficiency requirements, and the precision needed in cooling those lasers, thermoelectric solutions are an ideal solution to keep those lasers cool.

And we’ve been super excited to be the key high-volume manufacturing partner for cooling solutions into 1.6T. We have north of 40 million devices installed in these types of deployments, and it really does ensure that your lasers are performing accurately, delivering that high quality over time, and ultimately your networking gear supporting your overall AI performance.

[Allison Klein]
That’s really cool. And, you know, I think that one of the things that I think about with optics is that there are a lot of different solutions in the marketplace. And I guess, how does that extend into co-packaged optics?

And why does co-packaged optics become such a critical unlock for data center capacity in your mind?

[Matt Langman]
Yeah, it kind of goes back to the notion of needing to get higher and higher bandwidth, but also being energy efficient. And that’s what we’ve seen as far as the key enhancements that co-packaged optics is going to be able to deliver. It’s obviously a very significant architectural change in the industry, but it is in service to unleashing new levels of bandwidth while also being efficient.

Marvell came out just a couple of weeks ago commenting how, even within rack boundary conditions, the only thing that’s going to unblock that is co-packaged optics devices. So it is becoming a critical aspect for getting to that next level of performance. And it’s very similar —

You have lasers, and then you have an integrated optical engine plus ASIC. And those lasers need cooling. And we even see some compelling use cases for that integrated optical-engine-plus-ASIC area too.

[Allison Klein]
I love this, because it shows that there are multiple hotspots in a platform that you have to think about. But, you know, everybody’s thinking about the GPU — I mean, NVIDIA produces a lot of heat, let’s just be honest.

Why is that the bottleneck that everyone keeps hitting? I think that’s a pretty standard answer, like everybody’s looking for that GPU compute.

But can you walk us through why precise versus broad cooling works here on the GPU, and what the benefits are to the user?

[Matt Langman]
Yeah. So when you look at GPUs, or even the AI accelerators from cloud service providers, there’s the two main components — the processing cores themselves, and then, of course, the high-bandwidth memory stacks.

And what we’ve seen, through our own analysis and engagements throughout the industry, is especially with more and more inference workloads, agentic AI workloads, many of those are memory-bandwidth constrained. So what happens is those memory stacks start getting hotter and hotter. And right now, the way the industry is addressing it is through liquid cooling with liquid cold plates.

And we’ve done the analysis that says, well, by targeting the HBM stacks uniquely, we can combine that with intelligent firmware and software solutions. You can do some pretty innovative and creative things around extending lifecycle of your HBM, reducing throttling of HBM stacks as they get hotter and hotter, by looking at temperature signals, by looking at voltage and current signals that are being delivered to our devices. You can cool on demand, and you can then have this collaborative capability with your liquid cold plate so that you’re reducing throttling, extending lifecycle, but also being efficient and getting those PUE gains.

So you’re not affecting performance at all, and you really deliver that optimized solution.

[Allison Klein]
It’s kind of cool. So we’ve talked about co-packaged optics. We’ve talked about GPUs and the HBM that sits on top of that.

And it’s such an interesting design to think about, how we’ve stacked microprocessors — that’s just an aside that I’m thinking about while we’re talking. But there’s other areas — switch ASICs, CDUs — that all need cooling.

When you take all of that together, what do you think the cooling in data centers as a whole — what is the trajectory on that, and how are you going to tackle it?

[Matt Langman]
Yeah, no, it’s a great observation. And we see the world very similarly — that you do have these individual hotspots scattered throughout the data center. And kind of the way we’re looking at it is, by providing targeted cooling solutions that are then instrumented with firmware and software stacks, you could actually get into some pretty creative solution space in terms of overall optimization, because you can identify there’s a hotspot in this part of the data center.

There’s a different hotspot in this part of the data center. Okay, how do we do workload optimization — not only by workload placement, just typical job scheduling, but now you can get insight into your thermals and your actual heat that’s needing to be cooled.

So you can then do some interesting optimizations by looking specifically at the hottest spots of the data center, combining that with software and firmware stacks, raising that up through various industry standards like Redfish APIs, what have you. And now you’re not just doing cooling solutions, but you’re doing thermal control plus intelligence for overall workload optimization. It actually unleashes a bunch of new capabilities that we’re excited about.

[Allison Klein]
Yeah, and you know what’s interesting about what we’re talking about, with the integration of the CDU, you’re forming kind of an aligned management across traditional IT and OT equipment that goes into workload placement. So you’re managing power and cooling, you’re managing compute and storage and network, all in one unified stack — which makes people like me, that love telemetry and firmware, excited by the opportunity to expose all that data so that you can do that. How does that look in practice?

Where are we today, and where do we need to go?

[Matt Langman]
Yeah, great question. So clearly the industry has various telemetry solutions, right? The server industry, data center industry, has been doing that for many, many years, because getting insight into your devices is clearly critical.

You want to make sure you can understand, do you need to do a preventive maintenance event or a predictive maintenance event, what have you. We do see that the industry is not actually addressing, to the degree that we see — no pun intended — doing telemetry and workload optimization as a function of power and thermals. Clearly there’s a lot of optimization in standard workload job placement and job scheduling, but we do see that we’re at the beginning stages to be able to do even better, by connecting thermal and power data to your telemetry solutions and then actually having the full conversation.

It’s not just workloads, but now it’s heat. And how are you actually addressing load balancing and workload placement from that perspective? So from our perspective, we call that the Phononic Thermal Fabric, because it really does interweave all of those hotspots into one overall data center point of view, which will give better performance, better efficiency, and just better utilization overall.

[Allison Klein]
And your answer just completely redefined the value proposition of how I view your company, because what I was thinking about is cooling components and managing acute heat dissipation. But what you’re talking about is something completely different, which is integration into a cloud stack to manage workload placement based on that thermal data that’s being gathered in real time. That’s amazing.

Very cool. And, you know, I think very timely in terms of what I’m hearing from hyperscalers and neoclouds about how they want to manage their workloads. I think that we’ve made a lot of inroads in managing compute from that capability, but it really opens up —

You really need to manage all of the data center infrastructure. You need to manage every element of what’s running into that data center. And your products are a way to collect that data, for sure.

When you point to predictive maintenance — knowing a component is drifting before it fails — this is something that every data center manager wants to know, because they don’t want to replace something once it’s broken. They want to address it proactively.

How does that change the economics when you’re able to provide that predictive analysis?

[Matt Langman]
Yeah, it fundamentally changes it, because now, instead of dealing reactively and doing break-fix type of repairs, you are now able to look at different signals. So in our case, what we’re able to do is tap into those voltage, current, and temperature signals, and therefore you can actually look over time and do a snapshot in time and say, you know what, for XYZ workloads, XYZ inference workloads, we used to operate in terms of currents of A, B, and C. Well, it’s X amount of hours of deployment later, and those currents have changed to get the same cooling on the same workload.

Did something happen, right? And now you can actually choose to make a decision about how you want to address that ahead of time, and build it into your normally scheduled activities, as opposed to responding reactively. And we all know that uptime is a critical aspect for hyperscalers — any disruption to services or unscheduled work.

It’s just more cost. So being able to have access to that type of information, to be more capable in scheduling your maintenance events, is just a huge impact opportunity for hyperscalers and operators.

[Allison Klein]
You know, it’s so funny — all the things that we’ve been talking about today hearken to how long it sometimes takes from vision to implementation in the data center. You know, you and I marketed things like rack-scale architectures — that AMD is bringing to life with Helios, for example — many, many moons ago.

We talked about the telemetry and control of software-defined infrastructure and composable infrastructure for a really long time. And I guess my last question is this: we’ve got this great vision for management of thermals. As you look at the future over the next several years, how quickly do you think this vision will become a reality?

And are we speeding up because of AI data centers — the speed of tech transformation from vision to mass deployment?

[Matt Langman]
Yeah, I absolutely see us speeding up, and we’re seeing that in just even as simple as we’ve moved from 20-kilowatt racks to 100-kilowatt racks, going to megawatt racks, where it’s just all of this need to get to higher and higher power for more and more performance, because people are wanting to have better response times on their various AI workload services. You can’t just deploy all that heat and power and infrastructure without wanting to get an OpEx return on your investment as well. It’s not just the services, but more and more, the conversations we’re in these days are that balance of performance plus OpEx.

So instrumenting your hotspots, reading that data, and preventing throttling, extending useful life, are all super important — but the OpEx costs are becoming a bigger and bigger need. So we are seeing that momentum to instrument devices, use the telemetry, look at power and thermal data combined with other signals, because maintaining OpEx is just as important these days as the actual performance. So we do see that accelerating as a key concern for folks.

[Allison Klein]
Matt, it’s been such a pleasure. I knew that I wanted to have you on the show and you did not disappoint. And I’m sure that my Tech Arena audience is going to be Googling Phononic to find out more.

But if they want to engage with you and your team, where would you send them for information beyond your website?

[Matt Langman]
Yeah, absolutely. So you can reach out — we have our LinkedIn page. Of course, you can reach out to me directly.

We have lots of great information on Phononic.com — some nice videos and blogs and technical papers that go into a little bit more depth on the things we talked about here, Allison. And of course, we’re going to be at various industry events later on in the year — AI Infrastructure Summit, SC.

We’ll be able to talk to many of your viewers in person at those events. We look forward to it.

[Allison Klein]
Awesome. Thank you so much for your time today. It was so fun.

[Matt Langman]
Thank you so much, Allison. I really enjoyed it.