AI vs Energy: Can Smarter Chips and Local Clouds Save the Planet?
Autria Godfrey and Stephen Horn look at the energy consumption of AI and whether more efficient hardware and software can bring it down. Shreyas Sen of Purdue University describes chips, modelled on the human nervous system, that use the body to connect wearable devices at very low power. Hongyin Luo, a research scientist at MIT CSAIL and part of the startup BitEnergy AI, explains L-Mul, a technique that replaces floating point multiplication with integer addition.
Transcript
Welcome to Agents of Tech, the podcast that delves into groundbreaking innovations and ideas shaping our future. I'm Autria Godfrey and I am thrilled to have you with us today.
And I'm Stephen Horn. And today we'll be looking at the energy consumption of AI models, an increasingly urgent concern, both in terms of how we power these massive systems and how we finance the immense compute they require. The release of DeepSeek has added more fuel to the debate about AI and energy. Reports claim its final training run costs $6 million, while the company touts a far more energy efficient approach, this claim remains contested. So attention has focused on its mixture-of-experts technique, in which only the relevant portion of the model's billions of parameters is activated during training, potentially curbing the massive energy demands we typically associate with AI.
Achieving higher energy efficiency in AI doesn't just help the planet, it also makes AI more affordable and broadly accessible. As with the historical shift from mainframes to personal computers, lowering costs can open the door for more people to benefit from these technologies.
Coming up today, we speak to Dr. Hongyin Luo, a research scientist at MIT CSAIL, and part of BitEnergy AI. That's a startup developing a technique that could drastically reduce AI energy consumption without significantly sacrificing accuracy or speed.
We avoid very, precise math calculation. We just, make some mathematics precise enough for putting information back. So by doing that, we eliminated a large amount of unnecessary computation.
We'll also be speaking to Shreyas Sen, an Elmore Associate Professor of Electrical and Computer Engineering at Purdue University. Inspired by the human body, he designs chips that could make wearable AI more energy efficient.
Because the human body is conductive, you can create fields around you that give you this broadband nervous system like behavior through which you can send huge amount of data, including video, enabling AI to connect to the wearer's ultra low power wearables at a fraction of the energy compared to wireless communication.
So if this interests you, please make sure to like and subscribe to our show. It really helps us to shine a light on innovation.
I'm absolutely delighted to be joined by Laila. Hello, Laila.
Lovely to be here, Stephen, once again.
And you are, of course, Director of Research and Strategy at WebsEdge. Okay, Laila, so energy consumption and AI, so all we're getting in the news is bigger and bigger and bigger systems. More and more and more and more billions and billions and billions and billions. How does that leave us with energy consumption?
I think the thing with AI is with all technology historically, that it just does get more efficient to make it more accessible and that's just generally where the trend line goes. Whether we get there from a hardware perspective or a leap or breakthrough, which is from a software perspective, I don't know, we'll kind of find that out in the episode, or maybe it will be a combination of it. It's just going to get cheaper and cheaper and more accessible and more efficient. That's just inevitably where it's trending towards.
We're getting a war of announcements right now, aren't we? Where we're getting DeepSeek et al. sort of saying we're doing it on much less energy. And yet we're having more and more announcements from, you know, big technology companies that are saying they're going to spend more and more and more. They're going to have bigger and bigger and bigger data centers. So where do you think this is heading?
I think it's heading towards a point where no one can afford to be second place. And that's why they're just putting more and more behind it. Whoever wins the race is kind of the winner of the world. So I think that that's what's happening. There's not really small players left in it. And if there are any small players they're going to have to make huge leaps and huge breakthroughs that allow them to keep up in this contest.
So if we got a bigger and bigger race, it's getting faster and faster, faster to find the winner. Then how does that help us to conserve energy?
That's the million dollar question, right?
400 million.
Yeah. Or, 400 billion. It just gets bigger and bigger. I mean, if I had the answer to that, maybe I'd be sat in Paris or Washington.
Dr. Shreyas Sen, Elmore Associate Professor of Electrical and Computer Engineering at Purdue University, is taking lessons from the human body's nervous system to design chips that might enable AI to work offline for a network of wearable devices without the need to frequently charge them.
Fascinating. So Dr. Sen, I want to get started with how are you designing these custom chips for wearable devices to process complex AI tasks more efficiently, using less power, while they still mimic the way that our nerves and brains work together?
So what we do is build the chips such that ultra low power wearables can connect to each processing unit where AI will run like, a wearable brain. And that connects all wearables with ultra low energy communication. So this is the network that our chips enable. And this is similar to the network that humans have. We have a central processing unit, the brain. And all your sensors and actuators are connected through the nervous system.
So what gave you the inspiration in the first place to look at the human nervous system as a model for chip design?
My journey and background took me through, wireline communication. I was one of the ten people working on the first USB-C prototype out of Intel Labs. And then in my PhD, I had done wireless. So my journey in wireless and wireline communication taught me that when you try to communicate over the air, there is a thousand times extra energy that you need compared to when you are communicating using conductive mediums like nerves. And then when you look at today's wearables, we see that every one of them has a central processing unit, whereas if you look at a human, you just have one central processing unit and your image sensor and actuator does not have a CPU. So this dichotomy why biology is built this way versus today's wearables are built a different way, led me to question the architecture.
Very interesting. And are there any I mean, you talk about on the macro level that you took inspiration from kind of a central processing unit of of the human central nervous system. Are there any other specific aspects of the central nervous system that you are also applying, to your work, which you may not have applied yet, but you are interested by on a micro level?
As I was mentioning, communicating by radiating over the air means that your energy goes, you know, kind of isotropically in all directions, and you are only picking up a fraction of that energy, making your communication very energy inefficient. Now, what does nerves do? They do not take our biophysical signals and radiate it. They have a conductive medium through which you can send tiny electrical pulses that goes through this high bandwidth channel, that is the nervous system to the central processing unit, making it extremely energy efficient. So we started asking that around the human body, can we make this wire-like system? And turns out because the human body is conductive, you can create fields around you that give you this broadband nervous system like behavior through which you can send huge amount of data, including video, enabling AI to connect to the wearer's ultra low power wearables at a fraction of the energy compared to wireless communication.
So kind of on that point, given that models like ChatGPT, we know they don't demand huge data centers and a lot of energy in order to power them. How do these types of chips reduce the need for all of that energy? And that need for centralized cloud computing.
So to understand this in depth, we know we need to break down the systems and where the compute is happening. Today's ChatGPT is happening in a cloud. These clouds are megawatt-level systems. In future we are talking about tiny models. And, you know, some people are already trying to put some smaller large language models onto phones of today. Or in future, more capable compute devices around our body will run this on device AI, so that we call the edge device. Imagine it's like a wearable brain, but you are only going to have one of those.
You are not going to have these multiple wearable brains around your body. So you have the cloud. You have the edge. But to really make ultra low power sensors and actuators and other wearable devices perpetual, you need to go to another layer of the network, which are called leaf devices. So the cloud is megawatt, the edge device, the phone is about a watt scale platform. And these leaf devices need to be microwatt to milliwatt scale platforms. Some of the devices we wear today are going towards that, but it's not there yet. What will happen is you will have a smaller network running on that edge, wearable brain device and all of the data coming. So from today's ChatGPT a behemoth in the cloud. This is how AI is going to come towards humans in a distributed manner around humans.
So what has to happen to make that happen? Models have to be smaller, which people are trying, you know, lower precision models, more small models. Instead of training one thing for everything, they're talking about smaller models. Right. So this is one direction. So more model AI development will happen such that you can run it on the edge device. Edge devices computing power will increase so that you can keep on learning larger models. The models are becoming smaller, edge devices are becoming more capable. But you are not going to go from megawatt to microwatt. So along with that, what has to happen? These small devices around your body, which is not connected to the edge device need ultra low power communication around humans that is like a nerve, such that your brain gets access to all the data.
Wow, and you're developing brain implant concepts that could one day let us control devices with our thoughts. How far are we from that, and what are the challenges?
The nervous system inspired communication around the body is much more advanced. It is getting commercialized now. The brain one is more in the science stage, where we have demonstrated the physics that using the brain medium itself, you can communicate from small neural nodes that are inside your brain directly to a headphone that is outside, or other device that is outside. You know, what's the state of the art here? For example, if you look at Neuralink, they open the skull, put something inside your skull so that you don't have to take the skull absorption and then have it wired from there to go to the endpoints. What our physics is showing using the brain's communication medium as the conductor, you do not need tether the wires, and you don't need the opening in the skull. You can have free floating nodes inside the brain that can communicate directly to a wearable outside. That's the Nature Electronics paper and the progress that we have made.
Now, these free floating small nodes inside your brain that interacts with your brain to capture your thought. There is a lot of medical implications to it. So this is not ready for primetime. We have only looked into the one aspect of it, but many other the whole neuroscience community. And, you know, companies like Neuralink and many others are looking into it. We provide the ability to do untethered communication from implants to wearables.
I would imagine with this type of innovation, you have certainly come across some surprising technical, possibly regulatory challenges, maybe cultural challenges as well. I'd be interested in how you've come across those and what you've done to overcome them.
You know, the first question I often get is, oh, you are talking about electrical signals around humans. Are you electrocuting me? And it's absolutely the opposite of that. So the first thing we do is something called safety study. We have safety study papers where we analyze the effect, the amount of field that is getting created around the humans to do this. And you can see that those fields are 100-1000 times lower compared to, depending on the scenario, but compared to the safety limits established by ICNIRP, FDA, FCC.
How does that work? Imagine you have watches, right? The watch has a microcontroller. The microcontroller is switching some electrons. It has a parasitic field. You already coexist with the parasitic field that is much lower than the safety limits. It's okay, but nobody uses the parasitic field. We are utilizing the parasitic field to do the communication. It is understood that these are tiny level of signals like in the level of microwatt, compared to today's cell phones that is sending out about a watt when you are trying to talk to the cell tower. So this 5 to 6 orders of magnitude lesser energy basically means you are much safer.
Dr. Sen, at the moment all our headlines are about, hundreds of billions of dollars being spent on huge systems. So the bigger system, the better. And every day we have a titan of AI tech standing in front of a microphone somewhere in the world saying he and it's always a he - has got the biggest system that man has ever seen. Do you think with your work and with maybe a focus on more energy efficient systems, this is going to change?
Our compute, laptops, PCs are kind of in the middle. And when technology progresses, it's going to progress in both directions, bigger and smaller. So today, there is a lot of excitement around big AI, because big AI is able to do the kind of inference and understanding that computers were not able to do earlier. So there will be a lot of investment, a lot of electricity, a lot of training leading to these complex megawatt scale systems that give us this capability. Along with that, when you have that kind of a powerful AI, you are not always going to take your exabytes of data from humans and always shuttle it to the cloud and come back. So you then want smaller models that will run on device on you.
Once you start to have those small models that is running on you, this AI needs data, where's the data going to come from? One device is not ideally localized around humans. To have all the right sensors and actuators, my control should be in my hand, like just my fingers should be able to act like a mouse. So the sensing and actuation locations for this data source for the AI needs to be distributed. And with on device AI coming after this big impact revolution, that will be the need for this data generation with ultra low power devices. And my work is trying to intersect to that future.
Very cool. You said that this wasn't ready for primetime yet. How close are we to that?
So the brain device is not ready for primetime. There's a lot of medical analysis that needs to happen. It is very hard to predict. I would say it's at least ten years out. Whereas the nervous system around humans allowing distributed computing on video and audio and biophysical signals is ready. Our work has been, commercialized with a company called Ixana, and the technology is called Wi-R. And we won the CES Innovation Award last year and has been present in CES for the last three years, and multiple companies are already trying to integrate this technology and evaluating it.
Wonderful. Okay. Ten years. That's not that bad.
Yeah, that's not too far. That's not that far.
Alright, well thank you so much. This has been fascinating. We appreciate your time. Thank you very much.
Thank you. Thank you for your interest.
What did you think of that, then?
Mind blowing, to say the least. So what's your main takeaway from it?
I think it's, the fact that, yes, the emphasis is on these huge, large models, but ultimately like Dr. Sen kind of implied you can't do AI without data or compute, they're the two things. And the future of useful AI is at the micro level. It's at the local level where you're collecting data locally from an individual, and you're being able to compute it locally as well.
But we're kind of already seeing this aren't we with our watches and our rings, and all kinds of stuff, which is picking up data from us.
Right. And that's what we've been talking about, isn't it? Our data and how that's been shared and how then that's used in AI models to do goodness knows what.
But he's creating the infrastructure to enable that to be useful. And I think we've seen this thread kind of through all of our episodes, which I think is so extraordinary that it's reiterating the same fact that it's about the individual level that's what AI, allows us to predict accurately. And these bigger models, yes, they're useful, but when it comes to the domain, what's going on? How we actually addressing that.
Yeah. I think it's interesting, the fact that his prediction is that, you know, we could be ten years out from the brain component to what he's working on, which I think sounds very soon.
Yeah, absolutely. And I just think the most important thing about this work is the inspiration that it takes, again, from the most energy efficient intelligent organ, which is the brain. And fundamentally, I think the most difficult thing is to identify what about the human brain should we take inspiration from and which is fundamental that we need to kind of distill and what is just extra stuff? Because that's what enables progress. It was just trying to replicate the neuron, which led to artificial neural networks, but trying to get into the whole, like cognitive science. Where does our consciousness come from and trying to replicate that is impossible. So I think Dr. Sen has probably done a pretty impressive job of building upon that original work where it's like, okay, what is fundamental that there's one processing system, so why aren't we replicating that?
Joining us now is Dr. Hongyin Luo, a research scientist at MIT CSAIL, and also a part of the technology company BitEnergy AI. Thanks for joining us today.
Thank you for inviting me.
Certainly. So, researchers at BitEnergy AI have developed a technique that could drastically reduce AI energy consumption without significantly sacrificing accuracy or speed. And the team calls this breakthrough Linear-Complexity Multiplication or L-Mul.
So Dr. Luo, it's Stephen here. So, L-Mul, can you talk us through how it works and why it's so much more energy efficient? We've been having a conversation today in the studio about AI and energy efficiency, so we're very keen to hear all about this.
Artificial intelligence nowadays, more popular models are deep learning models built on neural networks. The foundation of neural networks is, large amounts of linear transformations and nonlinear activations. They stack them each all together to put together a large model. And among these transformations, the most important thing, perhaps, is, linear algebra, including many matrix and tensor multiplications.
And interestingly, the same set of computation is also widely used in computational graphics and geometry, which are handled by GPUs. So that is why GPUs are so popular in the AI area for speeding things up. However, there is a key difference. Ignored by researchers and developers, is the existence of training processes in machine learning. So different from computational graphics and geometry. The goal of this task is precisely reconstructing the geometry models in the space, and then map the points to exactly where they are. So they rely on very accurate linear algebra and precise math calculations. We really need very precise math formats number formats to map the inputs to the output.
However, machine learning is different. So we have a training process for learning. And then after the model is trained, we send the input into the model and the model will reconstruct the outputs. Instead of being like mapping the exact spatial locations, it's more like dumping all your furniture into a moving truck, but just make sure you can put them together back when they arrived the destination. So it's different. We avoid very, precise math calculation. We just, make some mathematics precise enough for putting information back. So by doing that, we eliminated a large amount of unnecessary computation.
And the way we're handling it is replacing the very complicated floating point multiplication with integer additions. So the format of floating point numbers is that it has, it has three sections. So one assigned signal meaning that if the number is positive or negative the second section is exponent, and the third section is meant to solve some numbers. Traditionally, hardware needs to separately handle these three chunks. So GPUs will handle these three chunks more efficiently. FPGA does not. But we've found that we can simply add the mantissa part and the exponential parts together as they are in the one integer number. So doing this one time addition will be as precise as separately handling each part with multiplication. And based on existing research, we found that, integer addition is like 30 times more energy efficient than floating point multiplications. Just consider if you have 16 digit numbers. Adding 16 digit numbers is much more simple than multiplying these two numbers. So that's the contribution we make.
So to talk a little bit more about those floating point multipliers, can you explain for us how these help large language models work?
Large language models are built on neural networks. And the traditionally neural networks contain a lot of parameters and most parameters. and most inputs are like not integers. So you have some integer part and some mantissa part. In computer systems, these numbers are represented with floating point numbers. As a result, full precision neural networks like for use in training are all based on these floating point numbers. So in other words, we can say regardless of the fancy neural network designs, no matter if it's a transformer or it's a Mamba model, it's all based on multiplying floating point matrix and tensors. So I would say floating point addition and multiplication is the most basic element of neural networks where we are replacing the computation element for floating point multiplications. By doing this, the efficiency of the entire matrix multiplication can be significantly boosted.
We've had announcement after announcement, haven't we, about AI getting bigger and bigger and better, and hundreds and hundreds and hundreds, literally of billions of dollars being spent on research and building bigger data centers and bigger models. Does your research and technology, like your technology, sort of question the need to get bigger and better?
People are building data centers for AI. I believe it's a it's something that that must happen because in future computation will be bound with everyone's day to day life more closely. So we will need more and more AI computation in the future. The thing we question would be where should the computation stay? And, what are those computation devices? Do we all need computation staying in those data centers?
I have different ideas. So some people are saying that, we will have edge computing, like every device, like cell phones, AI device, cameras, they can all carry AI and they can interact with their, with, certain, natural language or vision fusion signals with, AI models. If that is true, many, many computations from the data centers can move to the edge. So users do not have to send their private data to your data centers, which is insecure. My idea on this is that perhaps in future there will be family clouds and, small to medium sized business clouds. Every household can have that cloud for for serving AI for different applications and devices. And that might cut the need for data center AI computation.
And the second question is what other computation devices, data centers need to adopt. So currently many people are using, especially using NVIDIA GPUs. I've seen data centers trying to move to AMD GPUs, in some cases claiming that they have a similar performance. And also you see the recent DeepSeek, they claim they are combining different kind of GPUs for the best efficiency. So maybe in future there will be many different kind of devices like GPUs, ASICs, even cloud FPGAs served for different applications depending on the latency. And there's a cost model of these applications.
So as we look into the future, it seems like that's the goal so that we can minimize the amount of, you know, energy being utilized and the amount of space needed for all this data. But I guess my question to you then is, what are the challenges in, adopting these approaches like L-Mul today?
L-Mul is not directly supported on GPUs today. And, so if you want to implement L-Mul on GPU, we need to work around its code and it won't be as efficient as the native matrix multiplications. Implementing L-Mul on GPUs does not help. Which is very unfortunate. It can be implemented on GPUs, but implementing on GPUs won't help. So our current strategy is implementing this algorithms on cloud FPGAs. Which allows us to customize the circuit to make things more efficient. And our FPGA experiments shows that, our implementation is 12 times more efficient than the traditional implementation. So that's our next step.
So I've got one final question for you. I want you to look into your crystal ball, and I want you to look to the future for me. And talking about AI efficiency today, so if there was one big moonshot for AI efficiency, if there was one thing that we could really do in the future to make AI systems more efficient, what would it be?
You should use your answer to be proposing more efficient neural networks or proposing more efficient training data. But I got a feeling that we might need to stop pre-training. Because, number one, all these large models have consumed all the available pre-training data, and some of them might have been using illegal pre-training data. We might need to move from pre-training to reinforcement learning.
Also, perhaps they need to find better methods to edit the knowledge gained from pre-training. So if we can stop pre-training, it's perhaps a not so technical perspective, but if you can stop pre-training many GPUs in data centers previously used for pre-training can be released to the market so developers can use them for inference tasks and their price will go down. So efficiency is mostly about the price of the GPUs. So if people stop doing pre-training then I would say it will significantly benefit the efficiency of the entire AI research and AI industry.
I was really drawn by your comments earlier about localization. You know, having family clouds and business clouds and devices, and surely that by not having as much need for GPUs to do that, pre-training is going to help with that whole movement in a way similar to mainframes to PCs.
As far as I'm concerned, PCs cannot handle very complicated models, but perhaps in future, everyone will have more and more data, and some of the data related to people's work need to be precisely processed. So perhaps we need a bigger model to process this data. Then the models can be run on a PC or a laptop. And I noticed there is a current trend called NAS, which is like a family storage cloud that different users from household can upload their photos, their documents onto that cloud, but it's still a private cloud. So perhaps in the future we might have a medium sized GPU carrying a medium sized model running the household.
Well, thank you so much indeed for joining us today. It's been a real pleasure to talk to you. So thank you very much.
Yeah, I enjoyed the conversation very much too. It was a wonderful conversation. Appreciate it.
Stephen, Autria, what did you think of that?
It was fascinating. I'd say it's really interesting, isn't it? And we're talking about moving into the future and we're talking about kind of efficiency. Just on his point that, if AI didn't need to be trained, that's just incredible, isn't it? Because that frees up so much.
But I think that's really interesting because we will get to a point where we reach superintelligence or such good AI where we don't have to train any more. So then there will be this point in the graph where it just goes straight up vertically, because it will all of that compute to compete against each other, to make better AI's will go to people using it.
Right. And then it's true AI, right? So it's human. It's true human intelligence level, which is the holy grail everyone's absolutely looking for, isn't it. So it feels like that's not too far away.
No, not at all.
And I thought you brought up an interesting, you know, second point that you talked about, you know, the idea of, family computing or, you know, like the smaller business meaning, and it kind of makes me question like, why wasn't that the the original plan? But that also made me think about maybe the next kind of step, like the next opportunity is creating the opportunity for families to collect their data like, like cloud, photos and things like that. So people who are making those applications which allow people to have that data collection, but on a medium size level, are maybe, you know, laying the groundwork for the future.
All right. Because a couple of issues come up there. So he's sort of saying at the moment, the problem is with training data, all the data has been taken. Right. And these are big. These are large language models and they're run by large companies. And companies have all those resources to buy the chips and to scrape the data. And what he's saying is in the future it won't be large necessarily. You know, it will be, as you were saying, family clouds or small business clouds. So this is a complete transformation in the model. It all boils down to the maths. And that's what kind of blows my mind that it's just a couple of equations.
Fascinating. All right. Well that is it for today's show. If you enjoyed our discussion don't forget to like and subscribe. It really helps us improve upon the show. Thank you so much for joining us today. Until next time, goodbye.
Bye. Bye.
Agents of Tech is brought to you by WebsEdge. We want to thank our studio engineer Adam Deen, our editor Abbie Harries, and our producer, Cath Sheehan.