Alphabet's Chairman: LLM’s Aren't the Endgame
John Hennessy, the Turing Award-winning computer scientist, former president of Stanford University and chairman of Alphabet, on what AI has to deliver to justify the money being spent on it. He argues that even a few percent of extra productivity a year adds up in an economy the size of America's, that reasoning is the next frontier, and that the return may take time to show up.
Transcript
725 billion dollars, that's what Big Tech is projected to spend on AI just this year. But even with all that investment, the bubble could still burst. John Hennessy, the Godfather of Silicon Valley, argues that AI's potential is enormous, but it's not guaranteed.
Clearly to justify the investment, you've got to somehow yield some productivity enhancement, or some business influence.
There's trillions of dollars worth spending to get those incremental improvements.
The AlphaFold was a 20-year jump on our understanding of protein structures, which are critical to modern medicine and drug design. So, we need a few things like that to really deliver.
The world's most successful company, spending their entire cash generation on arms race, if you like. This is unprecedented, it's kind of never happened before.
You can only really get that projected return if you replace people. You need quite large-scale disruption, which is not going to be good.
It's a new Industrial Revolution. It's one that potentially is much larger size than earlier ones.
So, is AI dominance inevitable, or is it about to become the most expensive disappointment in history?
First though to you, Stephen, John Hennessy, he's a big deal. He's a Godfather of Silicon Valley. What would you like to hear from him today?
Well, it's so exciting to have him on, isn't it? It is!
Yeah, I mean, I want to follow up with that. I want to know kind of what the full potential of AI is and kind of how we get there. I want to get from him what the future looks like. And that's what I'm most excited about. Because we get this kind of ping pong over valuations all the time. The valuation depends on the value that those companies bring and what the future looks like. And that's what I want to hear from him.
Yeah, I also think, he's just maybe the most uniquely positioned person ever to talk on this. Because he's been in academia, he's been president of Stanford, which is basically the feeder school to every single Silicon Valley company, you have ever heard of. And he's chairman of one of the most important companies ever. So, I think he-
And he won the Turing Award for his work on chips.
Exactly, so, and he's seen it historically play out, so he can actually compare it to other technologies. He's not just doing a kind of historical paper on it, a literature review. He lived through it and he was on the front lines. He's started companies by himself. He's been in big companies. So, I just don't think there's anyone better to actually ask about what's actually happening right now and what's going to happen.
Yeah, all right, well, let's get to it. Here's our conversation with John Hennessy.
So, you've made the argument that we are at risk of misusing AI and risking missing out on its greatest potential. What are we getting wrong right now?
I think that there is a lot of focus on some of the negative aspects of it clearly. And that's created some public backlash about it. And not enough focus on the things that it could really do that would improve productivity. And particularly, I'm a particular fan of having AI do the parts of my job that I don't want to do. The drudgery, the things that are unpleasant, right? So, I'm an educator. One of the things educators, they hate, we hate grading. So, how about a really good AI system for grading that could do that part of the job better?
I think if you got an F, you wouldn't be happy as a student AI doing your grading, right? Right.
But so, to that point though, where do you think we're missing the potential for AI?
Well, I think it's early days still. This is a technology that has both broad impact and is moving very fast, and that's what makes it different, much faster than earlier technology revolutions. And as a result, I think there's still a lot of hubbub at the frontier, which is important. But we have to find applications in a vertical sector that will really make it different. So, pick a vertical sector, healthcare, pick healthcare. It's something that's very inefficient in the United States. Couldn't we do better using AI as a tool to improve the productivity of nurses and doctors and the entire healthcare system?
Professor Hennessy, you've spoken a lot about how we need to start having more of a conversation about what makes AI different to humans, right? So, in 2026, how would you design a new Turing test?
That's a great question! The crisp thing about the Turing test, was it was relatively simple and people agreed that if this machine in a black box could act like a human and converse like a human, we would regard it as being intelligent.
What we don't have is a Turing test for what I would call reasoning. The original Turing test is really language. It's nothing more than that. It's not reasoning. How do you solve a problem? So, we need a new set of Turing tests. Possibly the closest thing I've seen, is what's called Humanity's Last Exam. It's a collection of 2,500 hard problems. I mean, they're college level, top undergrad, senior undergraduate or beginning graduate student. And so far, our passing rate on that test for the best models, so far, they earn about a 60%, which in my book as a professor, that's a failing grade still. So, we still have ways to go to get to really reasoning models.
But do you think we are chasing the wrong thing, especially when it comes to productivity? Do you think it's wrong for us to chase human intelligence, because AI learns so differently from humans? And the value is actually in what AI can do that human intelligence can't. So, human attention is very serial, whereas AI attention is parallelized. So, should we actually be chasing like parallelization with AI rather than human abilities?
Well, I think we can see this if you look at the focus on agentic models, people using AI tools, but driven by a program, which is telling them to search a space and try out. One area where I think this has gotten a lot of traction, is in scientific research where you need to evaluate lots of different things where the evaluation is fairly complicated. So, you need a sophisticated AI tool to evaluate whether or not you're making progress. But you want to be looking at many, many different combinations. So, that's a productivity enhancement. Because otherwise, if you do this in the traditional way, each project is a graduate student. So, you need a lot of graduate students to explore a wide space.
I understand that AI clearly has a lot of potential. A lot of the applications that you've been talking about very much about potential. But from a business perspective, it's a huge investment that's going in at the moment and not just from the so-called hyperscalers, but from a lot of different companies. And is it responsible to basically say there's a lot of potential there?
Clearly to justify the investment, you've got to somehow yield some productivity enhancement, or some business influence. One thing to understand is if you look at what's often called the paradigm of technology shift and you look at what happened with information technology, it took a long time to realize value from the shift in information technology. It could take some time with respect to AI.
The other piece of this, which is critically important, is driving down the cost. We have to drive down the cost, not just of training. Training, there are a few companies that need to train really big models. Most don't need to train really big models. So, that's not the critical issue. The critical issue is inference cost. Because that's what's really growing by leaps and bounds. And we're going to have to optimize that using new algorithms and new things that people create to really drive the cost down.
One of the points that you made earlier was about productivity. And that a lot of different companies will be using these tools and gaining that productivity is what ultimately pays for the investment that's going through. A lot of the work that we've seen shows that right now, companies are doing a pretty poor job at this, that something like 80% of companies aren't really using AI tools to drive productivity. So, what needs to be done to help companies to actually understand how to make full use of these tools?
I think one of the gaps that still needs to be filled in, is to take an AI, set of AI tools and build them out into a vertical sector. Because in the end, it's really the productivity is going to be in that vertical sector, whether it's healthcare or marketing or sales or education. So, we need to develop tools that are specific to that vertical sector, not just a general LLM, but something that actually has knowledge and understanding of the sector.
And who is best positioned to do that build out? Is it the larger companies, is it academia? Is it government to help assist that? Like where do you think that build out is going to originate from?
I think it'll actually be a mix. I think you'll get different parts of the sector doing different things. Clearly, in academia, you'll get a lot of people using it for scientific research. You'll also get some people using it for education, which we'll see then how to carry that down through the rest of the educational system. I think government could certainly use it to improve efficiency in government. Certainly, there's lots of opportunity there, I suspect. But I think each, these sectors are different. That's why many of the companies, new companies that will be created, will be AI and a vertical sector. And they will use their knowledge about that vertical sector to train and post-train an AI system that specifically is targeted at that sector.
I want to talk to you about something that you said where you think AI will have the biggest impact, which is in scientific discovery and perhaps drug creation. Your buddy, Dario Amodei made the comments not too long ago that he thinks that within five to 10 years, we could see AI being used to wipe out most cancers and rare diseases. Do you agree with him on that? And if not that specific timeline, when do you see AI having the biggest impact on medical research and breakthroughs?
I think that's a dream. But that's a much harder problem than I think he indicated in that quote. I think we can make progress on solving these problems in that term. But also, drug discovery is one that I think is probably closer in. And in drug discovery, part of what we're trying to do is discover new drugs. But the other piece of what we're trying to do, is reduce the incredible cost that it takes to develop a new drug, take it through clinical trials, and then find out it doesn't work. So, suppose my AI system, what it really does is it gives me a much higher probability that a candidate that I discover in the lab, will actually get through clinical trial and will be successful. That saves a fortune in terms of getting down drug costs and getting new medicines to the market.
All right, coming up on the other side of the break, we will ask John Hennessy whether he thinks we should race, or regulate at this point in time. We'll be right back.
We've had a lot of stories recently about AI going rogue and breaking out and sort of hacking into kind of other systems. And so, there's a lot of talk now about regulation. How should this industry be regulated? Should it be self-regulated? Should it be government-regulated? Should it be internationally regulated? Where do you stand on regulation?
I think regulation is complicated. Because you want to prevent misuse of the technology, while not slowing it down overly much. That's the tricky thing about getting regulation right. I think there are a set of things we need to do as society to deal with misuses of this technology. Because it's a very powerful technology and people can intentionally misuse it. And I'm not sure while we have a tort system, we have a you can sue somebody. That system is a slow, long process. So, we need to deal with that issue.
Then we need to deal with the safety issue as well, right? Unintended consequence, as you indicated, these things breaking out of sandboxes, doing other kinds of things. I think some alliance between government and industry would be the right way to do it. Demis Hassabis has proposed sort of the equivalent of what we do for financial, for banks, for example, to really do that sort of process. I think something like that. We've got to get the companies involved in it though. Because they understand the technology and how quickly it's going to move and be responsive to it.
You talk about misuse. And you recently mentioned that you were concerned about AI being misused in the sense of autonomous weapon systems. Is that a concern that is born out of maybe agents going rogue, or the technology ending up in the hands of bad actors? And what's your solution to that?
So, I think it's technology ending up in the hands of people, who are willing to use it in an autonomous fashion. If you look at what's happened with drones in Ukraine and the role they're playing in Ukraine, or in the Persian Gulf. So, now, imagine that those drones are even more powerful and that are completely autonomous. There isn't somebody watching a target. They send the drone out and the drone does its own thing. It's going to completely change the way we interact. And I think we need to have a global agreement on what's allowed and what's not allowed in the same way we did for things with poison gas or biological warfare. We need to do that internationally.
From your vantage point, how feasible do you think that is right now?
I think it's difficult. But the alternative is less attractive than the difficult process of slogging it out.
So, you've spoken a lot about the benefits of AI and how that's often underappreciated. And right now, I think the American public in particular, doesn't recognize the benefits and really sees a lot of the issues they're experiencing in relation to climate change, job loss, et cetera, attached to AI. And these are deeply interlocked issues. So, how do you propose we go forward to try and reverse that and allow them to actually see the benefits? As in Dario Amodei recently said, the only way is to cure cancers and that's the only feasible way to turn the tide on this backlash. But what do you think?
I think actually, getting some really compelling applications that people see AI systems as augmenting rather than replacing humans and making them more effective. So, let's take an example of an experiment we've been doing at our medical school where physicians use an AI assistant to do their clinical notes. So, the doctors are telling me, it's saving them about an hour a day. So, what are they going to do with that extra hour? They're going to see some more patients, which is what they went into medicine for, not to write clinical notes or deal with insurance forms. That's not a good use of their time. So, I'm a particular fan of finding places where we can enhance the quality of people's jobs and thereby, we're improving productivity as well. But we're making their jobs more pleasant. So, that's a very positive approach.
It is a positive approach. But playing devil's advocate for just a second, all those improvements, if you like, are incremental improvements. We're saving an hour a day. And it's the current investment that's going in, and now, a lot of it being off balance sheets as well into trillions of dollars, is trillions of dollars worth spending to get those incremental improvements? Do we not need a killer app? Do we not need an artificial general intelligence, a super general intelligence, cure for cancer? Do we not need a big play for the valuations to make sense?
Well, it's a large economy. If you could get a productivity enhancement in the economy, even a few percents a year adds up in economy of the scale of the US. So, that's part of what you have to get. But obviously, we'd love the killer app. The killer app would make an enormous difference to people understanding what this technology could do.
To Stephen's point, if that ends up being what comes out of AI, is that we've got these programs that can really help with some of these admin tasks, do you think that we're at risk of AI being considered one of the most expensive failures in history if we don't have that cure for cancer?
We're still in the early days of AI, right? The transformer revolution, which brought about LLMs is eight years old. Remember that for many years, 50 or 60 years, the investment in AI didn't produce big breakthroughs. Now, it's starting to produce big breakthroughs. And what happened, for example, with AlphaFold? I mean, that was a 20-year jump on our understanding of protein structures, which are critical to modern medicine and drug design. So, we need a few things like that to really deliver and really convince people.
And I'm completely on side with that. And I completely agree with you on productivity. But I have never seen the amount, the world's biggest companies, the hyperscalers are arguably the world's biggest and arguably the world's most successful companies, spending their entire cash generation, plus some off-balance-sheet numbers on a sort of arms race, if you like, to this extent. This is unprecedented, it's kind of never happened before. So, it's difficult to find a historical perspective that kind of helps one make sense of that.
This is a transformation. It's a new Industrial Revolution. It's one that potentially is much larger size than earlier ones. But it's going to play out over time. And I think then we'll really understand what kind of transformation it creates in the economy.
You've said that we're still in the early days of AI. But I know you've also previously said that we've picked all the low-hanging fruits. So, if we're still early and a lot of the low-hanging fruit has been picked, where do we go from here? Because is the path forward only just harder?
I think you can see this with some of the new attempts to build agentic models. They're much more focused on reasoning, right? Because if you think about what people do, it's sure they have language, they communicate, but they reason through solutions to things. As these AI systems get better and better at reasoning, they'll be able to solve harder and harder problems. And that will then unleash that value.
In terms of like inference demand and that being one of the major issues, do you think with our current investment and current resources, to be frank, we can meet that projected demand, or do we have to do something revolutionary to get over that obstacle?
Those of us that are technologists, have to do is continuing to think about how to optimize the whole stack. And the other piece of this, which is a subtlety, you don't need necessarily to use the most capable frontier model for a lot of problems. A lot of problems can use models which are a lot smaller. And we've seen this with, for example, the open-source models from China, the open-weight models, that small models can do a lot of these problems. And you don't need the cutting-edge model. Well, the small model is much cheaper than the cutting-edge model. So, I think we're going to have to tune our systems to get efficiency out of them in a way that right now, it's a race. Everybody's going as fast as they can. Then you've got to be efficient as well.
How are we doing in the race?
Well, I think the US is still in a lead. But I think China has proved to be much more capable than we thought in terms of ability to stay up. They're going to give the US companies a run for the money.
Should we be selling chips to China, even if they're outdated ones?
Well, I think we should be very careful about deciding that. That's a complex national security issue. But I think we better think it through carefully.
What a diplomatic answer!
Now, I have a question for you. A couple of interviews we've done, have been about world models. And when discussing the world models, people have been somewhat disparaging, somewhat disparaging about LLMs, saying LLMs are just kind of one aspect to this. What do you think about that? What do you think about world models?
The statement that LLMs are just one piece of it is absolutely true. What LLMs demonstrated to us, was that language is a really deep human characteristic that we associate with being human. And LLMs, because they are great natural language systems, they persuade us that they're brilliant, right? But they managed to figure out that the capital of France is Paris. They didn't actually, nobody actually taught them that. They just read it enough times that they now know it, right?
I think the world models, the ability to figure out the physical world, understand the limitations of the physical world, and of time, the fourth dimension, is absolutely crucial. And if we're going to make the breakthroughs in robotics, for example, that the AI community currently hopes, it can make robots that are more flexible, capable and humanoid-like, then world models will be crucial to that.
And on the point of LLMs, as in throughout the years when we've scaled them, they have surprised us. Looking forward, what do you think in the field of AI we will be surprised by?
When the next major algorithmic breakthrough, I mean, transformers fuel the LLM revolution. Now, imagine somebody comes up with another algorithm that's much more capable, particularly at reasoning than the LLMs. Then there'll be a new set of breakthroughs.
All right. We have a couple of rapid-fire questions just to get your thoughts. These are the questions we ask all of our guests here. Mine is, where do you think people will draw the line when it comes to inviting AI into their personal lives?
I think they'll draw the line when they conclude that the AI system is finding out more than they know about themselves.
Oh! That's scary and very true.
And mine is what's something that's universally accepted in your field that you actually disagree with?
That LLMs are naturally smart somehow and operate similarly to the way humans operate.
And my last one, which is not going to be a difficult question for you, is what are you optimistic about?
I'm optimistic that enough smart people working together will find a way to bring this technology to do real good for society.
Okay, so, that really ran the gamut. I feel like we covered a lot of topics there. What was your biggest takeaway from our conversation with John Hennessy?
He, throughout that conversation, really emphasized the power of reasoning and like how that's the next step. And I don't think we've really looked forward in terms of the next step when it comes to actually how are we going to do this. It's just been like, "Okay, we're going to cure human disease, we're going to do this, we're going to meet all the demand, we're going to build these data centers." But actually, I think he did start to pin, like where we're going with it on a technical basis. Mm-hmm.
I think as well, he outlined, we asked him a lot of stuff, right? We did. A lot of questions, covering a lot of ground. And I think to me, came out, it's going to be a journey, right? There's going to be stuff that's going to be discovered. There's going to be breakthroughs. There's going to be disappointments. There's going to be need to regulate. We're going to do this globally. This isn't going to happen in six months, right? So, it's not about what's next out of the box. This is going to be journey-
Or five to 10 years even.
Right, and what did he say at the end? He said, "We just need enough good people." Yeah. And I can't but agree with him.
Yeah, I think he also kind of did a good job of putting into perspective, even if we don't get the cure for cancer, if we're able to completely revolutionize the way that doctors and nurses operate within the healthcare industry and that lends itself to better patient care, that is a success.
My only issue with that is that on your point about the valuations and the money being invested, you can only really get that projected return if you replace people. You can't do it if you accelerate them a little bit, or make their lives a little bit easier. You need actually quite large-scale disruption, which is not going to be good.
Yeah and it's like one of our previous guests said that what's probably going to happen, if that's the return that we see, is that some companies, that will be fine and some, it won't. That's not going to be fine for everybody. That's not a big enough return for the amount of money that's being spent. It doesn't mean it won't be good enough return for some people. But I also, I was very interested in the point that he said there'd be a whole new range of companies that come in with the sectors. Yeah.
I think that's going to be very interesting. So, I think the jury's out, we just don't really know. But what a fascinating conversation.
Such a good conversation! Our thanks to John Hennessy for taking the time to be here with us on "Agents of Tech" today. We would love to hear what you think, weigh in in the comments below, and be sure to like and subscribe. Let us know what you thought about our conversation with the godfather of Silicon Valley. We'll see you next week on "Agents of Tech".