The verdict: AI's Success Is The Danger
Hosts Autria Godfrey, Laila Rizvi and Stephen Horn revisit clips from four past guests to ask whether an AI collapse or AI success should worry us more. Gary Marcus argues we are probably in a financial bubble, Emily Bender says the hype is driven by money and marketing, Derya Unutmaz says AI will save billions of lives but nobody has a plan for the disruption, and Santa Ono of the Ellison Institute of Technology makes the historical case for calm. The hosts then give their verdict on valuations, risk and how unprepared society is.
Transcript
The entire world has gone all in on one bet, trillions of dollars in spending, massive infrastructure build outs, and whole economies riding on the promise of AI changing everything.
We're probably in a financial bubble. The valuations don't really make sense.
The risk is ultimately the entire global economy potentially having a meltdown.
But if it's right, well, then what? The uncertainty of a success could be equally as unsettling.
The thing that nobody is really staking out, what happens if this is a big success?
AI is the greatest thing that will happen to humanity. The path to that is going to be very painful. It's going to be extremely disruptive. And there's no plan, there's no preparation.
If the bet fails, we lose trillions. If it pays off, maybe we're not ready. So what frightens you more, an AI collapse or AI achievement? On today's show, we've put that very question to some of the sharpest minds molding the AI landscape, and they've given us completely incompatible answers. AI skeptic Gary Marcus says that it's a bubble.
It's just not as smart as it looks.
While Emily Bender says the hype is bought and paid for by those behind the billions.
There's a tremendous amount of money behind it. And money means marketing and money also means access to corridors of power.
But Derya Unutmaz, the biggest optimist we've ever had with us, says AI will save billions of lives.
Will literally save billions of lives and reach a point of abundance.
But nobody has a plan for how to get there.
All right guys, let's get into it. Today's talk is going to be about really whether you think we're headed for success, headed for failure, and which of those end destinations concerns you more. Stephen?
Well, you know me, Autria, I always think we're headed for success and-
Ever the optimist.
And with AI, it's no different. Now the scale of that success, I don't know. You know, it could be a big success, it could be a smaller success, but I don't know. Now, which am I worried about? Well, I'm not worried about either, but I do think the thing that nobody is really staking out is what happens if this is a big success. I don't think the implications for society or for all of us have properly been thought through.
Laila?
How do you define success is my question as in-
Billions.
Billions on what timescale?
Right.
Because yes, as in the demand is crazy. You know, we're running-
Well, what did Derya say? "Give me a couple of years and you'll live forever."
Yes.
But how can you know that? And I think the valuations, the whole equation is based on when we get that success, which no one knows. So I think that's a difficult question to answer.
And you're right, to Stephen's point, I think everybody is measuring the definition of success very differently, wildly differently.
But I think what we're all worried about is the whole narrative is, you know, success or failure, failure, failure. What's going to happen if the hyperscalers are wrong? What happens if the billions are wasted? And I just think people are looking at the wrong end of the spectrum. What happens if it's not wasted?
Right.
And we're all-
Well then we're looking at potential mass layoffs and high unemployment.
And we're already seeing that now because the companies who are spending the money are making the money, and therefore I think we're already seeing progress. Now, the big question is not success or failure is does the progress, does the success justify the amount of money spent on it? That's the big question. It's a different question to failure.
All right, well, let's start with the man who has made this case longer and louder than anyone else, cognitive scientist and AI author, Gary Marcus. He weighs in on why neither the numbers nor the technology are adding up. Take a look.
We're probably in a financial bubble. The valuations don't really make sense in terms of two things, the economics and the technology. So from the technological perspective, which is really how I come into this, I can look at how a large language model works, and what it does, and I can see that it has problems. In fact, some of those problems I foresaw a quarter century ago, literally. I wrote a book in 2001 and I pointed out that technologies that are kind of the ancestors of today's technologies would hallucinate. I didn't use that word, but I said that they would inevitably overgeneralize. And that's what's happening. They hallucinate, they overgeneralize, they get things wrong.
On the technical side, the reality is they're like a magic trick, a parlor trick. It's an incredible trick. There's some genuine uses for it, but it's just not as smart as it looks. LLMs are interesting, but they're not the answer to so-called artificial general intelligence.
All right, so he's really complaining about both the financial evaluations of all of these companies, but then also saying that LLMs are really falling short. I know that I tend to be more on the side of Gary Marcus than the two of you, so love your perspectives.
I just think if it's a parlor trick, why is the US government going to such lengths to ban it for everyone else, as in is that just US marketing? We're at a crazy point, and the reality is that it does keep getting better, it gets cheaper as well, and the demand continues to increase. I think the demand has gone up 6.5 times since the last six months, and so it just is an upward trajectory. And if it carries on on that path, there's just not enough GPUs, there's not enough data centers to meet that demand. And so is everyone in the world using AI for just because it's an entertaining parlor trick?
I think that there are a lot of people that are using it just in that capacity right now. You know, yes, ChatGPT is helping people script questions, helping them at work, but, you know, we haven't seen a lot of the medical promises and a lot of the scientific breakthroughs that have been promised so far.
So I've got a question for both of you. What would be the price at which you would not pay for AI? Because I keep running out of tokens, and I am actually willing to give my money because I am so dependent on it for my day-to-day work, as in I do need it to be as productive as I am today, which is half as productive, I think, although it's anecdotal. Being recorded-
I don't know, your boss is right here.
Recorded in the data, but I need it.
Right.
I rely on it.
But I think that's very person dependent.
Laila raises a great point, and I think it's about return on investment, right? You willing to spend if you get a return, and individuals are like that, and companies like that. And what you see at the moment is it flatters to deceive a little bit. So if you ask AI to do something in a business sense, the first result that you get is, oh my goodness, that's fantastic. I've just solved the world's problems. And then when you look a little bit deeper, you probably haven't, but you probably still have made a productivity improvement. And as I keep saying, the key to this is not the productivity improvement that AI can give you from working the way you are now. The key to this is how do you change the way you work with AI at the center? That's the key.
But even those within the field, Alex LeBrun in particular, we had him, you know, here on "Agents of Tech" and he's even seeing that maybe there is a limit to LLMs, thus the world models that he's now working on. So I feel like even some of those inside the industry are starting to think, okay, maybe LLMs can't take us as far as we had originally hoped or promised.
But I think it comes to Stephen's point about, well, what is the use case for them and how you employ them in a business? So yes, I guess if they want to take us to Mars, maybe LLMs in your computer won't do that. But in terms of where value accrues, I think that's a different kind of question and discussion because I don't think it's going to be the people who are creating the models and the intelligence, which is now becoming a utility. It's the people who are getting the intelligence really cheaply. This intelligence is subsidized. I'm getting it for like 20 bucks a month. It should be far more expensive, and I can employ it the same way I can employ a human employee. So yes, I don't think maybe the valuations are correct for those supplying the intelligence, but for those using cheap intelligence, might be a very good story.
But how do you-
Okay, I'm glad you're talking about valuations. I'm glad you're talking.
I was going to say, how'd you know, how could anybody know whether the valuations, the current valuations are wrong, right or indifferent?
Well, I'm glad you brought that up, because what we did not hear there in that clip from Gary Marcus was the why as to why he claims we're in an AI bubble. So what makes the valuation so overblown in his mind? Well, for that, we're going to bring in Bender. Professor and author of "The AI Con," Emily Bender gave us perhaps the most cynical answer for why the money keeps flooding in, and also the uncomfortable part about it is how hard it is to refute. Take a listen.
There's a tremendous amount of money behind it, and money means marketing, and money also means access to corridors of power. Another is it would be lovely to imagine that we can just build something that will solve all of the crises that we're facing right now, right? It would be lovely to think that with just a little bit more compute, then yes, climate crisis solved. I understand that desire. It doesn't make it true.
Okay, so again, I know-
You were shaking your head the whole way through.
Do you still disagree with Emily on those remarks?
Well, this is the problem with clips like that, right? So basically we're saying, will AI solve the whole world's climate problems? And the answer is no, clearly no.
But that is what a lot of the tech bros are promising.
No, but no, no. But will, but is there a use case for AI? Is the investment that's putting in now, is there a use case for that investment? Yes. I mean, I know business can be a bit boring for those people who aren't kind of doing my job every day.
CEOs.
Right. But productivity improvements are really important. And AI, if AI can get you a 20 or 30% productivity improvement, that could be worth millions or billions of dollars. You don't have to get super general intelligence tomorrow. Now, do I believe? I'm with Derya. I believe we will get super general intelligence, and we will see those, but you don't need that to make the investment case worthwhile. It doesn't have to be an all or nothing.
Laila?
I do also understand from the hyperscaler's point of view that when they're assessing risk, is it more risky to not take the risk or put in all that money? As in I think from an individual perspective, which is the rest of us, you think, "Oh yeah, it was really risky to risk all that money," but for them it-
So listen, I am going to-
The opposite.
Risk the wrath of a lot of viewers by saying this, right? But in my view, we have now some of the most successful companies the world has ever seen who are willing to bet hundreds of billions of dollars on this success. And I, for one, I'm not going to say that they have no idea what they're doing, right. And for me, that's almost where it goes. I mean, they know what they're doing, but your point is exactly right. I mean, where is the, and that's what I don't think we're getting our heads around here. Where is the risk?
But what is your litmus test for the success that they're all saying that they're reaching, which warrants these extremely high valuations?
Well, that's a great question.
That's what you pay me for.
Yeah. Giving him time to think.
But that's the thing. Right, right. That's the thing is you've got these companies, you've got your Anthropics and your Palantir, and you've got these companies that their valuations are, you know, in the Bs and Ts, but they're valuing their own companies so that they can present that to investors, that they can get more money, so that they can. It's a cyclical thing that-
Right, but again you're-
But what's the output that you're considering it a success?
Yeah, but, so you're doing a Bender here and you're-
I know.
I like how we've got a verb for it now.
And you're-
I've gone Bender.
You're conflating everything, right? So you're basically saying all these companies and all their valuations. And clearly there will be some companies whose valuations seemed excessive. I mean, SpaceX at $1.75 trillion is clearly a lot of money, but that doesn't mean that all the investment and all the valuations are wrong. And you will get some that don't sync up. But what my point would be right now, this is an answer that people don't like to give, the honest answer, we don't know.
Right, which goes back to then why are they able to value these companies at such high levels?
That Laila's point about the risk, the risk of not being there.
But also coming back to the other side of the risk, as in we say that, oh, the risk is that these companies will become bankrupt, or we get super general intelligence. But the risk is ultimately the entire global economy potentially having a meltdown. That could happen.
Listen-
Are we going to bail them out?
Well, so we're expecting-
Been there, done that, US.
So we're expecting Microsoft and we're expecting Amazon, we're expecting Meta, and we're expecting Alphabet to go bust?
It's not that companies that big haven't gone bust before, right? Not that I necessarily agree with that, but that is the opposing-
Need I remind you of AOL.
As in, and I agree with you that it's not going to be all of them that are going to get a return on what they do, but clearly that is a risk that they're willing to take with the possibility that they might make trillions, and I don't even know what comes after trillion.
We'll find out. It'll be through AI. Okay, well, clearly-
Quintillions?
We're going to have to agree to disagree on this, because coming up, we've laid out the arguments for why we are possibly poised for maybe not an AI collapse, but at the very least, a little AI fizzle perhaps. Can we at least agree on that? All right. But suppose the bet pays off, suppose it's right. We will examine the likelihood of success on the other side of the break. But first, where do you fall in the AI prediction? Are AI company valuations over-hyped? Can LLMs really only take us so far? Weigh in, and while you make your predictions, we'll take a break.
Welcome back, and we're back now with the clip that actually sparked this entire episode. Immunologist Derya Unutmaz is perhaps the most optimistic guest that we have ever sat down with. He believes AI will cure disease and save billions of lives, but let's get started with how he told us about we don't even have a path to getting there.
While I believe AI is the greatest thing that will happen to humanity, because, you know, will literally save the billions of lives and reach a point of abundance, the path to that is going to be very painful. It's going to be extremely disruptive. And there's no plan, there's no, you know, preparation for governments or organizations how to deal with that. I mean, the whole education system has to change. You know, well, what's the point of going to college just to get a job? You know, 30% of the college graduates cannot find a job in the US right now. So we have to rethink the whole fabric of society. I mean, this is the biggest transformation since the beginning of history, and, you know, it's very disappointing that we have very little discussions. Maybe hopefully it will start this year, but it might be a little late.
Well, we're working on that component of it. We're having the discussions here. But what strikes me about his stance is he's the optimist. He's the one saying this not only will work, but it's going to save lives, and it's going to, you know, reshape the world as we know it, yet he's still got a component of negativity.
No, he doesn't have a component of negativity. Listen, listen, I'm one of 305,000 people who really enjoyed this episode, right?
Okay.
And that is not what he's saying. He said there's a rocky path, right? And he's saying there's a rocky path because governments and societies, and everybody has to try and figure out how this change is going to affect society. And that's what I say every week, right? Instead of this bet, does it work or does it not work? Why don't we actually think that it might work and do something that's going to prepare that path? That's his point. His point is not that it's not going to happen, it's that there's going to be a lot of change to get there, and that's my point too. And rather than sort of talking about, "Oh, are we investing? Is it working?" Let's just figure out the change to society that this is going to cause.
All right, well, let's get to the other side of the coin though, because the historical case for calm was made by Professor Santa Ono from the Ellison Institute of Technology at Oxford. See why he had this particular take on the doomsday scenario.
These questions about AI replacing people, that will happen. Technological advances in the past have also eliminated jobs.
Right.
But they've created new kinds of jobs, jobs that are, in my view, even more interesting. You still have to be thoughtful. You still have to have, you know, policies and regulation, but it's not a reason to shy away from something that could really change the way we approach the issues and problems that we face. Even today, I can tell you that artificial intelligence is revolutionizing and shortening the time required for, as you know, rational drug design.
Right.
Totally transforming it in a positive way, and it will save lives.
All right, so he might be the most calm and collected one out of everybody saying, yes, growing pains are going to happen, but perhaps they won't be as bad as we're all anticipating. You tend to agree?
I think a lot of the points he made about humility and being humble, and about thoughtful and listening was all right. But again, he was talking about all, you know, the good things that we're going to get with drug discovery, you know, and I agree with all that. But he's not actually saying that the path isn't rocky.
No, that's what I was going to say. Did you think he was, as in he sounds calm and collected, but I think what he was actually saying-
Maybe it's his delivery.
Maybe, yeah. I think it's his delivery. It does hypnotize you that you think, you know what, maybe we're in safe hands. It'll be okay, yeah.
But he is also telling you, and the bet that his boss is making is telling you about the size of the bet that they're making on this technology, right? So I mean, they are all in.
Yeah. Also, the other thing is, I know you say that we don't know often whether there are going to be productivity increases, et cetera, but it has been really successful from a science point of view. Like, going back to what Santa was saying, what Derya was saying, that it's totally changed the game, obviously with AlphaFold, but Isomorphic Labs has achieved some crazy things in the last year, couple of months. So to say that, "Oh, it's just not coming up," as in science is accelerating as a result.
It is.
And we have done some really interesting episodes about the future, right, with materials and with worldview models, and so there's a lot more to come, isn't there, that we're only scratching. You always say, was it three years since this came out?
People have amnesia that this just has not been around for that long.
It hasn't, right.
And you forget that ChatGPT was barely writing poems, as in we were doing things manually, and now you just can't imagine a world-
I think because it came onto the scene with such fanfare that everybody is thinking, "Okay, well, this is what it started out as," and I don't feel like they've seen a lot of progress in the last three years, when, to your point, there has been. You know, but it did, it came onto the scene with kind of an explosion, and people are like, "Wow, this is really cool. This is amazing," but what's the advancement of it?
It also arrived in a personal capacity. So everyone's experience of the end product is very different. So people who are in frontier labs, people who are in scientific labs are having a very different experience, and not necessarily of just LLMs. This is what we come back to a lot, that AI is not just LLMs.
Right.
And although Emily Bender makes that point a lot, she's right that it's not just LLMs that we're talking about.
Yeah, and, you know, you're not going to like this Autria, but the new generation of what I would call AI natives, you know.
100%.
We've got digital natives, we've got AI natives.
Yeah.
I mean, people are growing up now with this technology, aren't they?
And that's something that some of them alluded to is how this is completely going to change the landscape of education, which I think is something we really haven't delved into a lot here, but I think that's probably a conversation that's going to have to be had because, yeah, I don't think that the four-year university degrees are going to, I think they're going to either change drastically or become antiquated.
A perfect example of this is the job of a data scientist. Didn't exist till the 1980s. There's now millions of them, and that wasn't that long ago.
Right.
So, as in, and again, if you look at the timescales, AI has been around for three years. What's going to happen in 40, 50?
Right.
Well, think of those timescales. It was a year ago we sat down with James who introduced us to agents and we laughed, right? We thought it was the funniest thing that we'd heard and now-
Yeah, that they go to conferences and chat to each other.
And now you have agents and you have sub-agents, and you have people who manage your agents, don't you?
Yeah.
Right. All right, so let's get to our verdict today. We've gotten to hear from very different, four very different perspectives here who are really kind of in total disagreement on whether the AI bet pays off. There is one piece of common ground though that we are all unprepared. So how do we tackle that elephant in the room?
Well, I think the first thing that we have to have is an honest discussion, right? We need to start talking about some of these issues, and that's why I think we need to get away from good, bad success, and not success, underclass, not underclass. I think we need to get away from that whole dialogue because I just think that's not helpful. And I think we have to try and understand what the issues we face as a society are, and we need to, you know, we need to have a dialogue around that. My concern about that is how long that is going to take. You know, and from previous, it's taken decades, and I don't think we have the decades.
We're going to be playing catch up.
Yeah, I also think it needs to be taken more seriously. Honestly, it's just attached too much to the money rather than actually what's going on globally. The argument tends to be, "Oh, open Chinese models are going to completely collapse the valuation of American companies," and this is way bigger than that.
You know, companies are in business for one thing and one thing only is to make a profit. And then legally, companies aren't allowed to do anything else. That's what they do, they make a profit. So these hyperscalers that we're all laughing at, we're all chucking all these billions of dollars at, if they're successful, they're going to make a huge profit, and what impact is that going to have on the rest of us? That, nobody is trying to figure out.
Well, I do think that, you all know my perspective on this, I do think that the valuations might be a little overinflated. But as we've seen, if history has proven nothing, it's that the markets will correct themselves if they're, you know, actually incorrect. I think that'll come though at the cost of investors, and I think that when it comes to waiting to see how AI, the technology will advance, I think it'll depend on how much runway these investors want to give the companies before they really start seeing that ROI.
My only concern is when has consensus been right?
Right now, you know, a big bubble about chip producers, right? And people are sort of saying that bubble is going to deflate in the way that you're saying. But what we're seeing is a demand for compute rising exponentially every year, right?
It may not.
And it's that rise for compute that is the key and why people want the compute, and it's what the compute does for them. And as long as the compute does something for you, then yeah, you will see this revolution. And it is a revolution, that's my point, I think. And I don't think people, people aren't comfortable with that. People like the status quo.
Oh, yeah.
They like to believe things are not going to move that far.
Change is hard.
And I got news for you, they are.
Yeah. All right. Well, we shall see. If the bet fails, we lose trillions. If it pays off, maybe we're not ready. So which one do you feel is a likelier scenario? Weigh in in the comments. Are we headed for an AI takeover, like Silicon Valley is promising? Or is this all over hype and maybe inflated promises? Leave your thoughts and then be sure to subscribe. Thanks for joining us here today for our conversation, and we will see you next time on "Agents of Tech."