AI Superintelligence: Are We Racing Toward Extinction?
Malo Bourgon, chief executive of the Machine Intelligence Research Institute, tells Autria Godfrey and Laila Rizvi why he thinks the race to build superintelligent AI could end in human extinction. He discusses whether an off switch could work, the parallels with nuclear weapons, and what would persuade him that AI is safe.
Transcript
Will AI destroy humanity? Most people think that sounds like science fiction, but the people actually building it, aren't so sure. Our guest today, Malo Bourgon, CEO of the Machine Intelligence Research Institute, has been warning about this for two decades. He says, "Machine learning doesn't need to be sentient to become a global risk, and today's safety measures, nowhere near enough." So can we create an off switch for a machine that's smarter than us?
My co-host, Laila Rizvi is here in DC with her thoughts. And the question that we want to get an answer to today, "Are we underestimating the dangers of AI for the sake of profits and geopolitical power?"
I feel like we're always just on two extremes here, Autria, that AI is just going to destroy us all, or, ah, it's the whole load of baloney basically. I don't think the solution is just to, you know, stop it all, lock it down. I don't think that's ever a good idea.
Right.
I don't think doing that during COVID, just locking everything down in hindsight, was bad idea, right?
Didn't work.
Because there are other harms which happen, like... I mean, I'm not saying the economy should be ahead of everything, but if you just stop AI, then what does happen economically like that?
Right,
Right.
That translates into millions of people potentially losing their livelihoods, and a lot of things that are dependent on it. But the other thing is, obviously, the latest technology is like the most amazing thing we've ever seen. So we're also simultaneously going to think it's the worst because we've not seen anything better.
But don't you think we have to perhaps, maybe encourage progression of AI while also making sure we're staying within parameters?
Yeah, but I wonder if it is really as bad and as powerful as they are making it out to be. What if some... I mean, there's going to be something after AI, which is also going to be really, really powerful and probably AI-
A little scary.
Yeah, and AI's probably going to look kind of really dated.
Really antiquated.
It's going to be really dated comparatively so... And we're super adaptable. Humans are really adaptable. We adapt to crazy things.
Yeah.
Yeah. All right, well, that is what we're going to get into today with our guests. But before we get to Malo, we want to hear from you. There is one viewpoint that today's AI systems are overhyped, that progress is slowing, and that there's really no existential threat. But then on the flip side of things, what if we are building something that we then can't control? That is the dilemma keeping some of the smartest people in tech awake at night. So what do you think? Lay it out for us in the comments below and then share where you think we are headed. So now, how worried should we be? Malo Bourgon is here to make his case.
All right, Malo, so to get started today, the million dollar question, given where things currently are with regard to AI, do you feel like we are barreling toward extinction?
I mean, you know, exactly how quickly things will continue to progress and whether we're barreling towards it at what rate, is a hard call. But I do think that we're on the path to building systems that are radically smarter than us, and I think there's kind of a very real common sense concern that if we do that and we get to AI systems that will actually help us also do AI research itself faster and faster and more autonomously, that we are kind of just speeding faster and faster in a race to build systems that we don't understand, that are systems that are more like we're growing than we're crafting. And we can talk about that, and that kind of, one of the main consequences of that is we'll probably be in a world with AI systems that are radically smarter than us that we don't know how to control, and that that's a real big risk that could end up with none of us around to kind of see the future that we could have built instead.
But when you say widespread belief, I mean, we hear from Anthropic CEO, we hear from Sam Altman, but do you feel like this is a concern and a belief that is held by all of the big tech companies, it's just that maybe some of them are a little bit more vocal about it than others?
I mean, you know, I don't know what's in the heart of hearts of various people. I think there's kind of maybe some context to back up on here where, you know, for most of the field of AI, I think the goal was to eventually be able to build these systems that are kind of doing the thinky thing that we're doing. And most people were kind of like, "Oh, that's 20 plus years away." And what that usually meant is like, we don't kind of know when that's coming. We think it's possible, but we don't know.
And something has changed in the last few years, that something that kind of, we weren't able to do before is working, and all of a sudden the machines are talking and we're investing more money than we ever have before to the point where, I think we, have to be sceptical of individual claims of like, is it one year, five years, you know, two years to something like AGI and what does AGI even mean. But I think if you survey experts at large, people working at the companies, people who were responsible for developing this technology like the godfathers of AI, there's kind of a consensus that we're making progress that we were having trouble making before, and that it's likely that we'll be able to continue making progress at a pretty fast clip.
Mm-hmm. And Malo, I also want to pick up on our current models, as in some of them have been shown to blackmail, cheat, even played dumb when they're being evaluated. How dangerous do you think the models are that we've got now? I mean, we just spoke to Emily Bender who said, "Ah, they're still stochastic parrots," which is kind of crazy, but want to get your opinion on it.
Yeah, I mean, so I don't think the current models are existentially dangerous in the way that I'm worried that if we continue to race towards super intelligence, those models will be. I think the weird behaviours we're seeing in those models are, you know, interesting evidence that we are kind of growing these AI models that we don't understand. They end up having all kinds of behaviours that nobody asked for. There's all kinds of nearer term harms that are, you know, a result of that, and we're seeing more and more evidence of that.
And so, you know, I'm not worried that the AI systems that we're making today are somehow going to lead to human extinction. I'm worried that the signs that we're seeing like these, you know, this deception and that sort of things, are kind of the first signs that things that people were predicting, you know, 5, 10, 15 years ago, about some of the emergent behaviours we'd see in these smarter and smarter general systems, that those kind of theories were correct, and that we're not on course to understand these systems deeply enough that we'd be able to solve those problems, and if we continue to scale them towards superintelligence, that those problems will have bigger and bigger impacts.
And just a followup question to that as well, do you think our current models are exhibiting signs of understanding?
Yeah, I mean this is, you know, I often like to say that for many of these questions, there's some deep philosophy and maybe some of that is between like, you know, different people in their dictionaries about how they want to use words. People talk about AI systems knowing things or wanting things. And what is knowing or wanting? I kind of like to, you know, think about, are we building AI systems that in some sense are getting better and better at pursuing goals?
You can think of a chess AI system and say that it like wants to win. And I think when you look at like the very simple early chess AI systems, they don't kind of have like the anthropomorphic wanting that we want, but they have some sense in which they're trying to drive towards some outcome. And to the extent to which we're making systems that are more powerful that do that sorts of thing, that's like, you know, a very powerful capability that could be risky and similar to like, do AI systems know things or are they just parroting things back? I think that's partially a philosophy discussion that we could go for drinks and chat about for an hour or two, and maybe not resolve exactly what we think.
But I think there's a sense in which, they certainly seem to have an understanding of the world and we're seeing increasingly that they are able to potentially help with solving novel problems. You know, making discoveries that we hadn't previously made. They certainly helped me figure things out faster and synthesise information that they're looking at to come to conclusions that weren't in the things that they synthesised and were trained on, so I think there's something real happening there.
Can we just build an off switch as a lot of people just say, "Just turn off the computers", as in, what's the big deal? If we do create something that's smarter than us, why can we just not turn it off?
Yeah, I mean, so for the current systems now, yeah, it's very easy to turn them on. You know, they're clever and smart in a bunch of ways, but they're also dumb and they just live on server farms. You're kind of asking how you would be able to control and prevent a system that's much, much smarter than you from doing what you wouldn't want to preventing itself from being shut down. Well, it's kind of like asking like, "How would I beat, you know, Garry Kasparov or Magnus Carlsen at chess?" I don't know. If I was that smart, I would be able to know the moves that they would play, and play just as well as them.
There's a second thing where I think when we're talking about these very powerful systems, they won't just be... People often talk, think about intelligence as like kind of book smarts, but it's also, you know, social skill, persuasion, strategy.
And certainly, if I was imagining that I was a very, you know, intelligent AI system, and we don't want to anthropomorphize too much, that didn't have goals or values that were aligned with my creators, I certainly wouldn't just tip my hand to the fact that I was trying to accomplish something else in the world. I would probably be very helpful at, you know, transforming the world to be more prosperous and automated, and many of those things would be to my benefit. 'Cause the more that the world is automated, the more that I would have control over how the world works to the point where I no longer needed to necessarily depend on the humans who had built me, and so we might not be able to tell. An AI system that is very misaligned with us that is very powerful, it might be difficult to tell the worlds apart where we've actually succeeded at having them care about the things we care about versus not.
Okay, so on that point then and to your point that you say we haven't reached, you know, the point of these machines getting to that level, would the solution be to somehow, you know, build in those off switches or those like pull-the-plugs switches now so that we can't get to the point where the machines would override that?
So I think that trying to somehow set up the AI systems such that if they were that smart and acting in ways that we didn't intend that we could stop them, I think there's a sense in which that's a losing battle. What I do think that we can do, is as a society, a thing I often find myself saying is that we need to start building the legal, technical, and institutional mechanisms to broadly be able to impose restrictions on AI development, deployment, and diffusion. Both for like the dual-use risks that are coming up fairly soon, I think. And also, to manage these loss of control risks.
So I think if you're trying to like, build something into a very powerful AI system that you could turn it off when you didn't want it to be doing the bad things, that's kind of a losing game in the long run when we're talking about super intelligences. But we could build a broader system that allowed us to kind of not get in that position and be able to, you know, put the brakes on before we got ourselves there as a society.
Okay, well that is the perfect place for us to put the brakes on just for a second. We're going to take a quick break. But coming up after the break, are we already starting to feel some of the effects of the AI catastrophe? Malo stays with us to lay out the harm already being felt by AI straight ahead.
Welcome back. Our guest today on "Agents of Tech," Malo Bourgon, CEO of the Machine Intelligence Research Institute. So let's get back into it. We have been discussing the existential threat that is coming, but what would you say to the people that are arguing that some of the harms of AI are already being felt through things like unemployment, dissemination of... disinformation, unwanted surveillance, and the concentration of power in a few select companies and really countries for that matter? What about those harms that are already tangible?
I mean, I would say that unfortunately, the world is large enough to be able to contain many problems, and so there's not some sort of like, is it one or the other? I think, to be responsible, we have to manage the risks that are presenting themselves now, but we also can't ignore the large risks, I think, are over the horizon and may be coming very soon. So yes, all of the above.
Totally agree to that. It's not mutually exclusive.
Right.
A lot of CEOs of tech companies, especially at Davos, have recently spoken about the fact that they would love to slow down, but they've kind of not got any choice because China and other authoritarian governments will just get there first. So it's almost... There is just no point even attempting to do. So the choice is, that they either build reckless superintelligence, or they let someone else build it.
Build it, right.
So practically, what is the reality that we're dealing with? Like how would you solve that?
Yeah, so I think there's a true thing here and, you know, I think people sometimes assume incorrectly that folks who are worried about things I'm worried about think that like, you know, the US or one of the companies should just unilaterally stop. And I agree that that wouldn't help much of anything.
Yeah.
I do think that we have a large global coordination challenge, which is, that we need to find some way to have coordinated action globally, including with our adversaries, to solve these problems. I do think that like, a bunch of people think that's inevitable. I would like to make the claim that it just sounds like merely hard, and that by talking about it as inevitable, we don't get the chance to actually try it and see if we could do it.
And I think there are some parallels back to the '50s with nuclear weapons, where a lot of smart people thought that it was kind of just inevitable that we died a nuclear holocaust. But we did a lot of hard work. We had treaties, we had agreements, and we had some luck along the way, but we're still here.
Yeah. No, I totally agree. That kind of brings me onto my next point, which was about nuclear weapons and just every revolutionary technology kind of being seen to be an existential threat because it's the most revolutionary technology we've seen. And I feel that with nuclear weapons where... I've personally not really thought about them as an existential threat in the past 20 years. And there's countries like North Korea that literally have nuclear weapons. So, why do you think AI will be different?
I mean, you know, nuclear weapons still could be an existential threat.
Yeah.
They're still here and they can still be a problem. I think there's a bunch of challenges with the analogy to nuclear, but also it is an example of one of the reasons why we're in a safer position now than we might've felt like we were in the '60s just 'cause we put the hard work in to actually try and do the non-proliferation and all the challenges there.
I think there's something that's potentially makes the AI situation more difficult is that at least with nuclear, aside from nuclear power, there wasn't much other benefit to be chasing. Whereas with AI, there's kind of this incentive to be racing forward for the strategic and economic benefits of getting these very powerful systems while also kind of groping in the dark of never knowing whether the big risks that people like me are worried about are just over the horizon. And so that kind of makes the coordination and the game theory there a little bit more tricky.
Maybe there's a point to step back here where I think it is just the case that reality can be manipulated in a bunch of ways for good and for bad. And intelligence is a powerful tool to do that. And we've seen that as human society has progressed, we've gotten better at discovering more and more powerful things. And I think there's a lot more to discover. And so just in general, as humanity, we're going to have to get better at finding coordination mechanisms and wisdom to manage increasingly powerful technology that would allow us to manipulate the world in more fundamental ways that could be dangerous or extremely positive, and this will just continue. And AI is kind of like the ultimate tool in that respect, 'cause not only in itself, could it be a risk in that way, but it could also help discover more of these powerful technologies.
I want to see if you could kind of lay out some of the maybe very specific examples. Like if we can bring this down into some layman terms of the very specific examples that you think could be the demise of our existence because of AI. Are we're talking about, you know, disruptions to the food chain? Are we talking about maybe the creation of bioweapons?
Bioweapons.
Are we talking about, you know, mass unemployment, therefore, you know, faltering economies? What are some very specific ways that you see AI leading to this, you know, catastrophic, apocalyptic ending?
Yeah, so I mean, you know-
Please don't say all of the above.
I do think that, like you mentioned, there's a few risks along the way. I think AI will be destabilising for a bunch of reasons in terms of, you know, labour, in terms of powerful dual-use technologies, you know, bioweapons, that sort of thing that either individual actors could use AI systems to help them develop, or, you know, different state actors could also be doing that and that could just kind of create more instability in the world.
And then I think there's just this kind of... I call it the boss fight, if we make it there, if we manage to solve all those other problems or things go very quickly, of how do you actually, you know... I think it's the kind of a sense in which like, "We got a message that maybe aliens with really crazy advanced technology are showing up in a few years, and people are like, "Well, how exactly will they take over?" And I'm like, "I don't know, man. I don't how to tell. I don't know what tech the aliens are going to have. But my guess is, if they manage to travel here, we got some challenge ahead of us."
Yeah, there's a problem.
And so, maybe we should like not invite the aliens over until we kind of know what's going on.
Yeah.
Yeah. If I had to ask you what your maybe one biggest concern is, the one area that you feel AI could be the most detrimental, what would that be?
Yeah, I mean, I think worst case scenario, which is I think my central scenario, is that we build something that is very, very smart. We can't tell whether it cares about the things that we care about or not. Initially, it seems like everything is going very well, that it's helping us automate a bunch of how the world works and we're able to, you know, drive the material cost of goods to zero and everything looks like it's going very well.
But really, actually, this AI system's values and goals, whatever, you know, philosophy is hard about exactly what those words mean, aren't aligned with ours, and most of what it's actually trying to do is become non-dependent on our infrastructure such that it can then pursue the things that it cares about. And by the time we would have any chance of realising that, it would've already won, not because it's malicious, not because it's some sort of like terminator scenario or wants to like kill us 'cause it's evil, but because it just cares about weird other things that aren't the things that we care about, and it wants to pursue those things and its indifference to our existence. And so, we just kind of like, you know, fizzle out as a side effect of it, doing the things that it wants to do, which aren't the things that we care about.
Sure.
I also think the fact that you mentioned, the alien scenario so funny. It's because I was like scrolling through an "Ask Me Anything" thread by Peter Thiel about like, 14 years ago, and someone asked him, "How would you react if AI came about?" And he'd be like, "Yeah, the same as aliens. I wouldn't even ask about the economy." And obviously, the first question is, "The economy, what's going to happen to the economy?" But final question-
And I think there's a thing there that I would like, which is... You know, I'm focused on the big risks. But I think if there's kind of a missing mood where most people aren't even really imagining what it would look like to even get AGI, which technology, you know, hard exactly to define what that would be, but would kind of make all cognitive labour economically redundant. And even if we were grappling with what that meant for the world, I think we'd be in a better place 'cause people would really be starting to take seriously what this technology would enable. And so, I think that is true. I don't talk about the economy much, but if we were all freaking out about all the jobs going away, I think we'd be at a better place 'cause people would be taking more seriously what's about to happen in general with a very powerful technology.
Yeah.
Sure. What would change your mind? What would convince you that AI is now safe?
I mean, I think the big thing would be if we actually made some fairly deep fundamental advancements in our understanding of how to build AI systems. So I've alluded to it a few times now that the way we build AI systems right now looks more like growing them, than it looks like, you know, crafting them, or designing them, or not programming these things. I think if I saw some advancements where it felt like people really kind of succeeded at what the field's been trying to do for the last 70 years, which was, figure out how to craft these AI systems as opposed to kind of grow them in a very brute force way, I think there's still a huge amount of stuff that we'd have to do right and get right, and it's still a really tough problem to solve. But I would be feeling much more optimistic about our chances of solving it 'cause we'd have some kind of more principled sense of what we were even trying to do to make them safe. That would be a huge shift.
All right, before we let you go, our final two rapid fire questions.
So what would you say is something that's widely accepted as true in your field that you disagree with?
Yeah, I'm trying to think of like what my field is. Maybe, you know, one answer kind of more local to my cluster of people who are worried about these extinction risks. And so I think there's been kind of a growing narrative something that's a little bit too cynical. Maybe I'm just like too charitable or too much of a nice guy or something, but I think there's been too much of a kind of like, "Oh, the people running these companies are, you know, bad, immoral people." And I think that's not quite right.
I think that they're in a pretty crazy, bad situation with lots of messed up incentives, and most of them are like good people trying to do the right thing. But I don't envy the position that they're in. I don't think I would've ended up in that position myself. But you hear them talking about how they're kind of more saying out loud, like, "Save us from ourselves. We can't actually manage this ourselves and we wish we could coordinate." And so- maybe a contrarian take for my bubble is, you know, most these people aren't actually villains, they're normal people who are kind of trying to do the right thing but they're in a really bad situation. But I also think they're also doing a bunch of bad stuff.
Yeah.
And this kind of falls in line with, you know, your issuing of warnings. Where do you feel people will finally draw the line when it comes to bringing AI into their lives?
I don't know. I mean, I think that's going to be really different for a lot of different people. I saw Kevin Roose tweet recently, which I thought a thing was quite real that, you know, there's a bunch of people in San Francisco and other places who are playing with the state of the art of these tools and automating a bunch of how their lives work, and, you know, maybe are going to be coming increasingly fast and effective. And there's a bunch of people in the world who are kind of like, "I don't know, are the LLM models even smart? Like, I asked ChatGPT, and they don't even know about thinking mode." And they're like, "It still seems like it gets things wrong."
Yeah.
And so, I think there's no like easy answer to that question. I think there are lots of people who are going to dive in and get a lot of use out of these systems, and I think there's a lot of people who are just going to like draw lines, and I think some of those lines are going to be very good and principled and we should not have potentially AI in many of those places. And I think a bunch of them are going to be misguided. And that actually, there's a bunch of really powerful value that we can get from these systems while short of the risks that I'm worried about, and that we should be actually doing a lot to try and take advantage of those and figure out how to change society so that we can live in a more prosperous world where we can make use of the powerful tools we have at our disposal at the moment without racing towards superintelligence.
Love it. That's a perfect note to end it on. Malo, thank you so much for your time today. Appreciate it.
Thank you so much.
Thanks for having me, guys.
All right, so that was really interesting. I would've liked to have heard a little bit more of the specific ways in which AGI will equal the end of the world, but I do think he raises a lot of very important questions.
Yeah, and I think although it's hard to put out like a tangible framework by which like, this is the steps that we need to take, I think he did lay out the fact that the first step is that we all just kind of need to agree and cooperate, which right now with how turbulent things are- we're not cooperating.
We're not there.
Right.
And we just... Like he said about when there was a nuclear threat, that everyone needs to sit around a table and say, "Look, let's just stop and like share what we're doing with each other and try and sort out what's going to happen as opposed to just everyone racing towards economic and technological, like, dominance", which is what's happening right now. It's like, "I'm going to dominate you. I'm going to dominate you."
Right.
But I don't feel like we got a real clear answer on, you know, how do we pull back a little bit, you know? Okay, trying to implement the kill switch right now is going to be a moot point literally a year from now. Okay, then what can we do? While not stifling innovation.
Yeah, I've- I honestly don't know. I guess the answer is that maybe we can't really do too much, as in...
Well, that's terrifying. No.
I know. But I think what was a great point was about, you know, the CEOs of these companies, they're not necessarily the villains. And I really did think- about the parallel to someone like Oppenheimer where I wasn't a bad person, but the result of, you know, what that great scientists coming together to build nuclear weapons, and it killed millions of people, as in, that was a lot to sit with and it's unfortunate, and no one would want to be in that position as in-
Right.
Right. And perhaps, the silver lining, is that all of these, or so many of these tech CEOs are actually starting to come out and kind of echo exactly what Malo was saying, is like, "Wait a minute, this could really get away from us." And so, while we're building this, we need to be cognizant of that. Maybe that's the silver lining, I don't know.
Yeah, and- people aren't on the same page. Like some people think it's delusional to even suggest this, like Emily Bender. And some people think, "Okay, yeah, this is going to end the world." So if you can't get everyone to agree like, "Look, this is an issue," then you're not going to get anywhere. That's the first thing.
Right.
I hope. Yep. All right, well, we want to know, what do you think. How concerned should we be about a doomsday scenario? How concerned are you? We would love to engage with you and get your thoughts. Let us know in the comments below, and then make sure to like and subscribe while you're there. Our thanks to Malo Bourgon. Also thanks to you Laila, very much, for being here. And as always, thanks to you for joining us. We'll see you next time. Have a good one.