What AI Is — and Isn't: A Conversation with Emily Bender
Emily Bender, a professor at the University of Washington and author of "The AI Con", argues that "AI" is a marketing term and that the claims made for large language models do not stand up. She talks to Autria Godfrey and Laila Rizvi about intelligence, jobs, the workers who label training data, existential risk and the environmental cost of data centres.
Transcript
ChatGPT has 800 million users. OpenAI is valued at $500 billion. But our guest today says the whole thing is a scam. Emily Bender, author of "The AI Con," argues Artificial Intelligence is just a broad marketing term and "AI" just a label for unrelated tech, creating a false sense of an inevitable godlike entity. Is she a prophet, or is she just wrong?
Emily Bender is a director of Computational Linguistics Laboratory at the University of Washington, and she joins us today to argue that the people telling us that AI is revolutionary are the exact same people who profit from us believing it. It's a bold claim, but is she right? Laila Rizvi, my "Agents of Tech" co-host, is here with me in Washington today. And Laila, the big question that we're going to ask: are we being played, or, for my fellow millennials out there who might understand this a little bit better, are we being punked?
Oh. If we're being played, Autria, here, we're spending a lot of time, a lot of energy, and a hell of a lot of money on something that might not even work. We've seen this before with Web3, crypto, NFTs.
Remember those.
I mean, they were pictures that people were paying millions for. It was crazy.
A real flash in the pan.
And even metaverse, right?
Right.
Facebook changed its name to become Meta because the metaverse was going to be so big. I've not stepped foot in a metaverse. So it's going to be pretty awful if this is just a whole big scam. But on the other side, if AI is going to be an existential risk, we are looking at kind of a pandemic 2.0 level of unpreparedness here that it's going to be crazy. And even at Davos earlier this year, I think it was Demis Hassabis and Dario Amodei, the CEOs of Anthropic and DeepMind, who were talking about that we might only have one to two years left until we get to AI so powerful that it's an existential risk.
Right. They just wrapped up Davos. We actually have a little bit of that interview, so let's take a listen.
We might have AI that's better than humans at everything in, you know, maybe one to two years, maybe a little longer than that.
Maybe it would be good to have a slightly slower pace than we are currently predicting, even my timelines, so that we can get this right societally. But if that would require some coordination.
One to two years, that's not very long. That's a bold claim.
Yeah, but I don't know if it's just part of the theatrics, right? That they're saying, "Oh, we wish we could slow down," or if that's the actual reality, because these are the people who are working with these models the closest. So I'm very interested to see what Emily has to say.
It'll be interesting to see if Emily changes either of our minds today. But before we get to Emily, we want you to be a part of the conversation. Are the tech moguls lying to us? Is AI truly the revolutionary life-altering technology that all the tech bros brag that it is? Or are the sceptics the ones that are more likely to be on the right side of history here? Lay it out for us in the comments below. We look forward to engaging with you. Now, let's get to it as we welcome Emily Bender to "Agents of Tech."
Emily, thank you so much for your time today. It is such a pleasure to have you here with us on "Agents of Tech." We really appreciate it. We want to kick things off today with really the big question, that is: how intelligent do you feel these systems really are? You know, we've got the big tech leaders like Nobel Laureate Demis Hassabis of Google DeepMind and Anthropic CEO Dario Amodei making some big claims that essentially these systems will essentially replace the need for humans. Do you feel that they are intelligent enough to do so?
No. And I think in order to ground this conversation, we need to say what we mean by "these systems." I assume you're talking about effectively chatbots driven by large language models.
Correct, yeah.
Is there something else you have in mind?
Yes, exactly. Yeah.
And also, we don't have a definition of intelligence. And in fact, if you try to dig into that concept, it gets really problematic really quickly. But I can tell you that the only reason people sort of ascribe anything that you might call intelligence or cognition or intention, or capability even, to these systems is because they are systems designed to mimic how we use language. And language is a great way to grab our attention and to make us believe, in fact, project a mind behind the text.
So, Emily, I know you've said previously that there's no kind of "I" behind these systems; it's just something which makes them feel more human, makes them sound more human, and is actually a bit of a design issue. But we've now seen kind of evidence or research which has shown that these models are cheating, they're blackmailing people, and they've also been shown to play dumb when they know they're being tested. So how do you explain that?
I don't think any of that research actually stands up. What that research is looking at is text that comes out of these systems, and it is basically elaborate interactive fiction. That the scenarios that are set up are sort of the start of stories that allow the would-be researchers then to interpret the text that's coming back out of the systems as reflecting some kind of intention or desire that really isn't there. But that's easy to imagine because of how we interpret language.
But just following on from that, what do you make of Anthropic's recent research about introspective awareness, where these models are aware of their internal state before they've even kind of expressed that there's been a change? Wouldn't that suggest that they do have an internal state that they are aware of and that is separate from what they communicate to the evaluators?
So I haven't read this paper, but I would like to point out that Anthropic's, in quotes, "research" is basically blog posts that they put up. They are not participating in the scientific conversation; they're not engaging in, you know, peer review and all of that. So, you know, when we look at these artefacts on my podcast, we tend to call them "paper-shaped objects" rather than papers. But again, we do everything in language. And so the training data for these systems is going to include language that looks like introspection. And so the systems are going to be able to output language that looks like introspection, but that doesn't mean that they have anything that could be performing introspection. Like, it's just a category error.
You've said that AI won't take jobs; it'll just make them a little crappier, and that workers would basically just get hired to clean up machine slop. And we are already seeing the slopification of the internet of the workforce. But wouldn't you say that's exactly what we do with interns and kind of junior staff? And if AI is just a really cheap, fast intern, why is that degrading rather than liberating?
So there's a lot to say here, many directions to go, but it is true that many people who are at the start of their careers are exploited, that they are used to perform labour and not being paid very much for it. And it is also true that many companies believe that that same labour could be done through automation, but what's the point of an internship, right? It's not just to have a vulnerable person to exploit, I hope, right? Interns and junior staff are at the start of their careers, and they're gaining the experience that would allow them to grow in their careers. And anytime someone says, "Oh, we don't need that layer anymore because the senior folks can now just look at the output of the automated systems," well, where do you suppose the next senior folks are coming from?
That's a really good point. I mean, I don't think people really see it. Or a lot of people in the industry haven't really argued it in terms of career progression, right?
And Emily, something that you've kind of talked about with regard to this is how, you know, AI could really, like you were talking about, exploit specifically like the Global South.
Yes. Yes.
Can you elaborate on that a little bit more?
Yeah. So what's happening in these so-called AI systems is that there is human labour throughout the entire chain, right? So the underlying data that's used is the expressive work of people, sometimes for hire or sometimes in other contexts, that's been appropriated by the companies as their training data. And then there's these further steps, sometimes called reinforcement learning from human feedback, where, in order to keep the systems from outputting the worst of the worst, there are people who have to look at that output and annotate it as bad and bad in particular ways. And this is awful, gruelling work, and it is largely being done by people in places like Kenya and Venezuela and elsewhere where there have been economic catastrophes. And so people are desperate for work. The companies will come in and say, "We are setting up a tech campus here. You can work in the, you know, the flashy AI industry," and the people are brought in to look at just horrific stuff day in and day out without breaks, without mental health support, and without being able to tell the people around them what's going on.
And we know about this in great detail now because those workers are coming forward, and there's a wonderful project called "Data Workers' Inquiry" where you can see the inside of this from the workers' point of view.
So, can I play devil's advocate for just one second? Could the argument also not be made that by bringing AI and, you know, high tech into some of these areas that have very frequently been underserved, that you could not be elevating that population by providing these, you know, these new opportunities?
So these folks are making something like $2 an hour and being put through gruelling things that are harming their ability to have their relationships with their community. And that's what we hear from people like Mophat Okinyi, I think is his name, in the Data Workers' Inquiry. I'm sure that there are ways that tech companies could actually train up a workforce anywhere in the world, right? But that's not what's happening here. This is not training people to contribute to tech. This is asking people to look at horrible output of these systems so that the users, you know, those of us who are paying with our eyeballs or sometimes paying for subscriptions, are not seeing the most, you know, awful, bigoted stuff coming out.
Really sounds like the next generation of sweatshops, essentially. Okay, Emily, great place for us to take our first break. Coming up right after the break, we're going to be asking Emily to explain something she calls "Big Tech fan fiction." That's right after this. Stay with us. Welcome back. Our guest today on "Agents of Tech": Emily Bender, director of the Computational Linguistics Lab at the University of Washington. Emily, thanks for being with us today. We're going to get started with something that you and I actually align on here, but I do want to dive a little bit deeper into your perspective on something that, you know, you have called fan fiction and really the doomsday apocalyptic threat that AI can cause an existential crisis. What do you think?
Yeah, so I think you're probably referring to a podcast episode or maybe comments online where we referred to specifically the AI 2027 webpage as a kind of bad fan fiction. And the reason I call it fan fiction... Like, fan fiction is a lovely thing. I think fans engaging with the, you know, properties that they enjoy and imagining themselves in that world or imagining other stories in that world, that's wonderful. And it works because the world is already set up, right? There is whatever, the "Star Trek" universe, and then you can tell stories in that universe, and it's an easier kind of writing because you don't have to do the world-building yourself.
What's happening in that AI 2027 webpage is similar. So there is this world that has been built in the imaginations of people like Nick Bostrom and others in the Effective Altruist movement, where they tell this story, this future-looking story of what they imagine will happen with self-training artificial intelligence or whatever the thing is. And it is a story that's shared across this community. So then you get these things like the AI 2027 website, which is building fan fiction in that same world. And the problem is that it is presenting itself as something that's meant to influence policymakers instead of just letting people enjoy fan fiction.
Sure. But on that basis, do you think that AI does pose an existential risk?
No. So I mean, first of all, what do we mean by "AI," right?
I mean, like in this conversation, I would say the large language models that we're seeing now, and that how they have improved, because I think that three years ago, AI in the form of these chatbots or large language models, they struggled with elementary school arithmetic problems, and they were barely capable of writing a single line of code. Now, you know, Anthropic's Claude is basically writing all of the code for its new products. So Claude Cowork was written entirely by Claude Code. So, given that is kind of an exponential decade-long track record, why would that stop now?
Well, first of all, I'm not going to believe Anthropic's claims that Claude itself built this thing, right?
I was just going to ask you that question. Do you believe that Claude is actually coding itself?
No. And I mean, among other things, these systems don't have any agency, right? There has to be some input to get it to do something. So at the very least, somebody is putting an input that is provoking the system to output computer code, right? At that level, at least, it's not doing it on its own. And the kind of, like, what it takes to do software engineering isn't just outputting code, right? You have to do security testing and maintenance, and, you know, you'll get these statistics about how many lines of code were written with the assistance of these systems, even. And that's actually a terrible metric for productivity because you can write enormous amounts of code and have a system that barely does anything, right? So lines of code isn't a good metric.
And there is also part of that shared universe that the fan fiction gets written in: this notion that the core of what it is to be, you know, a thinking being that can make decisions, that can have goals, and so on, is somehow part of a trajectory that starts with elementary school math and passes through computer coding and onwards as if those are the most important things that we do.
Of course. But also think there's the argument that AI doesn't have to be necessarily superintelligent or all-powerful to be a huge risk. That powerful AI, as it's on the trajectory that it currently is, can still be an existential risk or a very, very, very big risk. Do you agree with that?
So I think there are definitely risks. Like, if we are going to put fundamentally non-deterministic systems in the middle of things that can have big impacts on the world, right? Connected to the stock market, connected to decisions about the power grid, connected to, you know, God forbid, nuclear weapons, then yes, things could go really, really wrong. But that would be because people chose to do something very dangerous.
That was going to be my next question, is if there were a very serious risk associated with AI, where do you think we might actually see it?
So the problems, I actually talk about harms rather than risks, right? So we've got outsized environmental impact, as this race for so-called artificial intelligence is prompting the construction of hyperscale data centres that are leading to maintenance of, you know, coal-powered, coal-fueled power plants. There's too many "powers" in that, but you know what I mean. That were slated to be taken offline, but they're still being run because we need to be able to have the electrical capacity to handle these hyperscale data centres. For what, right? For crunching these things. So that's a problem.
The people who are being harmed by these terrible jobs cleaning up system output that is horrible. The fact that people are losing their jobs, not because these systems really can do the jobs, but because they give managers the excuse to atomize, was once a good, solid role that was the start of a career or a mid-career job into gigafied work that can be sent overseas.
So, Emily, like, I've heard that people have done the calculations where they have compared the water consumption of AI data centres to your average fast food chain restaurant. And it's been shown that the "Colossus 2" data centre actually only consumes 2.5 times the amount of water as your average fast food chain restaurant. So is it actually as bad as people are making it out to be?
I mean, I don't think we should be irrigating golf courses in the desert either, right? But that doesn't mean that using water... And these data centres are being placed often in communities that don't have a lot of water, and they are drawing in, they require clean water, and so they're in competition for the drinking water, in competition for what's in the water table, and so on. And so, yeah, at a global scale, maybe it is not the biggest problem, but locally it actually can be quite a problem. And oftentimes, the deals are being made with a terrible lack of transparency, where people don't know even who the data centre is serving, and it makes it hard to push back.
Yeah.
Emily, I want to ask you if AI, and essentially the future of humanity, is going to be resting on the success of AI, is being overhyped, why have they had such a successful, essentially, marketing campaign? I mean, you have global leaders who are all in. I mean, if you look at the amount of investment from the United States and China alone, why have we been so quick to believe everything that Silicon Valley is telling us about: "If you don't get on board, you're going to get left behind?"
Yeah. So, you know, I'm not a social psychologist, but there's a few things that I can point to. One is that there's a tremendous amount of money behind it, and money means marketing, and money also means access to corridors of power. Another is, it would be lovely to imagine that we can just build something that will solve all of the crises that we're facing right now, right? It would be lovely to think that with just a little bit more compute, then yes, climate crisis solved, right? And so on. I understand that desire; it doesn't make it true.
And then on top of that, the part that I really can speak to from expertise is the way we interpret language means that when we see something that's coming out of these systems, that we see something that's producing well-formed, coherent-seeming text, it is very easy to imagine a mind behind the text.
Because, and I love saying this, but you might think that when you interpret something someone said, the person who said it took some words, packed them into... Took some ideas, packed them into the words, sent the words across to you, and you unpacked it from the words. This isn't what happens. When we are understanding what someone's saying, we are keeping in mind everything we know or believe about what they know and believe, everything that we have in common in our common ground, shared knowledge, and everything we know about what they must believe about their intended audience, which might not be us. And then against that background, we ask ourselves what must they have been trying to convey by choosing those words and in that order?
I'm just going to jump in here, Emily, as kind of a final thing on that basis of understanding, because Geoff Hinton, I would argue, isn't someone who is as influenced by the financial side of it, because he's not within any companies, kind of independent. And he was originally a neuroscientist, and he argues that the LLMs are understanding meaning by predicting the next word. So they're not storing sentences, like a string of sentences, and then pulling from that store of sentences and the next word, but rather they're converting the sentences and the words into features. And that's what they're storing, which he thinks kind of matches how humans operate, as in they've got a whole load of features, and then they will essentially output the next word based on the interaction of those features. So there are instances in which he said that there were models that hadn't been trained on concepts, but demonstrate a parallel between different concepts. So, concepts which both exhibit chain reactions, they could match them up as being similar. So how would you respond to that argument?
Yeah, it's very frustrating because Geoff Hinton really ought to know better. Like, not only does he have this background in cognitive science, but he also knows specifically how these systems are built, right? And yet he's taken in by them, and he has gone on this site. You know, when he quit Google, he took his fainting couch on tour and was, you know, all over the place on the media talking about how we have existential risk and so on. And more recently, he's been saying we have to train a maternal instinct into these systems so that they want to take care of us, which is bonkers, fantasy.
And you know, what's going on when you say it's got a concept that it wasn't trained on? Well, first of all, for this to be a scientific claim, we need clear, transparent access to the training data. And that is not happening with any of these big models. The exception is there's an outfit called AI2 where the training data and the algorithms, everything actually is open. So if you want to do these experiments to show that there's some, in quotes, "emergent" information that isn't in the training data, then you have to also be transparent about the training data. And that is not happening in these claims studies.
All right, Professor, before we get into our final quickfire rapid questions here, we do want to end on a positive note. Do you see any positive uses for AI? Where could AI truly, you know, enhance our entire existence?
So again, AI isn't one thing, and I... So disaggregating, I run a professional master's programme in computational linguistics. I train people to build language technology. So I do think language technology can be valuable. I benefit from spell check every time I try to type something.
You and me both.
Right?
Yes.
I think automatic transcription and automatic translation used carefully can be helpful, but they are not, you know, ever substitutes for things like professional translators in contexts where the content really has to be accurate, for example. Also, we're never going to say automatic transcription is good enough for deaf and hard of hearing people, and they don't deserve, you know, human care there. I don't see any beneficial use cases for synthetic text. That is what comes out of the large language models used as synthetic text-extruding machines because that's just a system designed to mimic the way we use language, which is effectively technology for fraud.
Yeah. All right. Laila, your rapid-fire question, you got to get to it. Go ahead.
Yes. So mine is: what's something that's widely accepted as true in your field that you actually disagree with?
Yeah. For the purposes of this question, I'm going to claim the field of AI as my field, even though I usually reject that. And say that it's widely accepted that AI or AGI is inevitable. It's a destination that we're all running towards, and it's just a question of who gets there fastest. And I really disagree with that inevitability argument. I think that the future's not yet written and that there is, you know, a lot of people out there who aren't interested in that direction, who would rather put their efforts towards building community and connection and human-based systems.
Yeah.
I am so happy to hear her say that because I absolutely agree. All right, my final question for you: what do you think, or where do you feel people will draw the line at bringing AI into their lives?
Yeah. So I don't do predictions, but I do hopes and fears, and I really hope that people will turn away from automated systems and towards other people. The scholar, Chris Gilliard, refers to these systems as "technologies of isolation," where the idea is that instead of talking to friends, right, if someone feels shy or whatever, and it's just easier to talk to a chatbot, they do that and have lost the chance to sort of build up the friendship, build up their personal skills and so on. So I really hope to see people more and more turning away from these seductive interfaces and turning towards each other.
I agree. And I really do think that we are going to get to a point where the pendulum is going to swing back the other way because we have created such a dystopia of isolated citizens.
I hope so. But you also do see literal AI psychosis where people rather have relationships with a chatbot.
Oh, yes.
It is terrifying.
It's wild.
I do totally agree with you on that point.
Professor Bender, thank you so much for joining us today. We certainly appreciate your time and your insights and expertise. Thank you so much.
Thank you.
Thank you.
All right, so I feel like you and I are going to have very different takeaways from this interview. I would love to hear your thoughts.
So I was trying to present, you know, the evidence and the research and stuff, and Emily just shot it down, as not true. Like, is that valid? As in, where do you go?
I do think that she makes a point that you can't take, you know, Anthropic's word at face value when they're the ones conducting their own research. So, you know, maybe if we had an independent auditor that comes in and conducts the research to test all the claims that they're spitting out, then I would put a little bit more faith in it. So I do appreciate her taking that with a grain of salt. But to your point, you're right. This has been going on for... You know, we're now more than a decade into this.
Yeah.
The progress is obviously there, so we can't discount it completely.
Also, it's a scientific field. As in, how do scientific fields function? Like, I understand Anthropic, DeepMind, that yes, they've got lots of money, but there are other AI researchers and computer scientists that reside in academic institutions. And it's not that the whole field is a load of bogus. Like, there are Nobel Prize winners. Geoff Hinton has also lost his, you know, head. And so and so has lost his head. As in, everyone just delusional, right, in the entire field that they're all AI pilled?
As I say, I think you and I both kind of come down, somewhere in the middle of all the, you know, tech bros claims and, you know, Professor Bender’s claims, there is a place and whether you like it or not, there's going to be a place for AI in the future. I don't know, though, if there is the, existential crisis that is looming, like they claim for those of us that don't, you know, get on board.
Yeah, I hope we're right in that regard. Yeah.
I also think it's difficult because, Emily obviously said that she doesn't consume, like, synthetic AI stuff, but it's hard to make a judgment if you're not consuming it. But at the same time, you might be subject to kind of complete psychosis by being too close to it. So, right. Can't win. That's what I'm saying.
Of course, you've got to be somewhere in the middle, I think, to have to have the best opinion. So we'll see.
We'll see how it shakes out. We would love to know what you think. How concerned should we be about a doomsday scenario? Are we getting played by Big Tech and how concerned are you? We would love to engage with you. Get your thoughts. So let us know in the comments below and then make sure to subscribe while you're there! Our thanks to Professor Emily Bender. Thanks to you, Laila, as always. And of course, thanks to you guys for joining us. We'll see you next time.