Fully autonomous agents should not be developed, says Hugging Face’s Margaret Mitchell

Margaret Mitchell, Chief Ethics Scientist at Hugging Face and co-author of the paper "Fully Autonomous AI Agents Should Not Be Developed", discusses the risks of handing control to autonomous AI agents. She explains why humans should keep a final say over what agents do, the security and privacy problems agents raise, and the role she wants governments to play.

Watch on YouTube

Transcript

Welcome to Agents of Tech, the podcast where we explore the cutting edge innovations shaping our world. We are diving into one of the most pressing questions of our time. What happens when we cede too much control to AI? DeepSeek, China's cutting edge, generative AI model that rivaled OpenAI's best has already shaken up the AI landscape. Well, now another transformative innovation is making waves. Autonomous AI agents that redefine what AI can achieve. In March, Chinese startup Monica introduced an AI system that goes beyond responding to prompts, it thinks, plans and acts independently. Manus operates with minimal human intervention, executing complex decisions across diverse domains.

This breakthrough builds upon OpenAI's deep research, Anthropic's advanced models, and the work of Perplexity, an AI powered search engine that provides precise, real time answers, signaling a new era of AI autonomy. And while that all might sound exciting, it also does raise some serious ethical concerns. What happens when we cede too much control to AI? How do we ensure safety, security, and accountability in systems that make their own choices? Well, today we are diving into these pressing questions with Margaret Mitchell, co-author of the paper "Fully Autonomous AI Agents Should Not Be Developed."

There is, sort of anthropomorphism and there's sort of trust, we tend to just sort of feel like things will go well and these systems will act like, you know, morally grounded people. Even though there isn't really good reason to believe that.

We'll discuss the risks of full autonomy, the security threats that AI agents pose, and whether governments should step in before it's too late. If you are enjoying our show today, please make sure to share, like and subscribe to Agents of Tech. This helps us shine a light on cutting edge research. Thank you. Now, I am delighted to be joined in our DC studio by my colleagues Laila Rizvi and Stephen Horn. Pleasure to have you both here today across the pond.

Great to be here again Autria. Yeah, great to be here.

Absolutely. Okay. Let's just dive right in. Stephen, the jobs market is set to be transformed by AI. As a CEO and an entrepreneur, what are you already seeing? And is Manus do you think the beginning of the new AI revolution, or do you think it's more of a ticking time bomb?

So I think it's probably both.

What a diplomatic answer.

I think it's the beginning of the revolution and it's a ticking time bomb. It's what Geoff Hinton has warned us about. Which is autonomous agents. And the difference when AI has autonomy, what AI will want to do. And the first thing AI will want to do if it does have autonomy, and let's see what this is all about first, will be not to have us. Right. Because we would just, you know, slow it down. So. So it is a bit of both of those. It's exciting. AI because all the drug discovery.

But also we're talking about self-driving cars, you know, there's a lot of technology here that, with being more autonomous, is going to save us not just time and trouble, but lives and and increase drug discovery, etcetera, etcetera. But there's also and this is going to be my question to you Laila, big ethical considerations here, safety considerations, what happens to all the data, you know, and what happens to the people that it replaces. So where are you on this Laila from a sort of ethical standpoint?

So I think I'm going to talk a little bit more cosmically here, maybe going well beyond where we are now. There are huge dangers with autonomous AI if it reaches a certain point, because unlike a human being, which people argue, well, if I've got an AI agent, it can do a job better than a human. It has a higher rate of accuracy, it's more productive. It's more efficient. Why can't I use that?

Humans are also fallible, but unlike humans, the physicality, they're very different. AI is digital, which means it can one it has no transfer loss in terms of knowledge. So it can replicate itself and share that knowledge with other AIs. And you just you can't simply just switch it off. It's like a computer virus, you know, whereas with a human being, you can physically restrain them. You know, we're analog, we're not digital. Whereas you don't have that physicality with an AI.

So you are getting into very, very dangerous territory. And I think this is also something that Geoff Hinton touched upon, that AIs have huge advantages upon us in terms of being able to share and replicate. If you don't understand what the agent is doing, so we're seeing this a lot in coding. There are very, very proficient coding agents, but people are using them and they're not secure. The code that they're releasing, the software they're developing doesn't have any robust cyber security.

But look at self-driving cars. Look at Waymo, for example, which is AI mixed with, with the physical agency, the car, and we're all sitting there saying, oh my gosh, right. What could happen. These things could crash, whatever. But, you know, so do human beings. This is the way of the future.

Yeah, I agree with you. If statistically they are causing less danger than humans, what's what's the problem? But that's talking about now. We constantly are talking about AI. There's going to be technology that evolves from that and concurrently which we can't imagine because we've not seen it.

But do you have concerns about data security?

I don't know, because if you ask someone, you know, 100 years ago, how would you feel if you had this device where it can basically predict all of your moves, what you're going to buy, what you're going to do, they'd be horrified. People are kind of willing to give their data if it means they get more efficient, more productive. If their life becomes easier.

Exactly. I mean, think about the people with, you know, the Alexas in their home now. I mean, you literally invited a recording device into your home where 50 years ago, you know, you don't want I think anybody looking at you.

Exactly. I think it's the price you pay for a higher quality of life where you can do more with less with technology.

So let's get to it. We are delighted to be joined now by Margaret Mitchell, Chief Ethics Scientist and Researcher at Hugging Face, a machine learning platform and co-author of the paper "Fully Autonomous AI Agents Should Not Be Developed." Margaret, such a pleasure to have you with us today.

Thank you. Such a pleasure to be here, I appreciate it.

Well, so I want to dive right into your paper, because you argue that fully autonomous AI agents should not be developed as is the, you know, title of the paper. With companies like OpenAI, Google and now, Monica pushing these AI systems with that plan that can execute tasks, even interact across multiple platforms. Have we already crossed the Rubicon here?

There's sort of gradations of autonomy when we talk about fully autonomous. Most of the efforts I'm aware of are in the the higher edge or the higher part of the spectrum. Things that can read and write and execute code are already being developed. Most of the projects I'm aware of that do this have a sandboxed environment, which means that there's some amount of secure code execution.

So one of the main issues that we're hitting on in the paper is that the, the sort of more autonomous the system is, the more we are ceding our control. And so it's critical to always maintain some level of human control. And so what I'm seeing with, with the Hugging Face work where I work, Manus is another big AI agent that's come out, talking about how it's fully autonomous. It's still not at that level where human control is fully ceded. It's still the case that, there is some amount of constraints on what it can and can't do.

With Manus in particular, which is really been a big, buzzy one in the AI world, you can actually see the code that it's going to execute. You can stop it. And it's within these secure environments where you can do, Docker or E2B, where you specify specifically like the dependencies that it can pull in.

So why is it that as humans, we think that it's a good thing necessarily that, the humans have to have the last word or the humans have to have this constraint. Take self-driving cars, for example. You know, humans aren't particularly good at driving cars. You know, we have thousands of road accidents all the time. And you look at maybe the experience of something like Waymo, perhaps machines are going to be better at driving cars than us. So why do we need, the human, you know, supervision, if you like?

Yeah, it's a really good question. And there's a few sort of dimensions to that. So the first dimension is that we're living in a human world. This is a human world, not a computer robot world. Right. So like humans are sort of controlling society. The society we live in and ideally computer systems are not controlling the society we live in. I guess. You know, to some extent, maybe.

But, you know, if we want to sort of be at the top of the food chain here, then it's important for us to sort of stay within control of that. So there's that aspect to it. So if computers were controlling us all, then maybe it would make sense for computers to control everything we do and have like full say over what we do. And so this is the sort of like higher level concern about whose values are going to be prioritized. That's one part of it.

The second part of it is that you brought up the autonomous driving example. I really like that example because there is clear utility for people with disabilities, or older people who don't have their license anymore, this kind of thing. I'm a big fan of autonomous driving as an assistive and sort of augmentative device. That said, humans living in a human world see things differently, process the visual world differently than computers do.

And so, you know, there's been examples of cars, autonomous cars that, you know, drag a person under, while it's still starting to go, and that is the kind of thing a human could see and stop. So it's sort of like, why would you remove that ability? Like, why would you remove the ability to stop the car? And so in a way, it's, you know, it's like two minds are greater than one kind of thing. Except I would say that the computer system is not a mind.

Margaret, I want to, talk more about coding and coding agents. And with OpenAI having released its new kind of dev tools to create agents, could this make it even harder for us to control risks which come with agents? I mean, we've got so much talk around "vibe coding" now, right, where people are creating, nuts software, which has huge cybersecurity risks. And I imagine we're probably going to end up flooding, you know, the internet with quite rubbish software which isn't reliable, isn't secure. So how do we actually ensure that security is being built in to these agents from the beginning, rather than as like an afterthought?

So this is one of the reasons why secure environments are really important to have. You could potentially have a system that writes and executes its own code in random locations on the internet, sort of like you're mentioning, and that can have lots of malicious, sort of outcomes or problematic outcomes that harm people with privacy breaches or reputational harm, that sort of thing.

When you have a secure sandbox around what a system can do, you're able to say what applications it can and cannot use, the kind of code that it can and cannot execute. But I should say that we don't have watertight security like that has not been developed yet. And so I would argue that if we want to have more and more autonomy, for things like vibe coding or whatever it is, then that should be developed alongside, secure environments or something like agent arenas.

This is just a term I made up, like agent arenas is not a thing, but I'm trying to like capture a concept here that corresponds to, watertight, airtight constraints on how a system can and cannot act. Because without that, you know, a lot of these systems are built on large language models, which are unpredictable. I mean, they're designed to be, like, stochastic, to put out different kinds of outputs in different environments that we can't necessarily predict. And so that opens the door for all kinds of problematic code to be created as well, that we can't foresee. We need to be able to put in place very specific controls about where the agents can execute code and where they cannot, and then what kind of code can be executed.

My formative years I worked at the BBC, I worked at the BBC as a producer. And the BBC had a very clear view of its position in the world and the position of its skill base and the position of television and broadcasting. And in those days, the view would have been very much that the production of media is only something that can be done by professionals, can only be done in this sandboxed environment that you're talking about and can only be broadcast in that kind of way because of all the harm that could be done otherwise.

Now you look at social media today and look at the fact that you know, there are literally billions of people with cameras all around the world producing stuff at will and throwing it out there wherever they want in the same way, in random bits of the internet. So, the exact model that you are talking about not wanting to happen, happened in our world.

And, like, so what? People are creating content for other people. It's within the realm of society, within the realm of what is actually sort of desired for, for people's interest or just, because they're trying to be better artists that sort of thing. And so that can't be equated with a computer system. People are creating content for other people. And it might be the case that a broadcasting corporation wants to have some sort of control over the quality. But ultimately society isn't built on broadcasting corporations. It's built on people.

It's part of human evolution. It's part of society. We're always innovating new things and working together and sharing. And that's just sort of what we do as people. That said, people still put out really problematic content, and I would argue that that isn't a net good. One of the things that is a massive harm for women on the internet is deepfake, pornography and what's called revenge porn. So these are massive, massive issues that I would love to see more control of.

It's not all good. There is still need for some kinds of constraints when it comes to content that harms other people. And then the thing about computers is that we don't know when it might harm people and when it might not. We don't have as much insight into that as we do for people, because we don't think like computers. We're not computers, right. And so it's a lot harder for us.

Kind of piggybacking on that, The Signal Foundation's president, Meredith Whittaker, also recently warned that, agentic AI poses a major privacy threat. So I think you've done a really good job of kind of laying out the argument as to why we need these constraints and, you know, why we need to keep this reined in. But my question to you is, if we do continue to not allow these, these machines and AI to have access to sensitive data like emails or, browsing history, financial accounts, those kinds of things. One will that continue to prevent these types of systems from becoming a security nightmare? But two, does it kind of handcuff the ability of these systems to provide the most accurate data?

Yeah, it's a trade off. I mean, you're hitting at, I think, one of the fundamental tensions here, if we want to have very flexible systems that can do whatever we want, with respect to the things we kind of say online, you know, personalized towards us, right? Taking into account our personal information. Then we have to give it access to that in order to be wholly personalized. But that also comes at the risk of really problematic uses of our personal information.

So for example, it could be possible with current AI agent technology for a system to, go through our email, not just our own email, but the people we're responding to. So not just reading what we've written, but reading what's in our email from other people, and then to post private information shared in those emails to social media. So there's social media agents right now that help to write, you know, social media posts if you want to be an influencer or that kind of thing. There are agents that go through emails and summarize.

And so if you connect those without having a final say of like, yes, post this on the internet. Yes, post this on social media. Then you are running the risk of having very highly private information and not just yours, the people you talk to as well shared on the internet without their consent. And so this, this personalization, versus potential privacy breach is a really serious issue. Whenever a system is going to share content on the web, it's really important for us to be able to see what it's going to do.

And I would add that we we need to start changing the internet, sort of updating it in light of how AI agents work. So that, you know, certain emails can't be accessed. That kind of technology is possible to do, but doesn't exist yet. We need to be developing this other kind of technology that helps keeps those risks in check.

How do we prevent people from conceding complete control to AI because they want the benefits of that kind of full autonomy? Because I feel like we're currently in a space, like you said, we've not developed that technology to protect against those risks. But for example, with MCPs Model Context Protocol, with Claude, you can give the AI access to your kind of hardware, your terminal and people I mean, although the security risks are high, I mean, people are doing it because it's easier. It's easier to get Claude to push code to go into your files, amend them, etc. So how do you stop people from releasing the genie and the genie causing disruption to everyone?

This is part of why I'm speaking out. Why Meredith is speaking out. Because education is really important here, because people will just naturally sort of give up their rights, give up their privacy, give up all these sorts of things, in pursuit of ease and flexibility and that sort of thing. And that's combined with the fact that there is, sort of anthropomorphism and there's sort of trust, we tend to just sort of feel like things will go well and these systems will act like, you know, morally grounded people. Even though there isn't really good reason to believe that we just sort of do that, as people we do that.

And so, there will be people who, inappropriately or, you know, to their own detriment, give up their private information. These things. So that is likely to happen. But there are a lot of people who just don't know. And so, I mean, that's why Meredith, who she and I used to be colleagues, I have so much respect for her. Everything she says is like gold. That's why she's speaking up. That's why I'm speaking up. Because education is really, you know, a key here to make sure that we mitigate as much foreseeable harm as we can.

And what specific areas do you think that the benefit of using kind of an autonomous system significantly outweighs the risks?

Personally, I am terrible at forms. And in the US it's tax season. I see a potential in a few different ways. So one is just these really tedious tasks that we hate doing or that we don't have time to do, and you don't have to fully cede, you know, control all that. This can be kept within environments that are secure for your information, another area that I always think about is the assistive and augmentative technology.

So, you know, there's a lot of people who can't, type on their keyboard for a long amount of time because they have, hand, you know, mobility issues. There are people who are blind, obviously, who's experience with the internet is fundamentally different than people who are sighted. Agent systems could help make these things a lot better. And now there's agent systems that can actually, like move around your cursor, as well as typing on your keyboard so it can actually move around windows for you click around on tabs.

And then also for, people who are aging and elderly, you know, a lot of the internet is designed with younger people in mind. So just being assistive for those kinds of things, that sort of insider knowledge of how to use the internet, could be something that agents really, really help with.

Common consensus has been previously that, you know, certain jobs like, you know, artistic creation are for humans and, you know, not computers. And yet we have a lot of issue at the moment, particularly in the art world of, AI tools and the use of AI tools to create art. But if you look back, a lot of people are calling this the fifth or whatever it is now, Industrial Revolution. Around 1800 or 1790 or whatever, the most prized jobs in a community were handloom weavers.

So they were people who had hand looms and they wove things themselves and, you know, and you got to the point where a very skilled handloom weaver could do four looms at the same time. Then you come by 1810, 1820 all those people who were four loom weavers have now lost their jobs. Right. Because the factories with thousands of looms have replaced them. So if we are getting to the point now where, you know, computers or AI systems, however you want to describe it, agents can do some of the tasks that were highly rewarded tasks how are we going to prevent economic displacement?

I think you're hitting the nail on the head with the analogy to the industrial revolution, because we saw that as some jobs disappeared, other jobs appeared. And so, you know, with innovation comes new potential for new kinds of technologies, new kinds of, things that are needed for people to do.

So if you were a four loom weaver, you know, you were being paid a lot of money if you were a 15 year old who was now taking that job, you were paid virtually no money because society by the use of technology could change the way it remunerated. So I'm not saying that the jobs won't be there for, you know, for artists or whoever else, but will they be at the same level of pay?

Yeah. I think this is, sort of a function of, of governments and things like capitalism, and less whether or not the jobs are going to be there or not. So I, I agree with you. I was sort of going to make this point that I'm concerned about the companies that are, developing, you know, claiming to develop AGI, that sort of thing. Will end up gathering a lot of profit based on the work of other people who lose their jobs.

And there is no, rebalancing so that the people who have lost their work to these automated systems are able to recoup their losses or able to have some other high paying job. And I think this is where governments tend to come in to try and, you know, level the playing field. I think this is where, you can come up with new kinds of innovations that other people can take on as jobs, and you pay well if you want to, if you want to value that. But I'm concerned that there isn't an incentive to do that. And so I do see I see a lot of risk with centralization of power and centralization of wealth.

Sure, and on that point Margaret, if governments would step in to regulate AI today, what would be your top recommendation? Do you think it would be banning certain levels of agentic autonomous AI, or would it be a more nuanced approach? Because, you know, people have this fear of a totalitarian government coming in and using AI progress as an excuse to you know, exert their own agenda. But what side of the spectrum would you kind of recommend?

I've tended to take a rights based approach to regulation, or rather, I should say, to my view on regulation. And when I talk to, congressional leaders and stuff, I, you know, there's the carrot model and there's the stick model. I think governments are well set up to protect people's values, to protect people's rights. And going the other way, the stick way where you say, you can't do this, you can't do that. That can end up, creating more harm than intended. And also creating more, you know, ill will amongst those people who are, being told they can't do this or that.

And so they are two sides of the same coin. So like saying, you know, children must be protected. So I would love to see a world where children are not represented in data. Children can't consent, right? Like within the US, you're not the age of consent. You cannot consent. And so ideally, you know, you would be able to say, like, we have to protect the rights of people who are not able to consent. And if there isn't consent, then they can't be included. But that's a view of what their rights are. That's prioritizing, safeguarding individuals as opposed to centering on how to stop technology companies from doing this or that.

It's like a person centered approach, a rights centered approach, as opposed to what technology should we stop? And so for me, you know, I'm a computer scientist, so I don't really know policy and government. But just from working within the technology sector, seeing how this technology affects people, it would be very, very useful for governments to play a role in protecting the rights of people where corporations are not able to because of market incentives or because they don't want to.

It does seem like in the United States, though, that our lawmakers are starting to wake up to the potential harms that not AI, but all technology. I mean, we're looking at, you know, like social media, for example, and the effect that that has on young people and adolescents. And we're having lawmakers, you know, finally start to put some, restraints on Meta and Google and some of these companies. What do you think the the current temperature or the current, political climate is when it comes to AI and trying to keep a thumb on AI so that we don't essentially not to steal Laila's term, but let the genie out of the lamp?

There's a very strong narrative, that's put forth in particular by Big Tech corporations that regulations stifles innovation. As someone who has worked in the tech industry, I definitely see that point because essentially because you don't understand how the technology is being developed. And the relationship between what you're developing and what gets deployed, it is possible to create, exactly the outcome you don't want, or to stop innovation on something that would bring about, really positive outcomes.

So like a really good example of this is like inferring gender. So there's like an argument that says you shouldn't be able to infer gender because this is a privacy violation. You don't have consent to do this. But it turns out within machine learning, if you want to guarantee that something discriminates based on gender, then you have to have some representation of gender.

But that said, I also think that regulation can stimulate innovation. And a great example of this is differential privacy. So within machine learning we have privacy guarantees that within this large database of information, any information that we extract from it will preserve people's privacy. And that comes from a constraint on having information be private. That is, you know, it's sort of like, necessity is the mother of invention kind of thing that when you put a goal, if regulation can put a goal to hit, then I think it's possible to innovate towards that goal.

I appreciate that governments are paying deep attention to this. This is something that, you know, I've wanted, governments to pay attention to from a from a positive lens. But I do think that governments shouldn't just be in the position of saying, no, you can't do this. No, you can't do that. But actually thinking about what kinds of really positive things would I like to see from technology and setting goals for for companies and innovation to hit?

All right. Final question for you. If we look ten years down the road, you know, we're a decade down the line. What is your biggest fear about AI development, but also what is your biggest hope?

So one of my biggest fears is is around the centralization of power, which we talked about earlier. I really, concerned about there being more and more of a wealth divide. The haves and have nots, where there's a very small, section of the population that has accumulated massive wealth built on the work and labor of so many other people. And I don't know that that will be avoided unless governments can step in to help level the playing field a little bit.

But I do see a lot of potential for assistive kinds of thing. So like speaking of leveling the playing field, there is so much potential for technology to help people all be on the same page. You know, all be able to share their ideas in a way that everyone can embrace and understand and speak to. Instead of being trampled on. There's so much potential for technology to supplement and translate between people in a way that really helps empower everyone to work together. And so I would really, really love to see that happen. I think it's possible. And, you know, I'll be working on trying to push that forward. But yeah, we'll see what happens. Hopefully that'll pan out.

That's awesome. That is a great note to leave it on and wrap up with today. Thank you so much for your time. We really appreciate it.

Thanks so much. Thank you. Thanks. Thank you.

All right. Well, she was interesting.

Very fascinating. A lot to unpack there.

Yeah, a lot to unpack there. So, so I loved Margaret, but you know, it didn't make me feel particularly optimistic. Right. So for a couple of reasons, I have to say. I mean, let's talk about the privacy thing, first of all. And we're talking about putting these, you know, sandboxes around this kind of software thing. And what have we learned over the last ten years is that, you know, if the product is free, you are the product. And everybody loves being the product, right? There's not a single person I know sort of says, oh, I'm not having anything to do with Facebook or Instagram or, you know, even TikTok. And those companies make billions out of us and we very, very happily say my privacy. There you go.

And then on the second point, on the, you know, on the wealth thing, yeah, there will be some big winners and they will be they will be in the computer industry in, San Francisco, Seattle and, you know, in California, they will not be, you know, in parts of traditional industries that we've known and the idea that government is somehow going to regulate. So the well-paid people in jobs that go are going to get other well-paid jobs that they have no skills for. How is that going to work?

Yeah, I kind of totally agree with you. I agree with you to a certain degree. Whereas yes, when we sign up for Facebook and we sign up for Instagram, we check the box and the terms and conditions and we and you know, we understand that we are, you know, freely giving up our data and that Meta is making money off of us and that we don't control those things. However, I do in my head have a very different viewpoint between what is put on social media versus what is in my email and what's in my inbox, what is in my, protected portal between me and my doctor. So I think when it comes to that kind of sensitive data, that's something that I don't want AI to have access to.

To her point, I do appreciate the arguments that she's making when it comes to, you know, sandboxing certain data and certain, you know, information about people. What she mentioned about regulation that a lot of the time the government doesn't know what technology is regulating and how.

Oh, yeah, no, we we are a reactive entity in that regard. It's so hard, how do you build systems to do so. Because you inevitably do stifle innovation. Say we had government regulation from the onset of social media because of the potential harm that it might cause. A lot I think a lot of social media wouldn't even exist. Now, there are a lot of upsides to social media. It connects people. It brings together people, you know, that may have lost touch or it, you know, might help grow a small business. But there have been downsides to it.

And so the regulations have been very reactive to that. And now they're trying to put these safeguards in place to protect, you know, the people that she was, you know, talking about in particular that, you know, might be the most subject to harm. So I think having a reactive viewpoint in that regard is a positive.

I know. But the problem with this is and I don't disagree with you. Right. So I do agree with you. But the problem with this is, is that the rate of change is is incredibly fast and is driven by a profit motive. Right. For those companies to make hundreds of billions of dollars. Then donate to those same lawmakers who might be the ones trying to implement. By the time you're trying to put the genie back in the bottle, you a thousand genies and 100,000 bottles and that job, is that impossible? So the idea that and I agree with you on social media, but we are where we are. And the idea that that's not going to be the same, I just think is wishful thinking.

Yeah. When you're talking about the, the rate of expansion with AI and, you know, these we talk about all the time, all these new companies that are popping up. It's like we're drinking from a firehose right now. It's coming at us so fast and furious. So that does it for today's episode. If you enjoyed our discussion today, do not forget to like and subscribe and share as this really helps us to improve the show and create more content just for you. Thank you for joining us today. Thank you guys for being a part of today's Agents of Tech. Until next time, goodbye.

Goodbye.

Agents of Tech is brought to you by WebsEdge. We would like to thank our studio engineer, Madison Birch, our editor Abbie Harries and producer Cath Sheehan.