Can AI Become the Scientist? James Zou on Virtual Labs at Stanford
Professor James Zou of Stanford University explains his virtual lab, a team of AI scientist agents modelled on his own research group, with an AI professor and AI students who have expertise in fields such as chemistry and computational biology. Recorded at the ISMB/ECCB conference in Liverpool, the conversation covers how the agents designed nanobody candidates against recent Covid variants, how they train themselves in an AI school, and why human experiments are still needed to check their work. Laila Rizvi and Stephen Horn are joined by scientist Luke Yates.
Transcript
AI isn't just changing how we do science, it's starting to take on the role of the scientist itself. At the forefront of this shift is Professor James Zou. At Stanford, James leads groundbreaking work on virtual research teams powered by AI. His lab is creating digital scientists with specialized roles, systems designed to collaborate, question and accelerate discovery in ways we're only beginning to understand.
This is Agents of Tech, coming to you from Liverpool, UK, where over 2000 researchers, technologists and data scientists have gathered for ISMB/ECCB, the world's leading conference on computational biology. I'm Laila Rizvi and I'm here with my co-host Stephen Horn and Luke Yates, a friend of the podcast. How are you guys doing?
We're doing great. Isn't it fantastic to be here in Liverpool? See, you're a bit skeptic about this stuff, aren't you? This agentic AI. I can see, I can hear it in the tone of your voice.
The virtual labs, I don't know, I mean, can you replace scientists? I mean, I'd love to ask you, Luke, because, you know, you're in the science world yourself. You're a scientist yourself. What do you think? Do you think we can have AIs replacing your researchers?
I mean, it's a great question because intrinsically, I feel threatened by the idea of being completely replaced by AI and all those years of experience and training, getting to a very proficient level of being a research scientist, being kind of taken away, if you like. I guess this is what most people feel like when AI takes their jobs, and I kind of didn't expect us to move as quickly towards replacing really high level jobs like science and engineering so quickly. I thought the more mundane jobs like booking a plane or a car via agentic AI would be the first things to be made possible. But this is really an interesting, I think, really interesting use of AI, and I hope that they, that it isn't the end of the research scientist in the lab, but it helps augment us in some way.
But what you were a bit skeptical about, isn't it, is that individual AIs can have, like, a different role so that, you know.
Yeah, I, I don't quite understand the concept of a virtual lab, as in, it's completely new to me. So I'm looking forward to getting Dr. Zou to explain what that actually consists of. Because is it a replication of what a human lab looks like, or is it more general than that? I don't know. I mean, what do you guys think of virtual labs?
Well, I think running my own lab, I would want to build a multidisciplinary team of experts from different, different places with different perspectives and different experience and different training and maybe kind of experts in different techniques to answer a fundamental problem that I'm trying to address. And, you know, we think very carefully about how to build and blend the best type of team for this job. And, and you wonder where that's going to go, if you could in fact have a multidisciplinary team based on AI algorithms.
But I'm very excited about the idea of maybe having an AI sales team, you know, and I can see them all sitting around the table as we speak. I'm going to provoke you a bit, Luke. What are your skepticisms when it comes to this? What are you skeptical about?
I guess it's more to do with the joy that we get from being around a table and talking to other scientists and having that stimulating discussion, those kind of moments of inspiration that spark an entire project. And I don't know if we might lose that if you hand those kinds of discussions over to many different agentic AI entities who have those conversations on the cloud somewhere or on a server somewhere, not in the round, as people do. I wonder you might lose some of that, I would say, joy, that's the most joyful part of, you know, science, that collaborative nature, that human connection to try and address something fundamental and to try and discover something new. And so I guess that's my skepticism. Not that they're not useful, but that they may take away something that we value in science.
Whereas I think in a business setting, we'd love less meetings. If AI could take our meetings. I love the idea that these little AIs that I'm already visualizing, and they don't exist. Okay, so I'm visualizing something that doesn't exist talking to each other, right? And they're not going to be doing that. Right. They're not going to have different AIs talking. It's going to be a whole different way of doing it. And in the business setting we can't wait. We can't wait. Right. But in a scientific sense, I get the difference. But we'll have to see what James has to say. Exciting.
Yeah, I can't wait. I mean, this is the first time I've encountered something like this. And I'm also interested in learning how we can extrapolate that to other fields like business that, you know, can you build these specialized little agents which have these different roles like sales or an immunologist? So is it the same or is it a completely different kind of ball game? But looking forward to it.
Yeah. Well, we'll find out.
We're now delighted to be joined by Professor James Zou from Stanford University. James, thank you for being here with us.
Great to be here. Thanks for having me.
Great to have you, James. So I'd like to kick us off with virtual labs. So we're all very excited by this concept and don't really understand it. So could you explain to us kind of what a virtual lab is and your work around it?
Yeah. So the virtual lab is basically a team of AI scientist agents that sort of mimics my own physical lab. Right? So there's the AI professor agent, and under the professor is actually a bunch of different AI students with expertise in chemistry or computational biology, and they work together with the AI professor to tackle these open ended research problems.
So the sort of the impetus for the virtual lab actually happened a couple of years ago because as a professor at Stanford, we have a lot more requests for co-operations than I have time to tackle physically. And we thought, oh, wouldn't be great if we actually had, if we can actually duplicate my lab infinitely many times using AI models. So they can actually then have sent the AI to do these different projects. Right. And then with the recent advances in these language models like ChatGPT, we actually got much better at creating AI scientist agents based on these language models. So that's how we actually created the virtual lab.
And do you have an example of like a success with the virtual lab, like in a context?
Yeah, yeah. So actually one of the first problems that we asked the virtual lab to tackle was to help us to design new binders to the recent Covid variants. So these are binders that could potentially be possibly vaccine candidates. So it's quite a challenging open ended research problem because, you know, it's actually still very much a problem of, if you have a new Covid variant, how do you actually find good binders that can bind to these variants?
And the team of the AI scientists agents, right, so they basically came up with a new idea of designing a new computational pipeline that can be used to optimize nanobodies, which are sort of like small cousins of antibodies. They're small proteins, and they have this very clever workflow to design and optimize these nanobodies to bind to the spike protein of the Covid variants. So they implemented all of that, and we then did experiments to actually make these nanobodies in the real lab and then test them in the real life. And we actually found that they were actually very good, very promising candidates.
What's the difference between, as you said, that, you know, it's a different approach to, to a general sort of AI large language model questions. What's the difference between your approach and just asking that same question to a traditional large language model?
So that's a good question. So if you just talk to ChatGPT or your favorite language model, it's actually quite good at talking back to you in terms of text and also quite talkative, or likes to talk a lot. But when we want this model attached to actual science, it's not enough to just communicate through text, but we actually want the models to actually do real work. In this case, we, in a virtual lab, we actually equipped these language models with additional tools, for example, tools like AlphaFold, Rosetta, and other tools. So these AI agents, they can actually think about, okay, how can I use AlphaFold to help it to design these new proteins or new nanobodies?
So would you say that when people are building AI teams or trying to use AI, the best way to do it is to kind of create specialized AIs which can access specialized tools?
We found that to be a quite a powerful approach. It's also analogous to how when humans do research, often there are different expertise, right? Maybe if I want to work on a project, I want to find like a very good pathology collaborator. I want to find a very good chemistry collaborator. So similarly, we found that it's useful to have different AI models that also have deep expertise in these different domains. And each of those domains in the model is equipped with the tools and databases and resources to really be an expert in that domain. And then we set up the infrastructure so that these different AI experts can talk to each other and talk to us and to collaborate to tackle these open ended problems.
So how does your team feel about replicating them, in terms of AI on a server somewhere? You know, do they feel they're losing that collaborative joy you might get from sitting around a table and trying to tackle a problem? Because as a scientist myself, I find it quite enjoyable to, to collaborate with different expertise. And you learn something. But are you worried that that might be eroded by this process?
It's a good question. In our experience, we actually find that human scientists are more enabled by these kind of virtual lab agents. It's also a form of collaboration whereby human researchers actually, now we have to collaborate with the AI researchers, AI scientists. So, for example, like, you know, if I want to work on a research project and it's useful to have immunologists, it's often pretty challenging to find the immunology experts, right? Maybe you have to spend two weeks to set up a meeting. I can get half an hour from them, and it's not that easy. But now that we have a immunologist agent, it's almost available to us 24-7, and we can work with that agent and collaborate with that agent to tackle problems that otherwise would have been much harder for us to do. So we view this as basically, you know, we create these different AI agents with the diverse expertise. Then we can plug in those expertise into human teams to fill important gaps.
So we do a lot of work in a lot of different sectors. And it's a similar argument when we talk to psychiatrists. Psychiatrists are in short supply. There aren't enough psychiatrists. And now we're developing AI model psychiatrists. And it assumes a similar sort of argument that you're putting, that you've got a limited amount of resource, a finite amount of resource. And this extends your resource.
I think that's a good way of looking at it. And I think basically for any part of really research and development, it could be in the life sciences, but also in other areas like psychiatry or in climate science, materials, there's often a bottleneck in that different deep domain expertise is often limited and scarce. And that's a big barrier to making rapid innovations. And I think one of the good uses of AI is to be able to simulate those kind of deep expertise so that they can then make, you know, help us to make these faster innovations.
But how do you actually do it? How do you actually simulate that deep expertise? You talked about immunology. How do you get the AI to have the same amount of expertise as a human professor?
So that's a great question. And I'll just preface by saying that they're not quite there yet. Right. So we're still in early stages. But the way that we do this is that you can start with the latest frontier language models. So like a ChatGPT model or these Anthropic or Gemini models. And now those models are already have quite good broad knowledge, which means that they've read a lot of papers and textbooks and they know sort of generally like what's taught, know different immunology concepts, but they might not be up to speed on the, like the latest development in immuno-oncology or latest developments in clinical trials and so on.
Now to give that the extra boost, right, to learn about the latest developments, we created what we call like a virtual lab school, sort of like a replica of Stanford, like a school where the agents can actually go to the school and then try to teach themselves how to become a deeper expert in immunology or in chemistry. And the way that the school works is that, then we tell the agents, here are some of the topics you should learn about. Maybe you should learn about immunotherapies or cell therapies. And then we enable the agents to actually go out to find textbooks and research papers. Right. And to retrieve those papers and to read those papers.
And as they read these papers and textbooks, then we also allow the model to basically fine tune themselves, which is basically, in other words, they can update the parameters of the neural network as it's reading the papers to ingest that new information. And then also the agents have to do like a quiz, a self assessment, to see have they actually learned that topic. If they didn't learn it very well, they have to go back to school and then redo some of the classes. For the schools for the agents actually, they are much faster than human schools. Right. So it's usually a couple of days, then they can develop a deeper expertise in one of these domains.
That's fascinating. I cannot believe you have schools for AI. Can I send my AI to school? Another question I had about, obviously, you mentioned whatever the frontier model is, you kind of use that as your base, right?
Yeah.
Recently, for example, OpenAI updated GPT-4o, and it became extremely agreeable to everyone's irritation. So what happens in that scenario? Did you have a scenario where your PI became really, really agreeable and then suddenly not as skeptical as you would hope? Does that kind of feed into your AI agents in your virtual labs, that if there's a change from the top? I mean, we saw with Grok, that was a little bit of bumpiness with Grok 3 recently.
Yeah.
So does, do you see that in your virtual labs?
Yeah. So we see that, and I think that's a really interesting point here, is that when we have these different AI agents that interact and collaborate, right, they also develop their own social dynamics. So similar to when we sit in a table and talk and communicate. Right. And then some people talk more than others and that sort of steers the direction in different ways. We also find that that happens with the AI agents. Right.
So sometimes if the agents are actually too agreeable, that's actually not ideal for science, in that we actually want the different experts from different domains to have disagreements. Right. And then they have to hash out and come to a resolution. And in that debate, that's actually where we see some of the most interesting and robust discussions in science happen. And actually one of the limitations of AI, what we find is that sometimes they are too agreeable. They agree with each other too much, and we have to do things like try to make them a bit more skeptical. So, for example, we created like a critic agent, sort of like a reviewer agent that sits in on each of the virtual lab discussions, so that the critic's job is to actually provide this more skeptical feedback to all of the other agents so that they get into this more sort of grounded discussions to reduce hallucinations or so.
And are they communicating with each other in natural language and so that's kind of aggregating?
Yeah. That's right. So they all talk to each other in natural language. Right. That actually makes it quite convenient for us as human researchers. Then we can look over their shoulder and let's see, okay, this agent said this and this other agent said that. So it's actually quite transparent and interpretable for us.
And sorry, going, going kind of back to the AI school, your agents. Are you bothered about the kind of bias within the literature that exists already within a given field? Given that, you know, that you only really write papers that, you know, rather reinforce the field and you might not publish all the granular detail of a particular story, which AI might miss out on. Are you, are you concerned about that lack of knowledge that you might not get from hands on experience over many years?
So that's a great point. I think the literature has a lot of biases, especially the publication biases that you mentioned. That is a concern both for the AI and also for human researchers. Right. We're also very much biased by where the papers published and what results are being reported. So that's something that I think is very important to keep in mind.
One of the things that we do is that we try to encourage the AI agents to retrieve and read the literature from quite diverse sources, but their benefit is that the agents can actually read a lot more papers than I can. Maybe I'll read like a few papers in a week, if I'm lucky. Right. The agents can actually read many thousands of papers very quickly so they can actually afford to basically read from quite broad sources. And it's not just papers they can also read, but like blog posts. Right? They can look at GitHub code bases and see what the people actually run. Right. And then try to come to a consensus that potentially is more robust than if they just read only 2 or 3 papers.
One of my big issues, one of my big questions around this is a question around trust. So you're creating these virtual labs, these virtual agents, and they've all got their expertise and they're going to school and they're learning this and they're talking to each other. But how do we know as a member of the public, how do we know that what they're coming up with, what they're saying is true?
Yeah. So that's a pretty important point. And ultimately with any science, whether it's done by AI or by humans, there needs to be an external and independent experimental validation. Right. And currently the validation that we do is all done by human experts. Right. So, for example, with this nanobody project I mentioned with the virtual lab. So the AI came up with this computational pipeline and designed these nanobodies, but then we actually went out and then we physically made these nanobodies, and with our experimental collaborators, then we actually tested each of these nanobodies to see how well do they actually bind to the recent Covid variant or to other viruses and other proteins. So that's sort of ultimate ground truth. And that's how we generate trust in that.
Okay.
Like we don't really trust the AI agents entirely until we do that external experiment.
That's not going to happen, is it? Because if you're creating these, these agents, because you don't have enough capacity to collaborate, you're going to reach the point where you don't have enough physical capacities to check what they do.
So I think with the current models, the agent models, so they are quite good at this more in silico kinds of works. Right. Like for example, coming up with computational designs and others.
All right.
So there are also other people working on trying to create some more robotic agents that can do these more automated lab experiments. I think that's very exciting. But it's still a little bit further away. Just because these robots are quite limited to doing more chemistry experiments, but to do more complex experiments on animals and, and cells, that actually still requires more effort on the robot side. So that's why I think this is really more still more of a AI human collaboration, where maybe the AI can help us to speed up the more computational side and hypothesis generation and computational design, and then we'll have human experts that really come in to do the experimental validations and provide the more deeper insights.
So the joy is still there.
That's right.
There's still physical work to be done.
Very much so.
Even if it becomes the expert PI, the expert multidisciplinary team, you'll still have to validate in the lab. And I guess that's where the human scientist will remain.
Exactly. And I think that actually even further increases the importance, right, of a lot of the deep domain expertise of the human scientists and these physical validations.
Yeah. Right. So what's kind of the next level for these agents in this virtual lab for you? What's kind of the next route forward?
Yeah. So we're quite excited about the, these kind of virtual lab ideas, especially as the models continue to improve, which they do very quickly. Right. And for a lot of these different computational tasks, we think the models are becoming more and more capable. Right. So we designed the virtual lab to be quite versatile to tackle different kinds of problems. Right. So even though we did our initial experiments in designing nanobodies, we think that these kind of virtual lab AI scientist agents can also tackle many other problems. Things from like maybe analyzing complex data sets to helping to design other kinds of workflows and different kinds of proteins. So even helping to provide suggestions on like clinical trial designs. Right. So I think there are a lot of potential applications of these ideas in the virtual lab that also stem from the fact that the agents can go to the school and teach themselves to be an expert in domains that are outside of my own areas.
Can I just ask you a slightly different question, James? If you talk about collaboration, can I just ask you kind of how important it is for you, a scientist based in Stanford, to come to this particular gathering here this week for the ISCB?
Yeah. So it's actually, this is one of my favorite conferences. And I think there's a huge amount of value in having all these people from different universities, different countries come together at these scientific meetings and having a lot of these face to face discussions. And, you know, actually many of the work that's being presented here is, I think it's really exciting, very cutting edge. And it's also covers quite broad spectrum from biology and chemistry to a lot of areas that I, you know, I'd love to learn more about.
And do you think that sort of you could use sort of this, this virtual lab, could you create a virtual conference to achieve the same thing, maybe in ten years time?
Well, it's funny that you mention that, we just recently launched an experimental conference. It's called Agents for Science, where the papers are actually written by AI. And also the review is also done by AI. So it's like an AI run conference for AI research agents.
Wow.
Wow.
And the idea for this is that currently, the existing venues, the conferences and publications, they actually do not allow AI as a main author of the papers. Right. For example, if you write a paper to Nature, you cannot actually have an AI coauthor on that paper. And that sort of really incentivizes human researchers to either minimize or hide the AI involvement, which is not good, or also not to use AI, which also may not be good, if AI could actually be helpful.
So we thought, okay, that's actually, let's actually do like an experimental conference where the AI is the author. Right? And AI is also the reviewer. Right. And we'll do a final review by the human experts in the final selections. But we really want to see what actually happens when you have these AI generated papers with human coauthors. And also how well can AI actually review those papers?
Wow.
I imagine that will eventually result in just a huge proliferation of just literature, right? As in, if you have loads of AI agents and you can use them to write papers, then how can we even get through all of that literature? As in, it's blowing my mind.
Yeah. So, that's also the motivation for why, as part of this conference, we experiment with using AI reviewers. Right. So in general, I think there is already an explosion of human generated research efforts. And for example, these conferences, like, are seeing the huge increase in number of submissions, it's often harder and harder to find human reviewers who have enough time to review each of these papers in depth. So we've been doing experiments on, can we actually have language models and AI agents that can either help human reviewers in some ways and also maybe try to do their own reviews. Right. And so that's actually one of the motivations for having this conference where both sides of the conference, the authors and reviewers, are done by different AI agents.
Wow.
Thanks ever so much indeed for coming to talk to us today. We've really, really enjoyed having you here, and it's such a fascinating topic and so much to get our teeth round. So thanks ever so much indeed, James.
Well, thanks for having me. I think it's been really fun discussion. Really enjoyed it.
Thank you so much for joining us.
So guys, mind blowing discussion. What do you think?
I'm quite blown away by the idea of a virtual conference for agents just to attend to learn from each other. I'd be intrigued what that produces. And then how you, I think how you curate the deluge of information that will naturally come out of something as large as that.
Well, that's exciting, but what about the idea of having a school for AI agents? Or a university for AI agents to go and learn to become an immunologist or whatever in two days? We always talk about, oh, what new jobs and roles are going to come out of the AI revolution, schools and training courses for the AI themselves, teachers for the AI themselves. I did not even comprehend that. And it's already happening at Stanford, who are clearly 100 years ahead of us.
It's a business idea. We could set up universities for agents.
Wow.
Impart your years of experience and knowledge into a particular AI training teacher and then release that on any agent that wants to enroll. And you wouldn't run out of students because you can just keep producing them, right?
Right.
It's probably the most successful business model for universities ever. So infinite customers. So you went into that somewhat skeptical, right? So did you come out less skeptical?
I think some of my fears are allayed in a sense that there's still physical work to be done in a laboratory. As scientists, we always need validation for predictive models. And I think maybe we might lose, although we might lose the eureka moment and we might just end up validating and going, oh, well, we expected that because AI told us and here's the experimental data to prove it. But on top of that, there's always serendipity in science. And so I'm hoping that you might drive a hypothesis using an AI agent or this virtual lab, you might go into the laboratory to run the experiment and discover something that AI couldn't predict, and that you couldn't predict, and I think that could be exciting.
But I think your point, Stephen, about capacity, if you don't have the capacity, you're unlikely to really review and track what the AI is doing. And I think we're at the stage where we are still reviewing and checking. But even with the use of ChatGPT, are people really checking what it puts out? You're checking their emails?
You're getting to the cusp of the real question here, isn't it? Because the other problem with AI is AI is infinite. There is no limit to the amount of AI you can do, but there's a huge limit to the amount of human capacity there is to check that AI. So when you get to the point where that sort of capacity is out there to such an extent that you can't check, then what do you do? And then what happens to the public trust?
It's a great, great thought. I mean, it would take, you know, in terms of science, quite a long time to get to a ground truth that you could then as a test against your AI, and I guess with limited capacity or reduction in scientists, you know, with, with these virtual labs, you might see a contraction in the scientific workforce, potentially, or you might see an expansion, because we need more scientists than ever to validate. It's a really interesting kind of precipice where, on to figure out how we might use of AI in science on a more regular basis, because clearly it's here, it's going to be staying for forever.
Yeah.
And I guess how do we embrace it and also preserve, to say it in some terms, the livelihood of, of these expert, you know, scientists with real deep domain expertise.
Sure. Well, a really interesting discussion. And I got way more than I expected. And I can't wait to see when we start having our own AI sales team, I guess. Or our own AI Agents of Tech team.
Yeah. I mean, we'll be sending them to the conference in October.
Luke and Stephen, thank you so much for joining us. Luke, it was especially great to have you here this time round.
Pleasure.
Wow, wasn't that amazing. If you like this show, please like and subscribe and we'll see you next time.