Understanding AI World Models with AMI’s Alex LeBrun
Alex LeBrun, co-founder and CEO of Advanced Machine Intelligence (AMI Labs), argues that large language models fall short in the physical world and that AI will need world models trained directly on real-world data such as video. His company has raised more than $1 billion to prove the approach. He discusses how world models are trained, how they could make agentic AI more reliable, and the challenge of competing for compute and capital. The hosts then debate whether the AI economy is too focused on scaling LLMs.
Transcript
The AI industry is being built around one simple idea: scale the large language models, add more data, more compute, more capital, and capability and precision will keep improving. And so far that bet has been very good for business. The biggest AI companies are growing fast. Investors are all in, and the infrastructure build out is massive. But Alex LeBrun thinks that this path may be a little too narrow.
He argues that if AI is going to understand the world, model cause and effect and reason across time, it may need to do something different: world models. His new company, AMI, has raised more than 1 billion to try to prove it. He isn't making a technical argument. It's a challenge to the whole logic of the current AI boom. So today, we're asking Alex LeBrun whether the industry is over invested in scaling just one dominant paradigm, and whether his proposition of world models are a real alternative or just a better theory in a market that may have already chosen its winner.
We are talking current AI world models and the future direction of the industry with Alex LeBrun, co-founder and CEO of AMI, the new AI company betting that the next big breakthrough will come from something called world models, not just LLMs. All right, Stephen, so the current AI boom is really built on the huge scaling of LLMs, do you think that they're getting it wrong?
Well, I think we're going to learn a lot today, right? First of all, we're going to learn what world models are. I mean, you know, I know the theory behind it, but I don't actually know what they look like and what they're going to do, when they're actually going to come about? But I also want to know where they fit in. Just to answer your point, you know, not only are we LLMs, but we've got this agentic economy, agentic revolution, some people would say, and by the time we see world models a year or two years down, what will agents already be doing? So that's what I want to know, what are they, when are we going to see them? And Laila, how do they fit in?
I think there's this story that it's like LLMs versus world models, because that sounds good, in the same way, it sounds really nice and dramatized that Claude Mythos is going to cause the entire world to meltdown-
Is you not?
I don't know. You'd have to ask Anthropic about that. I'm sure they'd say yes, but you know, all children are going to cease to learn how to read and write. But I don't think it has to be like that, as in both things can exist simultaneously and they can probably also work together. And world models might emerge from LLMs one day and vice versa, world models might, I don't know, be able to deal with language even though they were initially built to deal with the real world. So I just think this false dichotomy of one or the other, and like pitting Yann LeCun against Sam Altman and Elon, like everyone just likes to do that because it makes for a really good story, right?
Right, a juicy battle over AI.
Yeah, yeah, but I just don't think that's the way it's going to work. Like there's enough work in AI for everyone to do and to try different-
There's enough billions of dollars for everyone.
Yeah, I agree.
Well, they've captured one of those billions. Well, we want to know what you think. Is scaling large language models still the clearest path to more capable AI, or is the industry becoming a little too locked into one model of progress? Are world models the next real step or just an ambitious idea without proof at scale? Tell us in the comments below and be sure to like and subscribe. Here now is our conversation with Alex LeBrun.
Alex, thank you so much for your time today. To kind of start with a big but basic question, I would just like to know where you think LLMs are falling short?
LLMs are using language, they are very, very good at manipulating word token language and they fall short in the real world. If you want to understand or manipulate the real world, then LLMs are not good.
Are you saying that they fall short in the real world when it comes to implementation in the healthcare industry or across the board?
Across the board. Any industry where, like robotics, healthcare, of course, industrial applications, anytime you are not on a language-first task, LLMs are actually very poor. So they are already good in everything that is discrete, recognized, so like mathematics, coding, information retrieval, all these tasks are language first. And they are, it's super useful. We use it every day now. We cannot work without these. But it's only one class of problems. It's not everything in the world.
You are working on world models. Can you give us an idea on what a world model is?
LLM is trained on text. So as you know, you know, it's trained basically on the full internet. So it's about, you know, 20, 30 trillion tokens. World model is trained directly from the real world. So we want to skip the language step. And as animals, as humans, you know, babies, you're not training from books initially. You are just seeing, listening, touching through senses. And this is exactly the kind of learning we are trying to do with world models. So skip the shortcut of language, of text, and learn from direct experience in the real world. So it's a very different way to train the AI model.
But Alex, do you think that it's a bit of a false dichotomy to be pitting LLMs versus world models? And I think, I guess the more interesting question would be, if we hypothetically scaled LLMs further, do you think from that exposure to language at that scale can lead to implicit world models emerging, essentially? Like, do you think that's possible, and does it make sense, given how much money we're already putting into LLMs, that we just go down that route rather than starting from scratch?
So I don't think so, otherwise I wouldn't be doing what I'm doing, and no, we think there are actually like diminishing returns with LLMs, you can really see that, you know, every time you add like 10 more billion dollars, you get slightly better and you also get better through heuristics, through thinking, reasoning, you know, but it's not the core, we see really diminishing returns on the core progress of LLMs. And some people believe that you will actually lead to AGIs, to artificial general intelligence, with language models. And we don't agree with that.
And we think that eventually, to be really useful in the world, you need some real-world experience. If I take the example of healthcare, you know, doctors spend a lot of time to learn from books obviously, but they also do residency for years and years and years after practice medicine. You would never imagine a physician trained only from books. And so this is exactly why we think world models, you know, are not better or in this place, LLMs, but it's just complementary and you need both, a kind of learning to be very smart, so this is complementary.
When are we actually going to see something from world models and when we see something, what will we see?
Current AI is really bad in understanding the physical world. You think it knows about because it's just repeating words that read about the physical world. But it's a proxy, it's not a direct experience. And for instance, if you ask LLM, I want to take my car to the washing station at 100 meters away, should I drive or should I walk? And most LLMs, up to recently because they fixed this, and said, "Oh, you should walk because it just five minutes away." Of course, it's not true, you know, you need to take your car to wash it. So it's a good example where they fall short as soon as we are talking about real-world usability.
First world models are good at understanding the context, for instance, of a video input, knowing what's happening, what is there, and if you want to get something done in the real world, what is the plan, you know, what the robot, for instance, should do, so that will probably be the first implementation of world models.
Pause right there. Coming up after the break, we're going to ask Alex whether the AI economy is now already too invested in scaling to admit that there maybe could be alternative paths to how we get to the future of AI. Stay with us.
Welcome back, we're continuing our conversation with Alex LeBrun, and Alex, when it comes to money, you guys have raised over a billion dollars in funding, but there is a lot more money flowing into scaling LLMs right now, do you think that the AI economy is just too shortsighted to understand that there might be an alternative when it comes to the future of AI?
It's a very good question. I think we haven't done everything we will do with LLMs yet. And so the investment that is continued to be done in generative AI and LLMs is justified and you can see every three months, you know, all the value we get from these new models. So I think it is justified that huge investments are done in, like, the old or the current existing branch of generative AI and LLMs, but it doesn't mean you need to also explore new branches. And while one of the alternative branches and may be others, the fact that, you know, investors are ready to invest with more than 1 billion in this, I think, shows that some people at least agree that LLMs and GenAI won't solve everything.
1 billion is nothing to sneeze at. That's a lot of money. But to follow up on that question, do you get pushback from the people behind the LLMs or the people really promoting that version of AI? Do people think you're kind of a fringe participant?
I don't think so. You know, at least researchers, more and more researchers agree that they are willing to work on something different. And it's funny how in just one year we saw lots of scientific, the best scientists, you know, deciding that they should start to work on something else. So of course, there is a huge investments in scaling infrastructure for GenAI, you know, most of these investments will actually be also relevant for other models like world models. And I think after some time, more and more investments would shift to new kind of models that are not large language models.
Alex, following on from that, how are you actually going to do this? So in terms of the data sources you're going to use, because I know it's based on more sensory data about the world, and LLMs came about because they scraped the entire internet, which is trillions and trillions of data points, but what are you going to use and how are you going to do it at scale and how are you going to do it within a couple of years? I know you said that you weren't going to immediately show revenue, but how are you going to do this?
So mostly through a video to start. And so we are gathering, you know, lots of video, video is very dense data. You know, a 4-year-old child has been awake for about 16,000 hours and in these 16,000 hours, they receive about 10 to the 15 bytes of information through the optical nerves, which is more than the full internet and all the books and all the data LLMs are trained from. So it shows you that video data is extremely, extremely dense and full of information. And this is what we are using to train world models today.
How does that differ from other video models? Why wouldn't Google scrape all of YouTube to do the same and would they not have an advantage over you, if you're using video?
Well, if you start to say, yeah, if Google does the same thing, you know, what will happen? I think it's true for everyone, every startup, every project, the reason is they can do everything. But yeah, I see what you mean. So first YouTube is not enough because like 99% of YouTube videos are actually people talking in front of a camera.
Like us, yeah.
Like us.
And even if it looks like video, it's actually like language first because it's more like a, yeah, talk. And so we are working with partners to source more ego-centric, more interesting videos and it's a lot of work. This is what we are doing now.
Arguably right now, we live in an agentic economy, so not just a large language model, but you know, through agents and through things like OpenClaw, some of those agents can actually do things. I mean, Laila here can get video editing agent to actually make Autria and I look intelligent, believe it or not. So in the year or two or three where you are developing the world models, where would you fit in with those more advanced sort of agentic models?
Currently, these agentic models are very brittle. If you try to do like a long-term planning, it usually breaks, and also you have some accuracy that is not a hundred percent. And so for instance, in healthcare, you know, 99% accuracy is not enough. You cannot kill one patient out of 100 and you need much more explainability and deterministic behavior, which you will never get through the existing breed of agentic AI.
World models, you know, because it builds an internal representation of the world, which is more deterministic, will allow for this like longer term planning with more accuracy and would actually unlock the deployment of agentic automation in many industries where today, you know, beyond the demo and beyond the cool effect and some industries, it's okay if there is a mistake, but actually most of the real-world applications, you cannot afford that, so this is where world models will be very interesting.
Alex, you know, other very large AI companies, OpenAI, Anthropic, they've toyed with this idea of world models. They've, you know, explored adjacent ideas and then it seems like they've kind of shied away from that, what is it that your company, AMI Labs, is doing differently when it comes to world models?
The world model concept is not new. It's been around for almost 10 years and many researchers, you know, participated. And recently, about a year ago, it started to actually work. A team around Yann LeCun, our chairman, developed a concept called JEPA, Joint Embedding Predictive Architecture, as that is a way to train these world models through self supervised learning. And many of the problems that were there before suddenly started to be solved, you know, not everything is solved, but we are at a stage that may be comparable to where transformers were in 2018 after the, you know, the paper was released and then OpenAI started to scale and solve the real problems around that. I think we are at a similar stage.
So it's also similar to what happened in 2012, where deep learning, you know, the concept of deep learning had been around for almost 20 years, but it wasn't working because it was small scale, too small, not enough data, not enough compute. And suddenly, Geoff Hinton and people around him just scaled this and it worked. And suddenly in about one year, all the old problems in computer science, not all, but many of the problems were solved very elegantly with deep learning. So you have this kind of key moments where old ideas start to work, and we think we are exactly at this stage with world models, and people are joining our company from, you know, different labs across the industry because they believe in this vision.
What are we going to see first from world models?
It'll probably be some kind of video understanding model, you know, which is something that is not solved today. The way we process video today is really like we have a hammer, generative AI, and we see everything like a nail. And we are taking video feed, tokenizing it frame by frame. And so it's really not, it's very bad, you know, it works for very simple application, but it's not the right way to do it. And so this is an area where I think we'll see the first deployments of real world models.
Alex, you said in your previous answer that you're around 2018, where LLMs were just before they scaled. So what does scaling look like for world models now? Where do you go next? What does that look like?
So scaling means gathering more data. As I said, you know, it's harder than LLMs because you cannot just take the internet text, you need more like enough work.
How much data does that look like?
We don't know yet. You know, it's research. I'd love to know. I'd love to know. And you need to scale the data, then you need more and more compute to digest this data, which is also very expensive, very hard to provision today. And you need the best team in the world of scientists to do that. So these are the three ingredients we are mixing in the company.
I imagine you're obviously competing for that compute with everyone else out there. And in order to compete, you need the private capital. But private capital has so far chosen the things that give the biggest return. Like Anthropic's run rate is 30 billion now. So how are you going to ensure you've still got that flow of capital to give you the compute you need?
Yeah, so that's my job first as CEO, and to feed my researchers with more and more GPUs, and they are very hungry every day. We need to be, it's a very hard balance to find, because we need to be very ambitious, do work on research, long-term research, ambitious, but at the same time, we need to be able to show off, not in 10 years, you know, what we have as a results, like intermediary models, to prove that we are doing progress, to prove our point, and then be able to raise more capital. So striking this right balance between showing enough to get capital but not doing only short-term optimizations, but working on something completely new and long term, is a very interesting.
I'm glad you mentioned the 10 year mark because I wanted to ask you, kind of looking into the future of AI, do you see world models overtaking LLMs or is there a place where we're going to see them working in tandem?
They'll definitely work in tandem, you know, and this is the way your brain works. You have the two areas, Wernicke and Broca areas, managing language, understanding and generation. And you have the prefrontal cortex area doing the reasoning, and the world modeling is there. And so we are, you know, the LLMs of this and we are working on the three. And you can be, like, some humans have this Wernicke or Broca area damaged. They cannot speak, but they're very smart and they can behave in the real world. So that's, I think, very clear that they are complementary.
So Alex, if we live in a world where LLMs take care of language and world models take care of the rest, then what's left for humans?
I mean, these two-
Impossible.
Go enjoy life.
I think that's not going to happen. That's not going to happen.
These two things remain a tool for humans, you know, like our brain is a tool. So it doesn't mean that we replace human. We just hope to make AI more useful today. And you know, many problems you think are solved because there is so much hype and so much marketing in AI.
And actually, so for instance, I was visiting the Port of Singapore last week and, you know, under the containers, there are some locks to attach containers to one another. And even if containers have been standardized for a many, many years, the locks are not standardized. There is always something that is broken, you know, in the installation. And so it's very hard to put and to remove these locks. It's a very hard work. It's a little bit dangerous and no robot is able to do that to this day because it's a very complex, the gesture, you need to understand what kind of locks has been used. So problems that seems so simple are not solved yet. So we try to really overestimate what AI can do, I think, in the real world.
Going to make your life so much easier, so much more time for you to relax. Okay, Alex, final few minutes here. We have our rapid fire question round. These are the questions that we ask all of our guests here on "Agents of Tech." To start things off, I would like to know where you think people will draw the line when it comes to bringing AI into their own lives?
I think people won't let AI do too many decisions for them. And we see that for, you know, some music players like Spotify, more and more people don't want Spotify to decide what they will listen to. And it'll be true for many areas, I think.
Yeah, I agree-
True.
Especially like the travel, everyone wants AI to pick your travel.
But they get it wrong.
I want to do it myself. They get it wrong.
Okay, Alex, mine is, what is something that's widely accepted in your field today that you strongly disagree with?
We spoke about it. I think too many people think that the current breed of AI, LLMs and GenAI, will lead to artificial general intelligence. And I think it's not true.
So my question is a really good one for you because, you know, you are looking further out, so what can you see looking further out, one or two years down the line, that we are not talking about today that you think is going to be really important?
I think we are too pessimistic. We say, okay, AI is doing everything. People IQ will go down because children don't learn and won't have to learn and so on. I think what we are not seeing is the fact that the new generation is born with AI, knows very well how to use it, knows the limits, use it really well, and will actually be more augmented than replaced, you know, and I feel that we are too pessimistic, especially with how children will actually leverage AI.
Well, we look forward to having you back here on "Agents of Tech" five years down the road to see where the world models stand. Alex, thank you so much for your time today.
Yes, thank you, Alex. Thank you.
Thank you so much. Great to have you on.
Fascinating conversation with Alex there. When we started off, we had one big question. Do we think that the AI economy is too focused on just one paradigm of scaling LLMs? Do you think that Alex made the case for not necessarily a pivot to world models but an inclusion of world models?
He definitely did convince me of a little, like, at first, I was like, a billion dollars into world models where, and then I thought I would maybe give a couple of million if I had it, but maybe not quite a bit.
He'll write that too.
But yeah, I understood how he probably convinced a lot of those investors, but I just still think that the market doesn't decide on what is ultimately right. It's what makes it the most money. And there's a difference between that, that it's not what is necessarily the right answer.
And when we look maybe 5 to 10 years down the road and we see, oh, world models are where the money is, that's when you're going to start seeing all that capital flow in.
If the money is there quickly enough, isn't it?
Yeah, what do you think?
I think one of the things that people just don't pick up on enough in the kind of agentic revolution that we're seeing right now is return on investment, right? People are always talking about investment of billions, but people aren't talking about return of investment. And what you are seeing now in LLMs and agentic economy is that return on investment. You know, the big companies, the hyperscalers putting the money in are getting a return. And we're going to see the same with world models. And I liked Alex a lot, I thought he was great, I really bought into everything that he was saying. But when will we know whether he is going to achieve a return, and it's only at that point then it'll become commercially viable.
Yeah, and they've said, don't expect a return in a couple of years, as in that's the same length of time that Anthropic has existed, like, what, three years, and-
And I think he made a good argument of like, look at how much work we have to do to get these world models up and operating before we can even attempt to scale them.
Right, but the problem is, and he alluded to this really, it's about buying time, right? I mean, you have to buy the compute, you have to buy the researchers, and will they be given enough time, will they have enough resources to buy that time so they can compete?
Yeah, I agree. And also he mentioned, it's the 2018 moment, but for world models, but in 2018, there weren't any competitors to, really, OpenAI. They came out with GPT-3.5 and everyone was like, oh my god, what is it?
What is this? Yeah.
And then Anthropic emerged, Perplexity emerged.
Right, and they're now investing a trillion dollars a year, right, in infrastructure, a trillion dollars. And that trillion dollars has to come from somewhere. And so, you know, world models, if they're going to want a piece of that, are really going to have to show that return.
I think he also made a really good strong case for, if you're looking down the road in the future, there's room for everybody, we're still going to have the LLMs, we're still going to have agentic AI, but then also we're going to include, you know, world models into the conversation.
Right, in theory, right, in theory.
Right, he hopes, yeah.
But there's not enough compute for everybody.
Right.
Well, that's a whole different conversation. But we do want to keep the conversation going. We want to know what you think, world models, the new frontier in the world of AI, or is scaling LLMs still the best bet forward for seeing all the potential that AI has to offer? Do you believe that the industry is backing the right AI architecture or just the one that, as Stephen has pointed out, is easiest to fund and sell? Be sure to drop your comments below and let's keep the conversation going. Make sure to subscribe as well. And we will see you next time right here on "Agents of Tech."