Danijar Hafner

The best world model research I know of today is Dreamer 4: https://danijar.com/project/dreamer4/. Here is an interesting interview with the author: https://www.talkrl.com/episodes/danijar-hafner-on-dreamer-v4 Training on 2,500 hours of prerecorded video of people playing Minecraft, they produce a neural net world model of Minecraft. It is basically a learned Minecraft simulator. You can actually play Minecraft in it, in real time. They then train a neural net agent to play Minecraft and achieve specific goals all the way up to obtaining diamonds. But the agent never plays the real game of Minecraft during training. It only plays in the world model. The agent is trained in its own imagination. Of course this is why it is called Dreamer. The advantage of this is that once you have a world model, no extra real data is required to train agents. The only input to the system is a relatively small dataset of prerecorded video of people playing Minecraft, and the output is an agent that can achieve specific goals in the world. Traditionally this would require many orders of magnitude more real data to achieve, and the real data would need to be focused on the specific goals you want the agent to achieve. World models are a great way to cheaply amplify a small amount of undifferentiated real data into a large amount of goal-directed synthetic data. Now, Minecraft itself is already a world model that is cheap to run, so a learned world model of Minecraft may not seem that useful.…

Dreamer4 (https://danijar.com/project/dreamer4/) is a promising direction (by a frontier lab)

You'd be interested in Dreamer 4, very recently the first AI to acquire diamonds in Minecraft purely from offline knowledge: https://danijar.com/project/dreamer4/

https://danijar.com/project/dreamer4/ "We present Dreamer 4, a scalable agent that learns to solve control tasks by imagination training inside of a fast and accurate world model. ... By training inside of its world model, Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, aligning it with applications such as robotics where online interaction is often impractical." In other words, it learns by watching, e.g. by having more data of a certain type.

It gets diamonds at 1:48 in the top left video (might need to full screen to seek) [1]. The tools are admittedly really hard to see in the videos because of the timelapse and MP4 struggles a bit on the low resolution, but they are there :) [1]: https://danijar.com/dreamerv3/

Firstly, do these models learn a good physics grounding for nonsense actions? Like keep pressing down even when you are in the ground? Or will they phase you through the ground? Secondly, why are all videos like half a second long? I thought video generation came much farther than this. My guess would be that the world models unravel at any length longer than that, which is (and has always been) the problem with models such as these. Minus the video generation part, we had pretty good world models for games already, see Dreamer line of work: https://danijar.com/project/dreamerv3/

I’m not sure if dreaming is the right word here, but the recent ML model Dreamer v3 [0] “dreams”. This is more akin to thinking ahead since it’s really just predicting outcomes using its world model. It’s possible they could try training the model on some hallucinated scenarios to avoid over fitting, but I’m no ML researcher. [0] https://danijar.com/project/dreamerv3/