Physical Intelligence Blog

Vision-language-action models seem to be the broad category for the best approach, which basically combines a large vision-language model with robotic actions. For example, see https://www.physicalintelligence.company/blog/pi0

https://physicalintelligence.company is working on this – see a demo where their robot does ~exactly what you said, I believe based on a "generalist" model (not pretrained on the tasks): https://www.youtube.com/watch?v=J-UTyb7lOEw

just coming back to this thread, this paper is quite a good read: https://arxiv.org/html/2406.09246v3#S3 and as a follow-on, this blog post by Physical Intelligence was interesting: https://www.physicalintelligence.company/blog/pi0

The robot is only as good as the amount of time you’re willing to spend teaching it. If you put in, say, 20,000 hours and make "some" hardware upgrades, you can get it to do things like this: https://www.physicalintelligence.company/blog/pi05

It’s the fundamentals that underly Stable Diffusion, Dalle, and various other SOTA image generation models, video, and audio generation models. They’ve also started taking off in the field of robotics control [1]. These models are trained to incrementally nudge samples of pure noise onto the distributions of their training data. Because they’re trained on noised versions of the training set, the models are able to better explore, navigate, and make use of the regions near the true data distribution in the denoising process. One of the biggest issues with GANs is a thing called “mode collapse” [2]. [1] https://www.physicalintelligence.company/blog/pi0 [2] https://en.wikipedia.org/wiki/Mode_collapse

> We’ve long since proven that machines are better for back-breaking labor. This is contrary to my perception of current trends (and to a perhaps generous reading of Moravec's Paradox). Dexterous manipulation is still quite hard which is why a task like laundry folding is so impressive despite being relatively simple for many humans: https://www.physicalintelligence.company/blog/pi0 Perhaps we disagree on what "back breaking" entails but I'd much rather spend all day coding than folding laundry!

[2023] Some of the authors have gone on to found a startup called Physical Intelligence: https://www.physicalintelligence.company/blog/pi0

We are certainly getting close! In 2010, watching PR2 fold some unseen towels is similar to watching paint dry [1], but we can now enjoy robots attain lazy student-level laundry folding in real-time, as demonstrated by π₀[2]. [1] https://www.youtube.com/watch?v=gy5g33S0Gzo [2] https://www.physicalintelligence.company/blog/pi0

https://www.physicalintelligence.company/blog/pi0?blog Show me what was possible 20 years ago versus what we can do now. I think you have enough imagination to envision what might be possible 20 years from now.

Don't worry, those robots are coming! https://www.physicalintelligence.company/blog/pi0 Software moves faster than hardware

regardless of what is "really" happening under the hood, if this is model has any influence on the real world through robotics, this could lead to an actual fatality. Physical Intelligence has a robot with arms that can interact with the world. If a robot like this, or a more advanced robot like a car, ends up on the same thought pattern that this model did, it could make actions that can hurt people. - https://www.physicalintelligence.company/blog/pi0?blog

That's less a GenAI problem, more robotics. And perhaps it's here, literally doing your laundry: https://x.com/physical_int/status/1852041726279794788 https://www.physicalintelligence.company/blog/pi0 https://www.entrepreneur.com/business-news/jeff-bezos-backed-physical-…

> But factory robots haven't propelled Kuka, Fanuc, ABB, UR, Staubli and peers to anything like the levels of success nvidia is already at. A market big enough to accommodate several profitable companies with market caps in the tens of billions might not drive much growth for a company with a trillion-dollar market cap. That's because the past year of robotics advancements (e.g. https://www.physicalintelligence.company/blog/pi0, https://arxiv.org/abs/2412.13196) has been driven by advances in machine learning and multimodal foundation models. There has been very little change in the actual electronics and mechanical engineering of robotics. So it's no surprise that the traditional hardware leaders like Kuka and ABB are not seeing massive gains so far. I suspect they might get the Tesla treatment soon when the Chinese competitors like unitree start muscling into the humanoid robotics space. Robotics advancements are now AI driven and software defined. It turned out that adding a camera and tying a big foundation model to a traditional robot is all you need. Wall-E is now experiencing the ImageNet moment.