We’ve been talking for a while now about how language models can utilize tools—like searching the web, generating images, and with the latest advancements, even controlling a browser or an entire computer (as I developed and demonstrated in 2023).
If they can do all that and know how to code, then it stands to reason they could also control a robot, right? They just need to understand the structure of the body (or if we’re talking avatars) and manipulate the limbs and bones by determining coordinates.
That’s the theory, and it works in practice too. Is this how we will control our robots? I believe so!
AI Puppet on Strings
What’s the practical application of this? I can envision a ton of use cases, from robotics education to simulation testing. Even if we just think about browsers, this could lead to a more engaging assistant or a face for online shopping (maybe we’d be more inclined to shop this way) beyond just the basic conversational function.
Of course, it doesn’t always go smoothly; artificial intelligence isn’t perfect. I’ve even uploaded a few blooper reels right here 🙂.
All of this works in a browser too (though I’ve disabled the AI features here).
