The rapid evolution of artificial intelligence (AI) is ushering in a future where robots do more than just execute pre-programmed tasks; they can intelligently respond to their surroundings, understand speech, and act autonomously. This breakthrough could rival the impact of ChatGPT in the AI sphere, adding a fascinating twist: robotics is actually an older science than AI itself. But what’s the connection between the two, and how does it all work?
VLM (Vision-Language Model)
Let’s start with the basics! A VLM, or Vision-Language Model, is an AI model capable of processing both visual information (like images and videos) and natural language (text or speech). Think of it as a combination of “eyes” and “ears” in a brain: the model “sees” the world through cameras and “understands” commands like “Pick up the red apple from the table!”
VLMs typically consist of two main components. An image encoder analyzes visual input, identifying objects, colors, or positions. Meanwhile, a language model (like GPT) interprets the text and connects the two. For example, a VLM can answer questions about an image, such as “What’s in the picture?” or describe what it sees. These models learn from vast datasets where images and texts are paired, allowing them to “understand” the relationships that describe the world.
VLMs enable robots to interpret their environments.
What is VLA? (VLM + Action)
The VLA, or Vision-Language-Action model, is an enhanced version of the VLM that adds the “action” element. Here, it’s not just about vision and language; it also involves directing the robot’s physical movements. VLA models can directly generate commands for the robot’s motors, such as “Lift your arm 30 degrees!” or “Turn left!”
Learning for VLAs often relies on imitation: the model observes how tasks are performed by humans or other robots through videos or demonstrations and then mimics those actions. These models integrate visual perception (sight), language comprehension (instructions), and an action plan (movement). For instance, a VLA can navigate a kitchen, recognize objects, and manipulate them while continuously self-correcting based on feedback.
VLAs are particularly useful for complex tasks like folding clothes or cleaning a kitchen because they can think logically: “If this is in the way, then I should push it aside!” This capability makes them “smart” robot controllers.
You can delve deeper into these concepts in the glossary.
A Practical Example: Google Gemini Robotics 1.5
Google DeepMind’s latest development, Gemini Robotics 1.5, is a prime example of VLA-based robot control. It’s a multi-embodiment model, meaning it can be used across different robot bodies (like arms and humanoids). One of its significant innovations is its ability to autonomously execute multi-step tasks in an “agentic” manner.
Gemini Robotics 1.5: Enabling robots to plan, think, and use tools to solve complex tasks.
Extra: Raw Language Model Controls the Robot
On the aiszakerto.hu blog, I also wrote about the concept of “embodied AI”. Here, an advanced language model takes over higher-level control tasks. To put it simply, it transitions from the screen to “taking form” in a robot or avatar: it can control robots using coordinates, moving limbs. Google is training a specialized AI model for this, but the article illustrates how a standard large language model can also manage an “AI marionette” (now in a virtual space).
AI Marionette
In our biological brains, the final assembly of movement occurs across several layers. Simplifying things, older layers like the brainstem and cerebellum are responsible for basic movements, such as walking and balancing, which we don’t consciously think about. The motor cortex in the gray matter initiates movements, while the prefrontal cortex handles strategic decisions (what to reach for, where to go).
In summary, AI is capable of not just thinking but also acting within the physical world.
