Boston Dynamics’ robot dog Spot can now ‘play fetch’ – thanks to MIT breakthrough

Dog-like robots could one day learn to play fetch, thanks to a mix of artificial intelligence (AI) and computer vision helping them focus on objects.

In a new study published Oct. 10 in the journal IEEE Robotics and Automation Lettersresearchers have developed a method called “Clio” that allows robots to quickly map a scene using body cameras and identify the parts that are most relevant to their assigned task through voice instructions.

Clio leverages the “information block” theory, whereby information is compressed in such a way that a neural network – a collection of machine learning algorithms layered to mimic the way the human brain processes information – only selects and stores relevant segments. Any robot equipped with the system will process instructions like “get the first aid kit” and then interpret only the parts of its immediate environment that are relevant to its tasks – ignoring everything else.

“For example, let’s say there are a bunch of cards in the scene and my task is just to get the green card. In this case, we push all this information about the scene through this bottleneck and arrive at a group of segments that represent green card,” co-author of the study Dominic Maggioa graduate student at MIT, said in a statement. “All other segments that are not relevant are grouped into a cluster that we can simply remove. And we’re left with an object at the right granularity that’s needed to support my task.”

To demonstrate Clio in action, the researchers used a four-legged Boston Dynamics Spot robot running Clio to explore an office building and perform a set of tasks. Working in real-time, Clio generated a virtual map showing only the objects relevant to its tasks, which then allowed the Spot robot to accomplish its goals.

To see, to understand, to do

The researchers achieved this level of granularity with Clio by combining large language models (LLMs)—multiple virtual neural networks that underlie artificial intelligence tools, systems and services — that have been trained to identify all types of objects, with computer vision.

Get the world’s most fascinating discoveries straight to your inbox.

Neural networks have made significant progress in accurately identifying objects in local or virtual environments, but these are often carefully curated scenarios with a limited number of objects that a robot or AI system has been pre-trained to recognize. The revolution that Clio offers is the ability to be granular with what it sees in real time, relevant to the specific tasks assigned to it.

A key part of this was incorporating a mapping tool into Clio that allows it to split a scene into several small segments. A neural network then selects segments that are semantically similar, meaning they serve the same purpose or form similar objects.

Effectively, the idea is to have AI-powered robots that can make intuitive and discriminating task-focused decisions in real-time, rather than trying to process an entire scene or environment first.

In the future, the researchers plan to adapt Clio to handle higher-level tasks.

“We still give Clio tasks that are somewhat specific, like ‘find deck of cards,'” Maggio said. “For search and rescue, you need to give it more high-level tasks, like ‘find survivors’ or ‘reboot the power.’ So we want to get to a more human-level understanding of how to do more complex tasks.”

If nothing else, Clio could be the key to having robot dogs that can actually play, no matter what park they’re running around in.