Last week, I ventured a whopping 15 minutes from my house to see robots do some mind-boggling, jaw-dropping stuff.
I visited the Cambridge, Massachusetts, offices of a startup called Generalist AI, where I watched robot arms perform simple chores like stacking cups, putting blocks into bowls, and the like. I was astonished by how quickly they figured things out—it was reminiscent of a flesh-and-blood person.
The arms mastered a range of tasks after ingesting a short, instructional video and, most impressively, no specific training for a given task. One of the most striking examples involved a robot that was instructed to sweep a block into a bowl using a dustpan and brush. When the brush was removed from the scene, the robot improvised by using the dustpan like a brush and flicking the block into the bowl.
Courtesy of Generalist AI
In another case, a two-armed robot watched a videoclip of someone unzipping a purse before removing some banknotes. I watched—somewhat slack-jawed—as the robot unzipped a different kind of purse and carefully removed the notes. Most amazingly, when it couldn’t grab the money, it switched from using its right gripper to its left to get a better angle of attack. “Ha,” said one engineer standing nearby. “It never did that before.”
“This is exactly the kind of thing people were really excited about with GPT-3,” Generalist cofounder and CEO Pete Florence told me, in reference to OpenAI’s breakthrough large language model, released in 2020. “You could take that model and just prompt it to do a new task and it would have a real shot at doing it.”
Generalist appears to be focused on teaching its robots about the physics of the world, which seems inspired by the intuitive sense of physics humans exhibit from an early age. That may well contribute to the model’s ability to transfer what it has learned in one scenario to another. In fact, some of the company’s demos made me think of how children improvise and experiment when shown a task. The researchers have often been surprised by what the robot decides to do—one chose to sweep up items with a banana when it was placed in front of it, for example. This might seem trivial, but physical intelligence is something still largely lacking in machines, and the way babies learn so efficiently about their world may offer important insights for AI researchers.
Courtesy of Generalist AI
I met Florence and Andrew Barry, cofounder and CTO, in a conference room overlooking teams of people doing robot training with special grippers on their hands. The company’s other cofounder and chief scientist is Andy Zeng. The trio have impressive backgrounds: They previously worked at Google DeepMind and Boston Dynamics on some of the most advanced hardware and robotic models around.
Traditionally, training an AI-powered robot to do different tasks has meant feeding thousands of examples into the model. This is a notoriously imperfect kind of learning, though, and a robot will struggle with the task if you change something as simple as the lighting.
Generalist and some other robotics startups are investing heavily in a general robotic model trained by humans. The company builds special gloves resembling robot pincers that have cameras attached to them, which people then use to perform different chores. I saw a crate piled high with several hundred of these grippers destined for workers in Mexico and elsewhere.
Florence and team are cagey about exactly what recipe they’re using to train the robots, but they say the company has already gathered a huge amount of high-quality training data. In contrast to some other companies chasing smarter robots, they have also built their AI models entirely from scratch rather than relying on an open-source language model.
Danfei Xu, a roboticist at Georgia Tech who is familiar with Generalist’s work, says that the startup stands out among companies chasing more general robot models. “They have pushed this to the extreme, and they’ve done a really good job executing,” Xu says. Besides gathering a huge amount of high-quality data, he says, “they are excellent roboticists, and they have done really good science.”
Xu also says that the stuff Generalist has demo’d so far suggests that they have an eye on deploying robots in real commercial settings. “They are the closest to something that’s deployable,” he says.
“Generalist’s data approach is collecting physical interaction data at large scale without tying it too closely to one particular robot,” says Karen Liu, a roboticist at Stanford University who also knows the company. “Their strongest results suggest that this bet may be working.”
That said, Generalist says the learning skills of its models are not yet all that reliable. A robot is only able to complete a task it has been shown about 59 percent of the time, on average; ideally, its success rate would be somewhere upwards of 99 percent. It also seems unclear how well these skills will generalize to every imaginable task or setting.
Even so, the potential for robots to quickly learn skills in, say, manufacturing seems huge. One of Generalist’s engineers seemed to discover this late one recent evening. A video that captured the episode shows the engineer stacking small cups on the table in front of a two-armed robot, just to see what the machine might do. The robot suddenly joined in, grabbing and stacking other cups with its two grippers. As the robot finished stacking the cups into one neat pile, the engineer began yelling to no one in particular, delighted by the maneuver.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
