Imagine purchasing a robot to perform household tasks. This robot was built and trained in a factory on a certain set of tasks and has never seen the items in your private home. If you ask it to select up a mug out of your kitchen table, it may not recognize your mug (perhaps because this mug is painted with an unusual image, say, of MIT’s mascot, Tim the Beaver). So, the robot fails.
“Straight away, the way in which we train these robots, once they fail, we don’t really know why. So you’d just throw up your hands and say, ‘OK, I assume we’ve got to begin over.’ A critical component that’s missing from this technique is enabling the robot to exhibit why it’s failing so the user may give it feedback,” says Andi Peng, an electrical engineering and computer science (EECS) graduate student at MIT.
Peng and her collaborators at MIT, Latest York University, and the University of California at Berkeley created a framework that permits humans to quickly teach a robot what they need it to do, with a minimal amount of effort.
When a robot fails, the system uses an algorithm to generate counterfactual explanations that describe what needed to vary for the robot to succeed. As an example, possibly the robot would have been in a position to pick up the mug if the mug were a certain color. It shows these counterfactuals to the human and asks for feedback on why the robot failed. Then the system utilizes this feedback and the counterfactual explanations to generate recent data it uses to fine-tune the robot.
Nice-tuning involves tweaking a machine-learning model that has already been trained to perform one task, so it could actually perform a second, similar task.
The researchers tested this system in simulations and located that it could teach a robot more efficiently than other methods. The robots trained with this framework performed higher, while the training process consumed less of a human’s time.
This framework could help robots learn faster in recent environments without requiring a user to have technical knowledge. In the long term, this may very well be a step toward enabling general-purpose robots to efficiently perform every day tasks for the elderly or individuals with disabilities in a wide range of settings.
Peng, the lead creator, is joined by co-authors Aviv Netanyahu, an EECS graduate student; Mark Ho, an assistant professor on the Stevens Institute of Technology; Tianmin Shu, an MIT postdoc; Andreea Bobu, a graduate student at UC Berkeley; and senior authors Julie Shah, an MIT professor of aeronautics and astronautics and the director of the Interactive Robotics Group within the Computer Science and Artificial Intelligence Laboratory (CSAIL), and Pulkit Agrawal, a professor in CSAIL. The research will probably be presented on the International Conference on Machine Learning.
On-the-job training
Robots often fail as a result of distribution shift — the robot is presented with objects and spaces it didn’t see during training, and it doesn’t understand what to do on this recent environment.
One option to retrain a robot for a selected task is imitation learning. The user could exhibit the proper task to show the robot what to do. If a user tries to show a robot to select up a mug, but demonstrates with a white mug, the robot could learn that each one mugs are white. It could then fail to select up a red, blue, or “Tim-the-Beaver-brown” mug.
Training a robot to acknowledge that a mug is a mug, no matter its color, could take hundreds of demonstrations.
“I don’t need to need to exhibit with 30,000 mugs. I need to exhibit with only one mug. But then I would like to show the robot so it recognizes that it could actually pick up a mug of any color,” Peng says.
To perform this, the researchers’ system determines what specific object the user cares about (a mug) and what elements aren’t vital for the duty (perhaps the colour of the mug doesn’t matter). It uses this information to generate recent, synthetic data by changing these “unimportant” visual concepts. This process is generally known as data augmentation.
The framework has three steps. First, it shows the duty that caused the robot to fail. Then it collects an illustration from the user of the specified actions and generates counterfactuals by searching over all features within the space that show what needed to vary for the robot to succeed.
The system shows these counterfactuals to the user and asks for feedback to find out which visual concepts don’t impact the specified motion. Then it uses this human feedback to generate many recent augmented demonstrations.
In this manner, the user could exhibit picking up one mug, however the system would produce demonstrations showing the specified motion with hundreds of various mugs by altering the colour. It uses these data to fine-tune the robot.
Creating counterfactual explanations and soliciting feedback from the user are critical for the technique to succeed, Peng says.
From human reasoning to robot reasoning
Because their work seeks to place the human within the training loop, the researchers tested their technique with human users. They first conducted a study by which they asked people if counterfactual explanations helped them discover elements that may very well be modified without affecting the duty.
“It was so clear right off the bat. Humans are so good at this kind of counterfactual reasoning. And this counterfactual step is what allows human reasoning to be translated into robot reasoning in a way that is smart,” she says.
Then they applied their framework to a few simulations where robots were tasked with: navigating to a goal object, picking up a key and unlocking a door, and picking up a desired object then placing it on a tabletop. In each instance, their method enabled the robot to learn faster than with other techniques, while requiring fewer demonstrations from users.
Moving forward, the researchers hope to check this framework on real robots. Additionally they need to give attention to reducing the time it takes the system to create recent data using generative machine-learning models.
“We wish robots to do what humans do, and we wish them to do it in a semantically meaningful way. Humans are inclined to operate on this abstract space, where they don’t take into consideration each property in a picture. At the top of the day, this is actually about enabling a robot to learn a very good, human-like representation at an abstract level,” Peng says.
This research is supported, partly, by a National Science Foundation Graduate Research Fellowship, Open Philanthropy, an Apple AI/ML Fellowship, Hyundai Motor Corporation, the MIT-IBM Watson AI Lab, and the National Science Foundation Institute for Artificial Intelligence and Fundamental Interactions.