ComText, from the Computer Science and Artificial Intelligence Laboratory, allows robots to understand contextual commands.
Despite what you might see in movies, today’s robots are still very limited in what they can do. They can be great for many repetitive tasks, but their inability to understand the nuances of human language makes them mostly useless for more complicated requests.
For example, if you put a specific tool in a toolbox and ask a robot to “pick it up,” it would be completely lost. Picking it up means being able to see and identify objects, understand commands, recognize that the “it” in question is the tool you put down, go back in time to remember the moment when you put down the tool, and distinguish the tool you put down from other ones of similar shapes and sizes.
Recently researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have gotten closer to making this type of request easier: In a new paper, they present an Alexa-like system that allows robots to understand a wide range of commands that require contextual knowledge about objects and their environments. They’ve dubbed the system “ComText,” for “commands in context.”
The toolbox situation above was among the types of tasks that ComText can handle. If you tell the system that “the tool I put down is my tool,” it adds that fact to its knowledge base. You can then update the robot with more information about other objects and have it execute a range of tasks like picking up different sets of objects based on different commands.
“Where humans understand the world as a collection of objects and people and abstract concepts, machines view it as pixels, point-clouds, and 3-D maps generated from sensors,” says CSAIL postdoc Rohan Paul, one of the lead authors of the paper. “This semantic gap means that, for robots to understand what we want them to do, they need a much richer representation of what we do and say.”
The team tested ComText on Baxter, a two-armed humanoid robot developed for Rethink Robotics by former CSAIL director Rodney Brooks.
The project was co-led by research scientist Andrei Barbu, alongside research scientist Sue Felshin, senior research scientist Boris Katz, and Professor Nicholas Roy. They presented the paper at last week’s International Joint Conference on Artificial Intelligence (IJCAI) in Australia.
How it works
Things like dates, birthdays, and facts are forms of “declarative memory.” There are two kinds of declarative memory: semantic memory, which is based on general facts like the “sky is blue,” and episodic memory, which is based on personal facts, like remembering what happened at a party.
Most approaches to robot learning have focused only on semantic memory, which obviously leaves a big knowledge gap about events or facts that may be relevant context for future actions. ComText, meanwhile, can observe a range of visuals and natural language to glean “episodic memory” about an object’s size, shape, position, type and even if it belongs to somebody. From this knowledge base, it can then reason, infer meaning and respond to commands.
“The main contribution is this idea that robots should have different kinds of memory, just like people,” says Barbu. “We have the first mathematical formulation to address this issue, and we’re exploring how these two types of memory play and work off of each other.”
With ComText, Baxter was successful in executing the right command about 90 percent of the time. In the future, the team hopes to enable robots to understand more complicated information, such as multi-step commands, the intent of actions, and using properties about objects to interact with them more naturally.
For example, if you tell a robot that one box on a table has crackers, and one box has sugar, and then ask the robot to “pick up the snack,” the hope is that the robot could deduce that sugar is a raw material and therefore unlikely to be somebody’s “snack.”
By creating much less constrained interactions, this line of research could enable better communications for a range of robotic systems, from self-driving cars to household helpers.
“This work is a nice step towards building robots that can interact much more naturally with people,” says Luke Zettlemoyer, an associate professor of computer science at the University of Washington who was not involved in the research. “In particular, it will help robots better understand the names that are used to identify objects in the world, and interpret instructions that use those names to better do what users ask.”
The Latest on: Contextual commands
How to do math on the Linux command line
on April 17, 2018 at 7:09 am
The example below shows that using expr in this kind of context doesn't work because if works with the opposite orientation (0=true). The factor command works just like you'd probably expect. You feed it a number, and it tells you what its factors are. […]
Google Home: Your best friend who instantly heeds commands (Tech Review)
on April 17, 2018 at 12:05 am
Let's see how Google Home fares in the Indian context. The speaker is capable of understanding ... that light up when the device recognises the opening "OK Google" voice command. The two microphones utilise the neural beam forming technique to enable ... […]
The Power Of Commands And Events 🔊
on April 13, 2018 at 7:20 am
That subsystem is responsible for sending a message to the Human representing the command to open the Door. In this context, the Initialization is the Client; the Human and the Door are the Servers. When the Human subsystem receives the message to "open ... […]
5 'stat' Command Examples for Linux Newbies
on April 11, 2018 at 4:00 am
Tecmint: stat command is a useful utility for viewing file or file system status. It retrieves information such as file type; access rights in octal and human-readable; SELinux security context string; time of file birth, last access, last data ... […]
How to add a new file type to the Microsoft Windows 10 Context Menu
on April 10, 2018 at 6:38 am
If you right-click on the empty space of an open window in File Explorer or on the Desktop itself, you will be presented with a set of context-sensitive menu items. Besides the common commands like Share, View, and Control Panel there will be an expandable ... […]
Blue Jays’ Stroman battles command in unusual loss
on April 7, 2018 at 11:43 pm
To put that in context, 5,622, or 64 per cent ... “I’ve never seen that before,” manager John Gibbons said of Stroman’s lack of command. “The stuff was coming out good, he just couldn’t find the zone. He held them in check, they got to him ... […]
Researchers Teach AI To Understand Contextual Commands
on August 31, 2017 at 8:33 am
If there is one thing that separates humans from robots is context. As humans we understand context and subtext, where if we drop something on the ground, we can tell a friend to “pick it up” and where our friend will know what “it” is. Robots ... […]
Robots that understand contextual commands
on August 31, 2017 at 5:00 am
ComText allows robots to understand contextual commands such as, “Pick up the box I put down.” Credit: Tom Buehler/MIT CSAIL Despite what you might see in movies, today's robots are still very limited in what they can do. They can be great for many ... […]
MIT CSAIL teaches a robot to follow contextual voice commands
on August 30, 2017 at 11:31 am
MIT’s Computer Science and Artificial Intelligence Lab has devised a method by which robots can understand and respond to voice commands, stated in clear, plain language. The system is advanced enough to understand contextual commands, too, including ... […]
Microsoft adds contextual edit commands to Office for iPhone, Google makes Now more iPad-friendly
on April 11, 2016 at 5:27 pm
Microsoft on Monday updated its Office apps for iOS with new contextual functions on the iPhone, while Google updated its iOS search app to better sort Now information cards on the iPad. The latest version of Google's search app, meanwhile, organizes Now ... […]
via Google News and Bing News