To prove that sound can be an asset to robots, SCS researchers built a dataset by recording video and audio of 60 common objects as they rolled around a tray. The team captured these interactions using Tilt-Bot — a square tray attached to the arm of a Sawyer robot.
Carnegie Mellon builds dataset capturing interaction of sound, action, vision
People rarely use just one sense to understand the world, but robots usually only rely on vision and, increasingly, touch. Carnegie Mellon University researchers find that robot perception could improve markedly by adding another sense: hearing.
In what they say is the first large-scale study of the interactions between sound and robotic action, researchers at CMU’s Robotics Institute found that sounds could help a robot differentiate between objects, such as a metal screwdriver and a metal wrench. Hearing also could help robots determine what type of action caused a sound and help them use sounds to predict the physical properties of new objects.
“A lot of preliminary work in other fields indicated that sound could be useful, but it wasn’t clear how useful it would be in robotics,” said Lerrel Pinto, who recently earned his Ph.D. in robotics at CMU and will join the faculty of New York University this fall. He and his colleagues found the performance rate was quite high, with robots that used sound successfully classifying objects 76 percent of the time.
The results were so encouraging, he added, that it might prove useful to equip future robots with instrumented canes, enabling them to tap on objects they want to identify.
The researchers presented their findings last month during the virtual Robotics Science and Systems conference. Other team members included Abhinav Gupta, associate professor of robotics, and Dhiraj Gandhi, a former master’s student who is now a research engineer at Facebook Artificial Intelligence Research’s Pittsburgh lab.
To perform their study, the researchers created a large dataset, simultaneously recording video and audio of 60 common objects — such as toy blocks, hand tools, shoes, apples and tennis balls — as they slid or rolled around a tray and crashed into its sides. They have since released this dataset, cataloging 15,000 interactions, for use by other researchers.
The team captured these interactions using an experimental apparatus they called Tilt-Bot — a square tray attached to the arm of a Sawyer robot. It was an efficient way to build a large dataset; they could place an object in the tray and let Sawyer spend a few hours moving the tray in random directions with varying levels of tilt as cameras and microphones recorded each action.
They also collected some data beyond the tray, using Sawyer to push objects on a surface.
Though the size of this dataset is unprecedented, other researchers have also studied how intelligent agents can glean information from sound. For instance, Oliver Kroemer, assistant professor of robotics, led research into using sound to estimate the amount of granular materials, such as rice or pasta, by shaking a container, or estimating the flow of those materials from a scoop.
Pinto said the usefulness of sound for robots was therefore not surprising, though he and the others were surprised at just how useful it proved to be. They found, for instance, that a robot could use what it learned about the sound of one set of objects to make predictions about the physical properties of previously unseen objects.
“I think what was really exciting was that when it failed, it would fail on things you expect it to fail on,” he said. For instance, a robot couldn’t use sound to tell the difference between a red block or a green block. “But if it was a different object, such as a block versus a cup, it could figure that out.”
The Latest Updates from Bing News & Google News
Go deeper with Bing News on:
- Pandemic teaching, in their wordson October 6, 2020 at 9:44 am
Across America, the coronavirus pandemic has rewritten the syllabus for the 2020-2021 school year. Teachers are facing formidable challenges, whether educating students in masked-up, socially distant ...
- GITAI Aims to Bring Autonomous Robot Workers to Spaceon October 5, 2020 at 4:49 am
Since its start, GITAI has focused its efforts on various robots for space. Its G1, which is a humanoid torso, was its most advanced machine. It was designed to be controlled by an immersive ...
- Dallas artist David-Jeremiah prefers to let his work do the talking, but his story is worth hearingon October 2, 2020 at 4:27 am
Jeremiah has just had his work stolen. The interactive works he premiered in early July included stacks of cash donated by the public, placed in a series of offering plates. Sitting on the patio of ...
- ‘Kid A’ at 20: Why Radiohead’s Futuristic Masterpiece Sounds Right on Timeon October 2, 2020 at 2:00 am
Something about the band’s robot-blues reinvention seemed uncannily perfect for the fall of 2000 — the same thing that makes it perfect for the fall of 2020 ...
- Will We Go Back To Being Robots After Covid-19?on October 1, 2020 at 7:17 am
When it comes to our professional lives, the best positive impact of the coronavirus is the way it has made business less robotic and more human. Will it stay that way?
Go deeper with Google Headlines on:
Go deeper with Bing News on:
- The future of automation: Robots can help us be better humanson October 6, 2020 at 2:29 pm
Experts from the Advanced Robotics for Manufacturing Institute, Nividous and Pittsburgh Robotics Network talk about how education can help people advance in the robotics industry — and why we ...
- Miso Robotics’ Flippy ROAR Announces Global Commercial Availabilityon October 6, 2020 at 11:05 am
Miso Robotics announced global commercial availability of the Flippy Robot-on-a-Rail (ROAR) – a zero-footprint, cost-efficient intelligent robotic kitchen assistant for the evolving commercial kitchen ...
- Center City robotics startup acquired by Pittsburgh's Seegridon October 6, 2020 at 7:27 am
Seegrid, a Pittsburgh maker of self-driving industrial vehicles, said it bought Box to add the two-person company’s expertise in 3D lidar data to Seegrid’s existing navigation platform. Seegrid’s ...
- Seegrid Acquires Box Roboticson October 6, 2020 at 6:12 am
Seegrid Corporation, the leader in self-driving industrial vehicles for material handling, today announced that it acquired Box Robotics, a mobile robotics startup bringing high-definition maps, three ...
- Sarcos Defense Awarded Contract by U.S. Air Force to Develop "Smart" Dexterous Robotic Systems with Advanced Artificial Intelligenceon October 6, 2020 at 6:05 am
Sarcos Defense Awarded Contract by U.S. Air Force to Develop "Smart" Dexterous Robotic Systems with Advanced Artificial Intelligence ...