Real-Time Detector Sees Hand Poses and Tracks Multiple People
Researchers at Carnegie Mellon University’s Robotics Institute have enabled a computer to understand the body poses and movements of multiple people from video in real time — including, for the first time, the pose of each individual’s fingers.
This new method was developed with the help of the Panoptic Studio, a two-story dome embedded with 500 video cameras. The insights gained from experiments in that facility now make it possible to detect the pose of a group of people using a single camera and a laptop computer.
Yaser Sheikh, associate professor of robotics, said these methods for tracking 2-D human form and motion open up new ways for people and machines to interact, and for people to use machines to better understand the world around them. The ability to recognize hand poses, for instance, will make it possible for people to interact with computers in new and more natural ways, such as communicating with computers simply by pointing at things.
Detecting the nuances of nonverbal communication between individuals will allow robots to serve in social spaces, allowing robots to perceive what people around them are doing, what moods they are in and whether they can be interrupted. A self-driving car could get an early warning that a pedestrian is about to step into the street by monitoring body language. Enabling machines to understand human behavior also could lead to new approaches to behavioral diagnosis and rehabilitation for conditions such as autism, dyslexia and depression.
“We communicate almost as much with the movement of our bodies as we do with our voice,” Sheikh said. “But computers are more or less blind to it.”
In sports analytics, real-time pose detection will make it possible for computers to not only track the position of each player on the field of play, as is now the case, but also to know what players are doing with their arms, legs and heads at each point in time. The methods can be used for live events or applied to existing videos.
To encourage more research and applications, the researchers have released their computer code for both multiperson and hand-pose estimation. It already is being widely used by research groups, and more than 20 commercial groups, including automotive companies, have expressed interest in licensing the technology, Sheikh said.
Sheikh and his colleagues will present reports on their multiperson and hand-pose detection methods at CVPR 2017, the Computer Vision and Pattern Recognition Conference, July 21–26 in Honolulu.
Tracking multiple people in real time, particularly in social situations where they may be in contact with each other, presents a number of challenges. Simply using programs that track the pose of an individual does not work well when applied to each individual in a group, particularly when that group gets large. Sheikh and his colleagues took a bottom-up approach, which first localizes all the body parts in a scene — arms, legs, faces, etc. — and then associates those parts with particular individuals.
The challenges for hand detection are even greater. As people use their hands to hold objects and make gestures, a camera is unlikely to see all parts of the hand at the same time. Unlike the face and body, large datasets do not exist of hand images that have been laboriously annotated with labels of parts and positions.
But for every image that shows only part of the hand, there often exists another image from a different angle with a full or complementary view of the hand, said Hanbyul Joo, a Ph.D. student in robotics. That’s where the researchers made use of CMU’s multicamera Panoptic Studio.
“A single shot gives you 500 views of a person’s hand, plus it automatically annotates the hand position,” Joo explained. “Hands are too small to be annotated by most of our cameras, however, so for this study we used just 31 high-definition cameras, but were still able to build a massive data set.”
Joo and Tomas Simon, another Ph.D. student, used their hands to generate thousands of views.
“The Panoptic Studio supercharges our research,” Sheikh said. It now is being used to improve body, face and hand detectors by jointly training them. Also, as work progresses to move from the 2-D models of humans to 3-D models, the facility’s ability to automatically generate annotated images will be crucial.
When the Panoptic Studio was built a decade ago with support from the National Science Foundation, it was not clear what impact it would have, Sheikh said.
“Now, we’re able to break through a number of technical barriers primarily as a result of that NSF grant 10 years ago,” he added. “We’re sharing the code, but we’re also sharing all the data captured in the Panoptic Studio.”
In addition to Sheikh, the multiperson pose-estimation research included Simon and master’s degree students Zhe Cao and Shih-En Wei. The hand-detection study included Sheikh, Joo, Simon and Iain Matthews, an adjunct faculty member in the Robotics Institute. Gines Hidalgo Martinez, a master’s degree student, also collaborates on this work, managing the source code.
Learn more: A Computer That Reads Body Language
The Latest on: A computer that can read body language
- Here's What Your Partner's Body Language On FaceTime Dates Revealson May 11, 2020 at 2:56 pm
Eyes darting away from you. Crossed arms. A smile that lights up their whole face. Leaning in to listen. As they say, actions speak louder than words — and all of these subtle physical cues can say a ...
- Biased language paints an unfair, incomplete picture of “old people” today | Opinionon May 11, 2020 at 12:41 pm
If the coronavirus pandemic has taught us anything, it is that our lives, and our health, are deeply intertwined. My well-being is connected to yours — and to all the people we each know, love and ...
- How songwriters are learning to collaborate virtually amid coronavirus: 'It's hard to read energy'on May 10, 2020 at 10:59 am
Some songwriters forge ahead via Zoom or other virtual programs. Others decided to push "pause" on the creative process.
- Finding ways to move your body while social distancingon May 8, 2020 at 6:11 am
Recent Fitbit activity tracker data show a significant drop in physical activity worldwide that corresponds with the onset of the COVID-19 crisis. In the United States, physical activity has fallen by ...
- Facebook Names the 20 People Who Can Overrule Mark Zuckerbergon May 6, 2020 at 10:00 am
The new oversight board will review decisions to remove content. The members are accomplished professionals who are independent of the company.
- Meghan Markle’s Body Language Shows She Still Has Dominance Over Prince Harry, According to Expertson April 30, 2020 at 11:48 pm
Initially, Meghan and Prince Harry relocated to Canada when they moved out of their royal housing. That didn’t last for long, however. The pair appear to be settling down in Malibu, and the ...
- Harry and William's Body Language Was So On Edge During Charles and Camilla's Weddingon April 30, 2020 at 5:04 pm
Can you believe its been a week since we all realized Prince Harry and Prince William ... In the spirit of free time and the 15th anniversary of Charles and Camila's wedding this month, we did a deep ...
- 12 Places to Find Fat- and Body-Positive Workouts You Can Do at Homeon April 29, 2020 at 12:35 pm
Lately though, relying on workouts has come with some obstacles. Gyms and recreation centers are closed, and finding space to work out safely outside can be tricky. In response to the numbers of ...
- Melania Trump body language show how relationship with Donald has changed in White Houseon April 29, 2020 at 10:13 am
MELANIA TRUMP, 50, is the wife of the US President Donald Trump, 73. Since becoming First Lady, her main focus has changed, a body language expert told Express.co.uk.
- Melania Trump body language reveals why she did not 'embrace' role of First Ladyon April 23, 2020 at 4:50 am
However, a body language expert revealed why she may have ... Be Best initiative which focuses on the well-being of children. READ MORE: Melania Trump: How she pays for designer looks after ...
via Google News and Bing News