Technique from Computer Science and Artificial Intelligence Lab could improve augmented reality and reduce the need for CGI green-screens.
We learn a lot about objects by manipulating them: poking, pushing, prodding, and then seeing how they react.
We obviously can’t do that with videos — just try touching that cat video on your phone and see what happens. But is it crazy to think that we could take that video and simulate how the cat moves, without ever interacting with the real one?
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently done just that, developing an imaging technique called Interactive Dynamic Video (IDV) that lets you reach in and “touch” objects in videos. Using traditional cameras and algorithms, IDV looks at the tiny, almost invisible vibrations of an object to create video simulations that users can virtually interact with.
“This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space,” says CSAIL PhD student Abe Davis, who will be publishing the work this month for his final dissertation. “By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos.”
Davis says that IDV has many possible uses, from filmmakers producing new kinds of visual effects to architects determining if buildings are structurally sound. For example, he shows that, in contrast to how the popular Pokemon Go app can drop virtual characters into real-world environments, IDV can go a step beyond that by actually enabling virtual objects (including Pokemon) to interact with their environments in specific, realistic ways, like bouncing off the leaves of a nearby bush.
He outlined the technique in a paper he published earlier this year with PhD student Justin G. Chen and professor Fredo Durand.
How it works
The most common way to simulate objects’ motions is by building a 3-D model. Unfortunately, 3-D modeling is expensive, and can be almost impossible for many objects. While algorithms exist to track motions in video and magnify them, there aren’t ones that can reliably simulate objects in unknown environments. Davis’ work shows that even five seconds of video can have enough information to create realistic simulations.
To simulate the objects, the team analyzed video clips to find “vibration modes” at different frequencies that each represent distinct ways that an object can move. By identifying these modes’ shapes, the researchers can begin to predict how these objects will move in new situations.
“Computer graphics allows us to use 3-D models to build interactive simulations, but the techniques can be complicated,” says Doug James, a professor of computer science at Stanford University who was not involved in the research. “Davis and his colleagues have provided a simple and clever way to extract a useful dynamics model from very tiny vibrations in video, and shown how to use it to animate an image.”
Davis used IDV on videos of a variety of objects, including a bridge, a jungle gym, and a ukelele. With a few mouse-clicks, he showed that he can push and pull the image, bending and moving it in different directions. He even demonstrated how he can make his own hand appear to telekinetically control the leaves of a bush.
“If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond to existing forces and assume that it will respond in a consistent way to new ones,” says Davis, who also found that the technique even works on some existing videos on YouTube.
Researchers say that the tool has many potential uses in engineering, entertainment, and more.
For example, in movies it can be difficult and expensive to get CGI characters to realistically interact with their real-world environments. Doing so requires filmmakers to use green-screens and create detailed models of virtual objects that can be synchronized with live performances.
But with IDV, a videographer could take video of an existing real-world environment and make some minor edits like masking, matting, and shading to achieve a similar effect in much less time — and at a fraction of the cost.
Engineers could also use the system to simulate how an old building or bridge would respond to strong winds or an earthquake.
“The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn’t harm the real-world counterpart,” says Davis.
He says that he is also eager to see other applications emerge, from studying sports film to creating new forms of virtual reality.
“When you look at VR companies like Oculus, they are often simulating virtual objects in real spaces,” he says. “This sort of work turns that on its head, allowing us to see how far we can go in terms of capturing and manipulating real objects in virtual space.”
The Latest on: Interactive Dynamic Video
via Google News
The Latest on: Interactive Dynamic Video
- MPK Interactive Rebrands as Neon Media Group, adds Key Talenton October 14, 2019 at 6:02 am
NASHVILLE, Tenn., Oct. 14, 2019 /PRNewswire/ -- MPK Interactive, a Nashville-based video production company, today announces a rebranding as Neon Media Group (NMG). Neon Media Group is a full-service ...
- Hawkeye Systems' Radiant Images Installs Volumetric Light Field Capture System - AXA Stage for Sony Innovation Studioson October 10, 2019 at 1:06 pm
and we're pleased to be working with Sony as partners to enhance the future of interactive experiences for audiences," said Corby Marshall, founder, and chief executive officer of Hawkeye Systems. "We ...
- Mavenir Introduces Mobile Business Fabric To Transform Business Communications For Deskless Mobile Workerson October 9, 2019 at 11:12 pm
Mavenir, the industry’s only end-to-end, cloud-native Network Software Provider, today announced its Mobile Business Fabric™, a dynamic ... mobile video, rich communication services (RCS) and SMS ...
- ‘Wake up your inner Bad Boy’: Puig launches interactive Carolina Herrera Bad Boy animation at Barcelona Airporton October 8, 2019 at 4:16 pm
The dynamic activation features plenty of Puig’s experiential innovations ... The accompanying campaign stars British actor Ed Skrein wreaking havoc in a busy street – where he encounters Good Girl ...
- Dreamcatcher Feel 'Deja Vu' In Fantastical New Music Video: Watchon September 18, 2019 at 7:33 pm
K-pop girl group Dreamcatcher may be moving away from their haunting Nightmare-themed series, but are still weaving a magical dreamworld in the music video for their newest single, “Deja Vu.” Released ...
- 'Interactive Dynamic Video' Could Make Pokémon Go Even More Engagingon August 1, 2019 at 5:00 pm
lead author of the research Abe Davis shows how videos of a wire figure on a vibrating surface, a bush blown by the wind, and even YouTube clips can be manipulated using this technique — which Davis ...
- VideoTap to unveil interactive smart video platform, to deliver dynamic, personalised VODon July 28, 2019 at 5:00 pm
25 Feb, 2017 - 09:20 AM IST | By indiantelevision.com Team MUMBAI: Eros Now, the cutting-edge digital over-the-top (OTT) South Asian entertainment platform owned by Eros International Plc (NYSE: EROS) ...
- Interactive Whiteboard Market: A New Facet of Dynamic Learning, to Register a CAGR of 7.4% during 2019-2025on May 30, 2019 at 6:04 am
NEW YORK, May 30, 2019 (GLOBE NEWSWIRE) -- The global interactive whiteboard market is expected to witness a CAGR of 7.4% from 2019 to 2025 to reach 2,540 million by the year 2025. Advancements in ...
- Interactive Display Market Status, Dynamic, Growth, Share And Foresight 2019-2028on April 17, 2019 at 12:48 am
Interactive monitors, interactive kiosks, interactive tables, interactive video wall, and interactive whiteboard are types of interactive displays. The research report on Global Interactive Display ...
- How Dynamic Video Will Change the Way You Advertiseon November 28, 2017 at 4:00 pm
However, even with that strategy creating videos can still be resource intensive and extremely time consuming. So what’s a 21st century marketer to do? What is Dynamic Video? Luckily some companies ...
via Bing News