Now, grab what you see on your video screen

Bloomberg

Forget 3D, soon you could be able to reach out and grab the things you see on the screen.
Researchers from the US state of Massachusetts’ MIT’s Computer
Science and Artificial Intelligence Laboratory (CSAIL) say that they’ve developed a way of being able to “touch” objects that appear on the screen.
Called Interactive Dynamic Video (IDV), what makes the technique so potentially revolutionary is that it uses algorithms and existing camera technology, rather than highly expensive cutting-edge equipment, and the researchers have even been able to take existing video footage posted to YouTube and achieve results.
“This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space,” said CSAIL PhD student Abe Davis.
So how does it work? Davis will be revealing his and his team’s findings in full in a paper to be published towards the end of August. However, in principle, a piece of video footage of, say a cat — the internet’s visual currency — offers clues as to how that cat would behave in different circumstances to the ones portrayed in the video if you have the right algorithm to identify them and amplify them.
Moving images ‘vibrate’ and the researchers’ algorithm can see these tiny movements and use the information to predict how the object would behave under different circumstances. What’s more, just five seconds of video footage is enough for IDV to work.
At the moment these forces are applied by clicking or pushing a mouse, but if paired with a virtual reality helmet or viewer motion tracking systems, like, for instance a Microsoft Kinect system, viewers could literally reach out and start poking and pushing objects they see on the screen.
But IDV could be revolutionary in more than simply consuming video content. “The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able
to test the stress in a safe virtual environment, in a way that doesn’t
harm the real-world counterpart,” said Davis.