Dec 9, 2015 | By Alec
If you’ve ever worked with 3D scanners for a 3D printing project, for example, you’ll have seen how those machines essentially work with countless 2D photos. If you step into one of those 3D printing selfie booths, you’re photographed from all sides, and the results are stitched together to form a single 3D image that can be 3D printed. But do you need to scan someone to do so? Can’t you just rely on existing photographs? Well, it turns out you can. A team of University of Washington scientists have developed a revolutionary 3D model algorithm that relies on the endless number of celebrity images on the web to make 3D models. This tool is so advanced, they can even mimic facial mannerisms when talking, or project one celebrity’s (such as Tom Hanks) mannerisms onto another’s face.
This fascinating tool is best understood when looking at the clip below, in which the facial mannerisms of celebrities like Tom Hanks and former President George Bush are projected onto other 3D facial reconstructions, from Neil Patrick Harris to President Obama. This revolutionary 3D imaging tool has been developed over a lengthy process that has taken five years, and has been led by graduate student Supasorn Suwajanakorn and assistant professor of computer science and engineering Ira Kemelmacher-Shlizerman. The results are now finally presented in a paper at the International Conference on Computer Vision in Chile on Dec. 16.
And as lead author Suwajanakorn explains, the entire goal of the tool is trying to find out what makes a person look like himself when he’s talking, and how facial patterns can be predicted in 3D. ‘One answer to what makes Tom Hanks look like Tom Hanks can be demonstrated with a computer system that imitates what Tom Hanks will do,’ he says. To recreate this, they have built a fantastic 3D facial reconstruction tool that touches every level of 3D design – from tracking, alignment, multi-texture modeling and puppeteering.
While it might be easier to do so with a hypothetical face or by using a facial scan of one of their team members, they have even demonstrated how it’s possible for the tool’s algorithms to rely on nothing but an online photo collection. And as you can see in the clip, a well-photographed celebrity is all the tool needs to create a realistic 3D image – provided there’s enough visual data to mine. While they have been quite successful over the last few years already, their latest success – to transfer expressions onto another person - is particularly interesting for 3D modelers and animators. Now it’s possible to make Obama talk like Bush, and the other way around.
The goal of this entire tool, as the researchers explain, is to be able to breathe new life into people that have passed away years ago. Think of the photos in the Harry Potter world, where the characters in the photo continue to live long after they’ve died. With this tool, a family photo album can be taken to create new images, even videos of people that you haven’t seen for years. ‘You might one day be able to put on a pair of augmented reality glasses and there is a 3-D model of your mother on the couch,’ said other senior author Kemelmacher-Shlizerman. ‘Such technology doesn’t exist yet — the display technology is moving forward really fast — but how do you actually re-create your mother in three dimensions?’ Now that would be a VR app that would be truly useful.
You could even, they speculate, talk to your heroes or to historical persons without actually talking to them. ‘Imagine being able to have a conversation with anyone you can’t actually get to meet in person — LeBron James, Barack Obama, Charlie Chaplin — and interact with them’ said co-author Steve Seitz, professor of computer science and engineering at the University of Washington. ‘We’re trying to get there through a series of research steps. One of the true tests is can you have them say things that they didn’t say but it still feels like them? This paper is demonstrating that ability.’
Of course, hologram technology already exists, but it relies on high level existing footage or on 3D scans – something impossible to get from someone who has already died. Even a high quality animated film requires painstaking attention to every single detail – something that this algorithm is capable of generating automatically. The idea is that even your video game character could be modelled after you using just a few photos. As you can imagine, making 3D printed selfies (or models of celebrities) will become easier than ever before too.
For now, however, this imaging tool requires a lot of data to create these renderings. A minimum of 200 images of celebrities in different scenarios and poses were required for the renderings of Tom Hanks, Barack Obama, Daniel Craig and the rest. ‘We asked, ‘Can you take Internet photos or your personal photo collection and animate a model without having that person interact with a camera?” said Kemelmacher-Shlizerman. ‘Over the years we created algorithms that work with this kind of unconstrained data, which is a big deal.’
The real challenge is in capturing those very detailed expression-dependent textures – such as the exact movement of your mouth when smiling or opening your mouth. This is key in transporting one person’s speech onto another. Suwajanakorn only recently had a breakthrough in this process, which was achieved by manipulating lighting conditions in photographs. This enables them to ‘control’ the model of a person and implement another person’s speech into it. ‘How do you map one person’s performance onto someone else’s face without losing their identity?’ said Seitz. ‘That’s one of the more interesting aspects of this work. We’ve shown you can have George Bush’s expressions and mouth and movements, but it still looks like George Clooney.’ This could be a breakthrough with effects for modeling fields from 3D printing to animation and video game development. For more information on the highly complex algorithm behind this innovation, check out their research paper here.
Posted in 3D Software
Maybe you also like:
- Materialise's Mimics 3D imaging software aids investigation into 430000-year-old murder
- 3D printing for schools just got easier: 3DPrinterOS releases educational and premium licenses
- Free and open source online 3D modeling tool CraftML launches beta version
- Ultimaker releases the latest version of their free Cura software
- MakerBot’s new iPad app can now turn your drawings into 3D printable models
- UNTITLED Creator is a fun app for making 3D printable Minecraft-esque designs
- Autodesk's free Tinkerplay app lets you design, customize & 3D print your own characters
- Lumi Industries releases free 'Text to 3D Braille Converter' to support communication with blind
- Onshape raises $65 million for CAD software, public beta goes live
Um all the pictures are jacked up.
3Ders.org provides the latest news about 3D printing technology and 3D printers. We are now seven years old and have around 1.5 million unique visitors per month.
