Creating character models to fit around motion capture data has been commonplace in movies and video games for at least the last decade. Behind the scenes looks at hit movies like the Marvel Series and the more recent Star Wars Movies show the extent of motion capture data used to create the CGI enriched movies today. One type of performance that has been yet to be replicated with CGI is puppeteering. This week, however, Disney was issued a patent to animate a 3D digital puppet from audio and motion capture data.
Puppeteering can be very resource-intensive, especially for children. A physical puppet needs to be created, stored, maintained, and requires a user capable of both operating the puppet and performing with it at the same time. Disney’s recently patented digital system provides users with an inexpensive, simple method to produce and perform digital puppet shows through gestures and sound.
WHat the invention would do
Disney’s digital puppeteering system only requires raw audio and video inputs to then create an animation of the puppet’s movements, including the proper mouth movements to make the puppet “speak.” This invention enables a way for users to create a puppet show with nothing other than a digital recording and the proper computer equipment. Utilizing a motion tracking algorithm, the system collects video and audio data from the user to animate a 3D digital puppet character model. The digital puppet utilizes the motion data to determine the positioning of the character on screen while the audio data is analyzed and used to animate the mouth movements of the character. The technology is capable of producing a full-scale puppet show production solely through interaction with a computer.
Why Digital Puppeteer Technology is interesting
Disney presents this technology as a solution to children losing interest in traditional puppet shows and seeking new digital forms of entertainment. Disney points out that digital storybooks have increased in popularity, but still limit a child’s ability to animate their own story. Disney sees an opportunity in advanced new ways for users to tell stories and optimize interaction.
Additionally, this digital system can allow users to switch control between multiple puppets via a display, enabling them access to multiple characters over the course of a show.
Written by John DeStefano, Technical Advisor
and Lauren Hawksworth, Marketing Administrator
June 29, 2021