This is part 1 of a two-part series. Continue reading here to learn about holograms and how the technology is being used in the arts.
Over the past twenty years, computer-generated imagery has become nearly ubiquitous in film and television productions. This ubiquity is due to enhanced computing power and higher resolutions coupled with increasingly lower costs. The use of CGI has reached a point in both cost and artistry that the computer-generated or digitally-altered character is for the first time being used outside of the science-fiction and animated genres with some regularity. Not only can directors raise beloved actors from the dead, they can also have their lead actors age (or de-age) forty years, all with a degree of artistry and precision that retains audience believability. Though the fundamentals of these technologies have been in use for some time, the recent advancements in the technology and—perhaps even more importantly—their new applications have begun to disrupt the film industry.
To understand how these technological advancements and their artistic applications in recent years are disrupting the industry, there are three key technologies that must be understood and disambiguated: computer-generated imagery (CGI), motion capture, and holograms. Of these three, CGI is the most foundational technology for the current state of production.
What Exactly Is CGI?
On a fundamental level, CGI is self-descriptive: it is any imagery rendered on screen through the aid of a digital generation system. Prior to CGI, all visual effects for live-action productions were “practical effects.” Practical effects consist of makeup, prosthetics, miniature modeling, or any other physical properties that the camera shoots in synchronicity with the actor’s performance. If a character aged through the course of the film, the age effect was achieved through makeup; if a character were non-human, an actor would either wear some combination of costume, makeup, and prosthetics to give a non-human impression (e.g. Chewbacca from Star Wars), or an artist would create a puppet manned by an out-of-shot puppeteer (e.g. Kermit the Frog). The first step away from practical effects and toward CGI came in 1978, when George Lucas founded Lucasfilm Computer Development Division (LCDD) to develop digital applications in filmmaking. He leveraged technology developed by LCDD in 1982’s TRON, which consisted of 15 minutes of pure CGI and 25 minutes of mixed CGI and live action (Thompson and Bordwell). Lucasfilm shortly thereafter spun off LCDD as Pixar, and the company exclusively used CGI animation in all of its short and feature film productions.
By 2000, Pixar had created the “Geppetto” system of digital character control. Like its puppet-master namesake, the Geppetto system allowed animators to control their digital creations like puppets. The animators would identify points on the characters’ figures that they could control, and the rest of the figure would follow, thereby creating movement, much like the strings of a puppet. Not only did Geppetto ensure that a given character would always move in the same manner, it also dramatically increased the efficiency of computer-generated imagery production. The film industry standard is 24 frames per second, and prior to the Geppetto system, artists created characters and rendered them frame by frame. For a 90-minute film, that is 7.8 million frames to be rendered individually. From this statistic alone, the need for improved efficiency is evident.
In the same year, Disney released Dinosaur, which was the first feature to place CGI animals against live-action backgrounds (Thompson and Bordwell). As all characters were dinosaurs, and therefore did not interact with human actors, there was no need to have “stand-ins” when the crew filmed the real-world backgrounds and environments. By placing CGI creatures in an identifiably real environment, Disney had created the standard for the ambitions of many science-fiction and fantasy films in the following years, including The Lord of the Rings and Harry Potter franchises.
In the mid-2000s, as computing power and memory grew, animators could add complex textures and control hundreds of points on a figure via Geppetto, and with increased efficiency. The greater rendering power also began the widely-accepted misnomer of “3-D animation,” as CGI programs gave characters and sets depth and volume rather than leveraging stereoscopic technology only to be appreciated through special “3-D” glasses. At this point, the level of control was sufficient to create and render non-human characters in a live-action environment with relative success. Productions could now employ CGI animation to create characters like the fantastical Dobby in Harry Potter, rather than using a practical puppet effect. CGI technology had not yet advanced to the point of rendering human characters in live-action environments, largely in part to the “uncanny valley” phenomenon. The theory of the uncanny valley – a term originally coined by Japanese roboticist Masaharo Mori - maintains that a “human appearance or behavior can make an artificial figure seem more familiar for viewers – but only up to a point. The sense of viewer familiarity drops sharply into the uncanny valley once the artificial figure tries but fails to mimic a realistic human.” To avoid such a fate, animated humans, therefore, maintained exaggerated features.
Figure 1: To avoid the uncanny valley of films like Polar Express, character designers for animated films tend to design characters with exaggerated features that suggest a human form, rather than attempt to mimic it exactly. Sources: Screenshots from the movies Polar Express and Up.
Such artistic representations could not hold up in a live-action environment, especially if CGI humans were to interact with human actors, giving audiences immediate points of comparison for both appearance and movement. As CGI technology continued to advance the capabilities of texture, lighting, and shading, movement became the most likely attribute of a character that would land it in the uncanny valley.
The issue of character consistency of movement in a CGI production environment was largely solved by the Geppetto system. A character could be programmed to move in a manner specific to that character, which allowed multiple animators to animate the character’s movements through common programming. Again, this works sufficiently for toys, cars, or other objects of animation that do not have innate or recognizable movement patterns, but when characters’ movements resemble a human or animal with recognizable behavioral patterns, more specific guidelines are required.
To achieve the desired motion effects, animators used a technique called “rotoscoping” for decades. Invented by Max and Dave Fleischer in the mid-1910s, the rotoscope allowed an animator to project live-action films frame by frame onto a piece of paper and trace the outlines of its figures (Thompson and Bordwell). By tracing the actions of the desired object of replication one image at a time, the animator could produce characters that moved naturally as whole figures. Animators in the Disney studios employed this technique to great effect when they created Snow White and the Seven Dwarfs, which is widely recognized as the first successful attempt at animating human characters with naturalistic movements. The theory behind rotoscoping, that of filming a human actor or model to act as the reference upon which to build a character, is the same theory behind the motion capture technology that came into use in the mid-1990s and is now used regularly to create the motions of characters that are completely rendered through CGI.
How Does Motion Capture Technology Work?
Motion capture technology facilitates animators and visual effects artists to acquire and leverage reference points. Motion capture is executed at the shooting stage of production, rather than in post-production. In this process, performers with reference points attached to their bodies are captured by digital cameras, and these reference points, often painted dots or small balls, become the key control points in building a digital figure. The data collected through this process is then uploaded into a motion capture program that connects each of the reference points and creates a digital skeleton that moves exactly as the actor who originated the movement. As these reference points are filmed at the standard 24 frames per second, the output of the program contains the exact number of frames needed for the final film. This output is then sent to CGI artists who apply the digital rendering techniques on top of this skeleton to the desired naturalistic effect.
Unlike the rotoscope, which captures only the outline of the performer, motion capture displays the connections between as many points as is desired. The more reference points placed on a performer, the higher the precision of movement in the final product. These reference points can be either “inertial/non-optical” or “optical.” When inertial motion capture began, the special digital “cameras” required large reference points (around the size of ping pong balls) to correctly collect the data uploaded into the motion capture program. To keep the relative space between each reference dot consistent from shot to shot and to ensure that nothing interfered with the data collection, performers donned head-to-toe motion capture suits with reference dots sewn on.
Figure 2: Actor Andy Serkis using motion capture technology for the film Planet of the Apes. Source: Engadget.
Due to a lack of computing power in the early stages of motion capture, only a small number of reference points could be captured so these suits typically had reference dots at major joints, much like the key points identified by animators within the Geppetto system. This small number of reference points resulted in the “digital skeleton” that relied completely on CGI rendering for the final images; no trace of the original performer’s appearance was included in the image, only the movements. In this inertial motion capture, the vantage point of the “camera” is irrelevant; the captured data reflects the reference points’ relative positions to each other. As a result, the digital skeleton output can have CGI effects applied from any angle, and the character can be moved to fit the other elements of the shot (Thompson and Bordwell). This non-optical system is most often shot on a controlled soundstage.
Optical motion capture systems use a similar system of reference points placed on the performer, but rather than only capturing the movement resulting in a digital skeleton, the reference points are used as visual cues to approximate the actor’s motion. These optical systems became more widely adopted as computing power increased because more reference points could be processed by the motion capture software programs. Reference dots on motion capture suits therefore could become smaller, which resulted in more precise capturing of movement. Due to this increased computing power both for capturing digital data and rendering the resulting images, productions gained flexibility in shooting locations, and motion capture suits could be worn on location. Peter Jackson used this on-location optical motion capture to acclaim with the Gollum character in The Lord of the Rings film trilogy (Thompson and Bordwell). This allowed for direct communication with fellow actors, which helped actors create naturalistic performances with their once-absent digital scene partners. With the capabilities of today’s computing power, reference points can be small enough to be placed on a performer’s face, which increases the precision – and therefore naturalism – of the resulting CGI renderings.
Figure 3: An example of how CGI can be used for reverse aging. Reference points are placed on Will Smith’s face to make the actor appear younger in the movie Gemini Man. Source: Variety.
The optical motion capture technique has evolved to the point of being near “markerless,” and these types of optical systems leverage software to track distinctive references, like an actor’s nose, and a few painted dots on an actor’s face. The optical reference points are now subtle enough that the actor’s image and likeness is not obscured whatsoever, but rather enhanced by CGI. Two films notably used this type of optical motion capture in 2019: Gemini Man and The Irishman. These films are notable in part for the accuracy and precision with which the motion capture software de-ages the well-known actors Will Smith and Robert De Niro, respectively. Both actors have had long careers, and audiences have a collective memory of what these actors looked like when they were truly the ages that the CGI effects aim to replicate. These films’ success in de-aging its stars to a level that avoids the uncanny valley and convinces audiences of its legitimacy is indeed disruptive.
Conclusion
CGI and motion-capture technology have evolved in tandem to disrupt film content and the production pipeline. Artists and curators have been experimenting with similar technologies in a live performance and museum context in recent years. To learn more about how this technology is foundational for “holograms,” and how arts managers can leverage it in their own organizations, read part two of this series.
Resources
B, Benjamin. “Face to Face: Gemini Man.” American Cinematographer, November 11, 2019. https://ascmag.com/articles/gemini-man-set-visit.
B, Benjamin. “Gemini Man: Building a Virtual Face.” American Cinematographer, November 12, 2019. https://ascmag.com/blog/the-film-book/weta-virtual-face-for-gemini-man.
Belson, Ken. “Capturing Football’s Snaps, Crackles, and Pops in Madden NFL.” The New York Times, January 28, 2014. https://www.nytimes.com/2014/01/29/arts/video-games/capturing-footballs-snaps-crackles-and-pops-in-madden-nfl.html.
Bishop, Bryan. “How The Shape of Water’s Visual Effects Turned a Merman Into a Romantic Lead.” The Verge, January 17, 2018. https://www.theverge.com/2018/1/17/16898382/the-shape-of-water-special-effects-vfx-cgi.
Dent, Steve. "What You Need to Know About 3D Motion Capture." Engadget, July 14, 2014. https://www.engadget.com/2014/07/14/motion-capture-explainer/.
Dyer, Richard. Stars. London: BFI Publishing, 1998.
Ebiri, Bilge. “A New Generation of Character Actors.” The New York Times Magazine, November 21, 2017. https://www.nytimes.com/2017/11/21/t-magazine/character-actors-andy-serkis-ben-mendelsohn-don-cheadle.html.
Haysom, Sam. “Robert De Niro Explains How They De-Aged Him for ‘The Irishman’.” Mashable, September 26, 2019. https://mashable.com/video/robert-de-niro-de-aging-the-irishman/.
Heller, Scott. “Extending the Stage with Technology.” The New York Times, January 25, 2019. https://www.nytimes.com/2019/01/25/theater/theater-technology.html.
Hu, Jane C. “The Whitney Houston of the 3D Hologram Tour Will Be Neither 3D Nor a Hologram.” Slate, May 22, 2019. https://slate.com/technology/2019/05/whitney-houston-3d-hologram-tour-technology.html.
Hsu, Jeremy. “Why ‘Uncanny Valley’ Human Look-Alikes Put Us on Edge.” Scientific American, April 3, 2012. https://www.scientificamerican.com/article/why-uncanny-valley-human-look-alikes-put-us-on-edge/.
Kadner, Noah. “Toy Story 4: Creating a Virtual ‘Cooke Look’.” American Cinematographer, November 1, 2019. https://ascmag.com/articles/toy-story-4-creating-a-virtual-cooke-look.
Markoff, John. “Camera System Creates Sophisticated 3-D Effects.” The New York Times, July 31, 2006. https://www.nytimes.com/2006/07/31/technology/31motion.html.
Marr, Bernard. “The Best (And Scariest) Examples of AI-Enabled Deepfakes.” Forbes, July 22, 2019. https://www.forbes.com/sites/bernardmarr/2019/07/22/the-best-and-scariest-examples-of-ai-enabled-deepfakes/#2c95cf912eaf.
Marsh, Calum. “VFX Companies Struggle Even as Their Movies Break Records.” Variety, May 30, 2018. https://variety.com/2018/artisans/production/vfx-summer-box-office-1202824543/
Metz, Cade. “Internet Companies Prepare to Fight the ‘Deepfake’ Future.” The New York Times, November 24, 2019. https://www.nytimes.com/2019/11/24/technology/tech-companies-deepfakes.html.
Reed, Ryan. “Whitney Houston Hologram Tour Plots First 2020 Dates.” Rolling Stone, September 17, 2019. https://www.rollingstone.com/music/music-news/whitney-houston-hologram-tour-2020-dates-image-885666/.
Ritman, Alex. “James Dean Reborn in CGI for Vietnam War Action-Drama.” The Hollywood Reporter, November 6, 2019. https://www.hollywoodreporter.com/news/afm-james-dean-reborn-cgi-vietnam-war-action-drama-1252703.
Serjeant, Jill. “James Dean Set to Appear in a Movie Six Decades After His Death, Horrifying Fans.” Reuters, November 6, 2019. https://www.reuters.com/article/us-people-james-dean/james-dean-set-to-appear-in-a-movie-six-decades-after-his-death-horrifying-fans-idUSKBN1XG34A.
Thompson, Kristin and David Bordwell. Film History: An Introduction. New York, NY: McGraw-Hill, 2010.
Tsukayama, Hayley. “How the Tupac ‘Hologram’ Works.” The Washington Post, April 18, 2012. https://www.washingtonpost.com/business/technology/how-the-tupac-hologram-works/2012/04/18/gIQA1ZVyQT_story.html.
Turney, Drew. “Volume of Effects-Heavy Movies Threatens Perception of VFX.” Variety, September 20, 2018. https://variety.com/2018/artisans/production/effects-heavy-movies-vfx-1202948827/.