This is part 2 of a two-part series. Read part 1 here.
CGI and motion-capture technology have disrupted filmmaking in the past two decades, and there are no signs that this trend is in decline. In part one, the history of and technology behind CGI and motion-capture is explained in detail. That framework and information will be applied to this second part of the series, which examines how similar technology is leveraged in the “holograms” that have become more prevalent in live performances and museum spaces in the past few years. Understanding what this technology is and how it works will allow arts managers to evaluate whether similar technology can be used to improve the patron experience in their institutions.
What Is a Hologram?
In part one, the de-aging effects applied to Robert DeNiro in The Irishman were described. This example of an aging actor’s more youthful image on-screen evokes similarities to a trend that has brought many to widely misuse the third term that requires explanation and disambiguation: the “hologram.” Holograms have been used as a shortcut term to describe the lately fashionable occurrence of posthumously incorporating the likenesses of deceased performers into live productions or films. This misnomer entered public consciousness upon the “hologram performance” of Tupac Shakur at the Coachella music festival in 2012. The technology used in this performance is not a hologram, but rather an updated version of the “Pepper’s ghost” stage trick that originated in the 1800s. The projected image was entirely two-dimensional and was achieved through the strategic use of a mirror and lighting. A newer iteration of the same mislabeled technology will be used in the recently announced posthumous Whitney Houston tour. These images, though they have the appearance of being three-dimensional from afar, are truly two-dimensional, much like the effect rendered by looking in a mirror. By definition, true holograms are static images captured on a special type of film and exposed using laser light rather than white light. The viewer’s perception of the hologram object changes as he or she moves in orientation to it, whereas no matter where the viewer sits in an audience for a live performance of these “holograms” or in a film, the viewer sees the same projection and image; the changing vantage point does not create a different viewership experience.
Figure 1: Image from the posthumous Whitney Houston tour. Source: Slate.
Though these projections are not technically holograms nor are they very difficult to create, the development of the videos and images that are projected in live performance requires the CGI techniques and capabilities described in part one. For both Shakur and Houston, a plethora of performance videos and images exist that could hypothetically be projected, but they are not high-enough resolution to be directly projected with any convincing realism. One of the marketing points of the Houston tour is that it features new choreography, which cannot be obtained from any archival footage. Footage of past performances, therefore, can act as a 21st-century rotoscope for CGI rendering. Dancers with a physical style iterative of Houston’s can be employed to wear non-optical motion capture suits, and the new choreography’s digital skeleton can be captured in a 3D environment and later projected with Houston’s likeness. Arts managers can take a lesson from the for-profit music industry’s playbook here. In fact, some museums, like the Dalí Museum and the National Comedy Center, already leverage similar technology to engage visitors and give life to their subjects.
As the projection technology is relatively simple, the truly impressive aspect of these “holograms” is nearly identical to the technology used in The Irishman; the main, though obvious, difference is the posthumous aspect. In both the posthumous “holograms” and the de-aging performances, there are visual and auditory reference points upon which to build the entire holistic performance. The industry is being further disrupted through the posthumous performances with no auditory or visual references, like the recent announcement that James Dean, who died in 1955 after appearing in only three films, will play a supporting role in a film to be released in 2020. Said director Anton Ernst, “We searched high and low for the perfect actor…which has some extreme complex character arcs, and after months of research we decided on James Dean.” The director plans to shoot a physical stand-in performer to wear optical motion capture references in addition to casting an actor for the voice work.
This is a fundamental departure from the posthumous “holograms” because those projections still retained some of the performer’s original work through vocal performance. As the film has a new script, there is no recording of James Dean speaking any of the dialogue; the director has noted that another performer will give the physical performance, so Dean’s mannerisms from previous films will also not be transferred. Truly the only aspect of Dean being “cast” in this film is his recognizable image.
Motion capture actors like Andy Serkis, widely recognized as changing the public perception of what constitutes “performance,” maintain that CGI is just digital makeup and wardrobe.
Under this belief, the performer lending his physical mannerisms and voice to the character will effectively be given digital makeup to have the appearance of James Dean. It is unclear why the director would claim that Dean’s image alone would lend for successful performances of “extreme complex character arcs.” In fact, the posthumous projections of Shakur and Houston only succeed because they are perceived from a distance, while Robert De Niro’s actual performance, complete with emotive eyes, are displayed in close-up. The proposed Dean film treads dangerously close to the uncanny valley, as the posthumous performance will be viewed with its lack of underlying performance in close-up.
Can a Star Live Forever Through Commercialization?
This film will likely not benefit from a deep character performance, but from the commodification of its news-making posthumous star, James Dean. The financial incentive to capitalize on the images of beloved deceased celebrities is an extreme extension of Richard Dyer’s theory of the commercialization of stars’ public images in the Hollywood studio system (Dyer). Per Dyer’s theory, the star is a commodity, not a person. As a commodity, the owner and licensor of the image has complete control over how audiences perceive and market it, as they would over any other product. In Dyer’s original context, this commodification was most closely tied to studios exercising complete control of tabloid perceptions of stars, merchandising of their images, and the like; in other words, the studios capitalized on the commodification of the stars’ images and likenesses for their financial benefit. Technology like CGI and motion capture now allows for literal and complete control of the star’s image, effectively creating star “puppets” that can be controlled to say and move however the image-owner sees fit to profit from it. This furthers the commodification even more, fully splitting the stars’ image from any personal actions or free will of the underlying actor. If the model of The Irishman were more widely applied, the commodification of the star image could prolong the careers of bankable stars, potentially enabling studios to increase their profits even further.
Magic City Films, the newly-launched production company producing the James Dean film, has not publicly released its visual effects or CGI budget for the film, which certainly lends for questions as to how successfully this already ambitious endeavor can be accomplished. Though the cost of CGI per frame and per character has decreased significantly since its inception, its ubiquity has created an economic environment in which previously unimaginably large budgets prevail. Not only has this caused fewer mid-budget films to be produced, but it has also created a deeper schism between the companies who use the technology and those who do not. Independent production companies and film producers and directors with small budgets are unable to use motion capture or CGI due to restrictive costs and high demand for visual effects artists. There have been some speculations that AI-powered “deepfake” technology will provide a cheaper alternative for smaller production companies. Though these deepfakes can convince viewers watching on smaller scale mobile and laptop screens, the resolution of this technology is not yet sufficient to be scaled to cinema projection. For smaller production companies to participate in these technologies, either CGI and motion capture must have significant cost reductions, or AI-powered deepfakes must become higher-resolution.
Conclusion
This emergence of CGI and motion capture has already disrupted the film industry’s production processes and budgeting procedures, but the disruptions will likely continue with the recent de-aging and posthumous applications. The effects outside of the film industry have already begun. The music industry’s “holograms” lend a clue for what the future may bring there. The gaming industry has been using motion capture for years to create realistic movements for athletes in video games like Madden NFL. It is certainly possible that motion capture will allow every player in games like this to truly move like their real-life counterparts, down to the victory dances. In the live performance space, motion capture has already been used in conjunction with VR technology to fully immerse the audience in the world of the play, and this will likely only continue with higher resolutions and more lifelike live-renderings. There have been discussions to leverage the same technology used to resurrect James Dean in cultural heritage projects to bring back images of important historical icons like Nelson Mandela. Of course this last example conjures dystopic images of deepfakes becoming more prevalent and realistic, furthering the propagation of fake news. Like those audiophiles who embraced vinyl as streaming music became the industry standard, there has been a niche yet strong embrace of “authenticity” and practical effects in the film industry. From shooting on 35mm to relying largely on practical effects to create the creature in The Shape of Water, this backlash against the digital revolution will likely continue. The proliferation of these technologies may also call into question the intellectual property and licensing rights to use an actor’s image posthumously, as well as the SAG compensation standards for motion-capture and voiceover performances in relation to live-action performances. Fundamentally, the understanding of performance and character has already subtly changed in the public consciousness, and this will only continue as computing power improves and the industry becomes even more dependent on computer-generated imagery.
Resources
B, Benjamin. “Face to Face: Gemini Man.” American Cinematographer, November 11, 2019. https://ascmag.com/articles/gemini-man-set-visit.
B, Benjamin. “Gemini Man: Building a Virtual Face.” American Cinematographer, November 12, 2019. https://ascmag.com/blog/the-film-book/weta-virtual-face-for-gemini-man.
Belson, Ken. “Capturing Football’s Snaps, Crackles, and Pops in Madden NFL.” The New York Times, January 28, 2014. https://www.nytimes.com/2014/01/29/arts/video-games/capturing-footballs-snaps-crackles-and-pops-in-madden-nfl.html.
Bishop, Bryan. “How The Shape of Water’s Visual Effects Turned a Merman Into a Romantic Lead.” The Verge, January 17, 2018. https://www.theverge.com/2018/1/17/16898382/the-shape-of-water-special-effects-vfx-cgi.
Dent, Steve. "What You Need to Know About 3D Motion Capture." Engadget, July 14, 2014. https://www.engadget.com/2014/07/14/motion-capture-explainer/.
Dyer, Richard. Stars. London: BFI Publishing, 1998.
Ebiri, Bilge. “A New Generation of Character Actors.” The New York Times Magazine, November 21, 2017. https://www.nytimes.com/2017/11/21/t-magazine/character-actors-andy-serkis-ben-mendelsohn-don-cheadle.html.
Haysom, Sam. “Robert De Niro Explains How They De-Aged Him for ‘The Irishman’.” Mashable, September 26, 2019. https://mashable.com/video/robert-de-niro-de-aging-the-irishman/.
Heller, Scott. “Extending the Stage with Technology.” The New York Times, January 25, 2019. https://www.nytimes.com/2019/01/25/theater/theater-technology.html.
Hu, Jane C. “The Whitney Houston of the 3D Hologram Tour Will Be Neither 3D Nor a Hologram.” Slate, May 22, 2019. https://slate.com/technology/2019/05/whitney-houston-3d-hologram-tour-technology.html.
Hsu, Jeremy. “Why ‘Uncanny Valley’ Human Look-Alikes Put Us on Edge.” Scientific American, April 3, 2012. https://www.scientificamerican.com/article/why-uncanny-valley-human-look-alikes-put-us-on-edge/.
Kadner, Noah. “Toy Story 4: Creating a Virtual ‘Cooke Look’.” American Cinematographer, November 1, 2019. https://ascmag.com/articles/toy-story-4-creating-a-virtual-cooke-look.
Markoff, John. “Camera System Creates Sophisticated 3-D Effects.” The New York Times, July 31, 2006. https://www.nytimes.com/2006/07/31/technology/31motion.html.
Marr, Bernard. “The Best (And Scariest) Examples of AI-Enabled Deepfakes.” Forbes, July 22, 2019. https://www.forbes.com/sites/bernardmarr/2019/07/22/the-best-and-scariest-examples-of-ai-enabled-deepfakes/#2c95cf912eaf.
Marsh, Calum. “VFX Companies Struggle Even as Their Movies Break Records.” Variety, May 30, 2018. https://variety.com/2018/artisans/production/vfx-summer-box-office-1202824543/
Metz, Cade. “Internet Companies Prepare to Fight the ‘Deepfake’ Future.” The New York Times, November 24, 2019. https://www.nytimes.com/2019/11/24/technology/tech-companies-deepfakes.html.
Reed, Ryan. “Whitney Houston Hologram Tour Plots First 2020 Dates.” Rolling Stone, September 17, 2019. https://www.rollingstone.com/music/music-news/whitney-houston-hologram-tour-2020-dates-image-885666/.
Ritman, Alex. “James Dean Reborn in CGI for Vietnam War Action-Drama.” The Hollywood Reporter, November 6, 2019. https://www.hollywoodreporter.com/news/afm-james-dean-reborn-cgi-vietnam-war-action-drama-1252703.
Serjeant, Jill. “James Dean Set to Appear in a Movie Six Decades After His Death, Horrifying Fans.” Reuters, November 6, 2019. https://www.reuters.com/article/us-people-james-dean/james-dean-set-to-appear-in-a-movie-six-decades-after-his-death-horrifying-fans-idUSKBN1XG34A.
Thompson, Kristin and David Bordwell. Film History: An Introduction. New York, NY: McGraw-Hill, 2010.
Tsukayama, Hayley. “How the Tupac ‘Hologram’ Works.” The Washington Post, April 18, 2012. https://www.washingtonpost.com/business/technology/how-the-tupac-hologram-works/2012/04/18/gIQA1ZVyQT_story.html.
Turney, Drew. “Volume of Effects-Heavy Movies Threatens Perception of VFX.” Variety, September 20, 2018. https://variety.com/2018/artisans/production/effects-heavy-movies-vfx-1202948827/.