By: Georgia Calawerts, Anjana Kankanala, Kody Proctor, Carolina Quintana, & Delaney Sears
INTRODUCTION
This research was created in fulfillment of the Master of Entertainment Industry Management degree at Carnegie Mellon University by the following team of graduate students: Georgia Calawerts, Anjana Kankanala, Kody Proctor, Carolina Quintana, & Delaney Sears. In this article, we focus on frameworks and industry trends of virtual production. In Part II, we share the results of our interviews, national survey, and recommendations.
Virtual production, defined in this project as the use of real-time technologies to combine traditional physical filmmaking with digital assets, has become a rapidly growing practice in the entertainment industry since LED screens were first used in The Mandalorian (2019) (Desowitz, 2020). LED stages have grown exponentially from three stages in 2019 to over 300 stages in 2022 (Giardina, 2022), expanding virtual production to now be a $1.8 billion market (Grand View, 2023). This project investigates the current landscape of LED stages and studios can capitalize on the growing market.
Image: LED Screen
Image Source: Unsplash
Our team analyzed virtual production’s artistic opportunities and challenges to help determine the best practices. We found the most common factors considered by industry professionals when determining to use LED volumes in their projects included the pipeline of trained technicians and the benefits and obstacles of the new technology. As virtual production is employed by multiple departments and artists working within
film, TV, and gaming, we interviewed 20 experts across industry disciplines. Those interviewed included virtual production executives, directors of photography, producers, gaming professionals, and educators. Companies represented in these interviews included Sony, Industrial Light and Magic, Paramount, William F. White, and New York University. A survey was also distributed to entertainment industry professionals to gauge their experiences with virtual production and LED screens.
The project’s fundamental goal is to examine how a studio should approach virtual production and LED volumes to remain an industry leader in this dynamic and evolving production model. Key factors such as labor concerns, artistic insights from creatives, and cost-benefit analyses will help assess how investment in LED volumes can be amortized across production slates and studio verticals, including film, TV, and gaming divisions. The project explored the opportunities and challenges resulting from the increasing cost-effectiveness of virtual production. The following two research questions drove the team’s Study:
How should a studio approach the rapid growth of this new technology, and how will it affect traditional production pipelines?
What are the current concerns - labor, cost, artistic, or otherwise - related to investing in LED stages (or virtual production)?
VIRTUAL PRODUCTION
As of 2022, the virtual production industry has grown to be a $1.8 billion industry (Grand View, 2023). Meanwhile, Fortune Business expects the industry to grow to $7.19 billion industry by 2029 (Fortune Business Insights, 2023). Virtual production combines physical and digital filmmaking techniques that include computer-generated imagery (CGI), visual effects (VFX), augmented reality (AR), virtual reality (VR), and, of most importance to this project, LED screens. Also called volumes, LED soundstages utilize video walls of light-emitting diode (LED) pixels that display 3-Dimensional backdrops in real-time. As opposed to green screen technology, this allows environments to be created in pre-production and shot “in-camera” instead of creating entirely in post-production. Now, it has expanded to be one of the most in-demand technologies in the industry (Desowitz, 2020).
Virtual production eliminates the need to travel to specific locations and allows for real-time manipulation of the environment (Desowitz, 2020). Volumes also work in conjunction with game engines like Unreal to create unique, personalized 3D environments that can be edited in real-time on set. This artistic freedom has pushed virtual production to be one of the fastest-growing technologies in the film industry. The technology is no longer reserved for studio blockbusters and superhero multiverse pictures; there are now more options for smaller productions, with the quality of the screen being the determining factor for the cost of production. Projects that embrace virtual production run the genre gamut, including HBO’s House of the Dragon (2022), Disney’s Thor: Love and Thunder (2022) and Obi-Wan Kenobi (2022), Warner Brothers’ Our Flag Means Death (2022), and James Cameron’s Avatar: The Way of Water (2022).
As LED stage technology and physical spaces grow, studios worldwide are racing to incorporate virtual production into their slates and keep up with a rapidly changing industry. Sony Pictures Entertainment (SPE) is a division of Sony Corporation, a Japanese-owned creative technology company focused on multimedia content production, including motion pictures, TV, video games, music, and electronics. Beyond their well-known technology and gaming divisions, Sony has a library of more than 3,500 films, including 12 Best Picture Academy Award winners (Sony, 2022). This project focused on research specifically for Sony Innovation Studios, a virtual production studio housed within SPE, and their Production and Post-Production Sourcing departments. SPE owns, invests in, and operates the LED screens used in their titles, such as Bullet Train (2022), and has won multiple awards for its advancements in virtual production.
Image: Example of LED Volume on the set of The Mandalorian
Image Source: What Do Actors Need to Know About Virtual Production? by Rob Leane, 2020.
DEFINITION OF TERMS USED
Below is a glossary of need-to-know terms from the Visual Effect Society’s Virtual Production Guide (Visual Effects Society, 2023):
● Augmented Reality (AR): Technology that integrates virtual elements into a physical environment.
● Camera Tracking: The solving of a live-action camera’s position and orientation. In a real-time scenario, these coordinates are sent to the engine enabling the equivalent virtual camera to move identically.
● Real-Time Engine: A software designed for creating and displaying real-time interactive content; initially meant for video games, now used for virtual production. Examples include Unity and Unreal Engine.
● Virtual Production: The use of real-time technologies to combine traditional filmmaking techniques with digital scenes and assets to shift digital innovation and visual effects into pre-production instead of only in post-production.
● Virtual Reality (VR): An immersive experience using headsets to generate realistic sounds, stereo images, and other sensations replicating a real environment or creating an imaginary world.
● Volume: The physical space in which performance capture is recorded. Also refers to a nearly enclosed LED stage in which a volume of light is emitted or a display surface for projected content.
EVOLUTION AND IMPACT OF LED TECHNOLOGY
The use of LED volumes started with the production of The Jungle Book (2016) and The Lion King (2019) (Seymour, 2019). Director Jon Favreau, a pioneer in VFX and LED stage use, employed predominately AR and VR production techniques in both films, helping to create a road map for industry-wide LED screen adoption. Both films used Unity Game Engine, which proved to have issues throughout the filming process, such as the director’s inability to see shots in real-time and other glitches that made it into the film’s final cut (Seymour, 2019). VFX supervisor of The Mandalorian (2019), Richard Bluff, stated that Favreau’s ground-breaking use of emerging game engine technology became the “spark” Industrial Light and Magic (ILM) needed to invest in LED screens for The Mandalorian (R. Bluff, personal communication, 2022).
According to Bluff, using LED screens (named StageCraft by ILM) for the first season was “a eureka moment” for creatives everywhere (personal communication, December 16, 2022). As the first production to utilize a 180-degree LED stage with photo realistic backdrops, they were able to improve and develop revolutionary technology in accurate camera tracking and perspective-correct imaging (Desowitz, 2020) – even if the first season was a “Band-Aid on a Band-Aid on a Band-Aid on a house of cards,” according to Bluff (personal communication, December 16, 2022). The team at ILM worked with the creators of Unreal Game Engine by Epic Games to update software for production needs, leading Unreal to become the gold standard game engine for virtual production. Unreal, according to Noah Kadner, the founder of The Virtual Company and virtual production industry expert, is the best engine for real-time compositing, rendering, and photorealism; if a screen is not using a proprietary engine, like ILM’s Helios, they are probably using Unreal (personal communication, December 14, 2022).
Image: Why is Virtual Production LED Wall Better than Green Screen
Image Source: Rigard
Volume technology became vital as a creation tool during the pandemic. The Mandalorian (2019) had proven that it could work and look good, and the innovation during the pandemic, although initiated purely out of necessity, proved to be successful. The industry quickly learned that multiple locations could be created on a single soundstage, lessening the need to travel. Virtual Production Executive at Amazon Studios, James Thomas, credits the pandemic for the rapid growth of LED screens after 2020:
“There was no desire to skirt COVID...but people have a desire to create. You needed less crew...You could be in a controlled soundstage where everyone could be tested and protocols could be followed. A lot of people used that as a way to kick-start their studio or get more productions through... so a lot of studios put volumes up all over the world.” (J. Thomas, personal communication, January 2, 2023)
Another aspect to consider is the number of carbon emissions created by traveling an entire crew to multiple locations. The pandemic made traditional on-location travel difficult, which proved to help offset the film industry’s carbon footprint. With virtual production, carbon emissions can be cut down from 95 total tons of CO2 on a traditional location shoot to 0.74 total tons of CO2 using LED volumes (Tangcay, 2022).
LED’S ROLE IN FILMMAKING TODAY
With COVID-19 cementing LED screens as an established filming option, several studios and companies have emerged as leaders. ILM and StageCraft continue to be a mainstay in the virtual production space, with the worldwide expansion of its StageCraft volumes to production centers such as London and Sydney. ILM’s investment in improving StageCraft technology continues to this day, with their motto being that they “mold technology to the filmmaker, not the other way around” (ILM, 2022). ILM’s pioneering innovation in the virtual production space has accelerated LED stage production throughout the industry over the past three years. In 2022, Amazon Studios unveiled the largest LED volume in the United States, located in Culver City, CA. The 34,000 square-foot soundstage houses over 3,000 LED panels and motion tracking cameras connected to the Amazon Web Services (AWS) cloud ecosystem, allowing every shot to be uploaded and worked on from anywhere in the world (Sakoui & Lee, 2022). “Amazon’s investment in virtual production is promising if it takes off and Amazon becomes the leader in the space,” said Ryan Patap of CoStar Group (Patap, 2023, as cited in Sakoui & Lee, 2022). With more cash on hand than most other production studios, Amazon positions itself to become a leader with its latest ventures into virtual production. However, the largest LED volume in the world is owned by the Canadian production services company William F. White. With its recent soundstage acquisition in December 2022, William F. White now has over 1.55 million square feet of production space across Canada (Vlessing, 2022). Because of the increasing production costs in Hollywood, William F. White’s Canadian jurisdiction has seen an increase in production traffic thanks to Canadian tax benefits for film production and the conversion rate of 1 Canadian dollar to 0.73 US dollars (Guagenti, 2023). Director of Virtual Production at William F. White, Edward Hanrahan, also credits Canadian tax incentives for the country’s “pretty mature VFX industry,” making Canada an unexpected leader in the virtual production space (personal communication, December 1, 2022).
The other industry leader is Sony. Along with Samsung, both companies are outliers in the virtual production industry because of their in-house machinery creation and proprietary LED technology. Sony created and owns the Crystal LED technology, a type of screen that is already regularly used in the industry but could benefit from further investment to become even stronger (E. Liebman, personal communication, December 1, 2022). This will be examined later in the research as a method of amortizing the high costs of LED screens, as well as standardizing the technology, which can vary anywhere from $3 million to over $300 million to build (Giardina, 2022). In October 2022, Sony acquired Pixomondo, the Emmy and Oscar-winning virtual production and VFX company behind Game of Thrones, to help meet increasing content demands (Tangcay, 2022). Since Pixomondo is Canadian-based, Sony can attract projects that want to shoot in Canada for the above mentioned reasons.
A significant aspect of the current virtual production industry that surfaced is the issue of labor. The rapid explosion of the virtual production landscape in recent years has exposed a significant lag in the labor market in terms of trained technicians and crew. According to Eric Liebman of XRM Media, an independent studio based in LA with tech investments, “training is happening very quickly, but the demand for technicians is far greater than that,” speculating that there are “maybe 30 to 40 people” in the industry who fully know how to work the LED volumes (personal communication, December 1, 2022). To run just one Amazon LED volume in Culver City requires a 20-person team (Sakoui & Lee, 2022). Schools like DePaul University and the University of Southern California have started to offer virtual production courses, but training still occurs on set most of the time. Admittedly, it is “a challenge for any institution to adopt, essentially, a whole industry overnight” (N. Kadner, personal communication, December 14, 2022). Thus, virtual production crews primarily learn as they go. By implementing an effective, standardized training pipeline within Sony or in collaboration with an accredited institution, Sony could solidify their stake as a market leader.
Across the board, the virtual production industry is flourishing. The cost of using this innovative technology depends on the production size and for many, it is still out of reach for smaller-budget projects. While many assume LED screens are mainly used for high-budget projects like The Mandalorian (2019), using screens for music videos, commercials, or even just car scenes allow production companies to amortize their massive cost. In a Hollywood Reporter article about the possibility of LED fatigue, Oscar-winning VFX producer Bob Grossman predicts the one-day democratization [making LED accessible to productions of all sizes] of the industry, stating that there is too much excitement about buying LED and not enough education about the technology or understanding that it is still unattainable for many film and TV budgets (Giardina, 2022).
LED screens still have issues; they will glitch and break like any technology. Ultimately, LED screens need time and practice to perfect through proper training and technology for use in filmmaking. The entertainment industry is working on perfecting the technology in service to create bigger and better content. Comparing the volumes to a bow and arrow, Liebman stated: “If you don’t know how to use it, you can shoot yourself in the foot. If you do know, you can shoot a rabbit a mile away” (personal communication, December 1, 2022). As stated throughout this article, LED volumes are a rapidly growing industry tool. Our subsequent research includes interviews with executives across the physical production pipeline, as well as an industry-specific survey that provided fresh insights into the world of LED’s role in filmmaking.
CONCLUSION
In summary, the globalization of virtual production in film and television has ushered in a transformative era, blending real-time technologies with traditional filmmaking to redefine storytelling. Yet, challenges such as labor shortages and accessibility for smaller-budget productions persist, underscoring the need for standardized training and continued technological refinement. Looking ahead, the future of virtual production promises continued growth and innovation, fueled by collaborative endeavors and interdisciplinary expertise. As studios navigate this dynamic landscape, embracing virtual production not only as a technological tool but as a fundamental shift in creative expression, they invite us to explore new realms of cinematic possibility, shaping the future of entertainment on a global scale. To understand the scope and potential, please continue to Part II for results from interviews and a national survey.
-
About AFI. American Film Institute. (n.d.). Retrieved May 2, 2023, from
https://www.afi.com/about-afi/#:~:text=The%20American%20Film%20Institute%20is,i
nspire%20artists%20and%20audiences%20alike
ASC Staff. (2021, December 17). Sony releases Crystal led C-series, B-series displays. The
American Society of Cinematographers (en-US). Retrieved January 29, 2023, from
https://theasc.com/articles/sony-releases-crystal-led-c-series-b-series-displays
Baker, L. (2016, August 9). Create the Look of Natural Light with LED Screens and Projection
Panels. The Beat: A Blog by PremiumBeat.
https://www.premiumbeat.com/blog/create-the-look-of-natural-light-with-led-screens-a
nd-projection-panels/
Desowitz, B. (2020, May 8). 'the mandalorian' leads the way: Real-time virtual production is
saving Hollywood during the lockdown. IndieWire. Retrieved January 28, 2023, from
https://www.indiewire.com/2020/05/the-mandalorian-real-time-virtual-production-savi
ng-hollywood-lockdown-vfx-1202230120/
Fortune Business Insights. (2023, January 16). Virtual production market size worth USD 7.19
billion by 2029: Report by Fortune Business Insights. GlobeNewswire News Room.
Retrieved April 27, 2023, from
https://www.globenewswire.com/en/news-release/2023/01/16/2589144/0/en/Virtual-Pro
duction-Market-Size-Worth-USD-7-19-Billion-by-2029-Report-by-Fortune-Business-In
Sights.html
Giardina, C. (2022, October 19). Too much volume? the Tech behind 'mandalorian' and 'house
of the dragon' faces growing pains. The Hollywood Reporter. Retrieved February 1, 2023, from
https://www.hollywoodreporter.com/business/digital/volume-house-of-the-dragon-stage
-mandalorian-1235244158/
Grand View Research. (2023). Virtual production market size, share & trends report, 2030.
Virtual Production Market Size, Share & Trends Report, 2030. Retrieved April 7, 2023,
from https://www.grandviewresearch.com/industry-analysis/virtual-production-market
Guagenti, C. (2023, April 27). USD to CAD: Convert United States dollar to Canadian dollar -
forbes advisor. Forbes. Retrieved April 26, 2023, from
https://www.forbes.com/advisor/money-transfer/currency-converter/usd-cad/
ILM. (2022). Stagecraft. Industrial Light Magic. Retrieved February 4, 2023, from
https://www.ilm.com/stagecraft/
Internet Archive. (2011, February 28). SONY COMPUTER ENTERTAINMENT
ANNOUNCES RELEASE of PHYREENGINE TM 3.0. Wayback Machine. Retrieved
April 5, 2023, from
https://web.archive.org/web/20120321154129/http://research.scee.net/files/press/2011-0
3-PhyreEngine.pdf.
Leane, R. (2020, August 24). What do actors need to know about virtual production? -
backstage. What Do Actors Need to Know About Virtual Production? Retrieved April
25, 2023, from
https://www.backstage.com/uk/magazine/article/actors-virtual-production-71540/
Kadner, N. (2022, November 29). Welcome to virtual production foundations - virtual
production foundations video tutorial: Linkedin learning, formerly Lynda.com.
LinkedIn. Retrieved February 5, 2023, from
https://www.linkedin.com/learning/virtual-production-foundations/welcome-to-virtual-
production-foundations?autoplay=true
McKenzie, T. (2023, March 23). Adobe is partnering with Epic Games & Autodesk. 80lv.
Retrieved March 0, 2023, from
https://80.lv/articles/adobe-is-partnering-with-epic-games-autodesk-to-provide-game-cr
eators-with-powerful-tools/
Sakoui, A., & Lee, W. (2022, December 6). Amazon Studios unveils massive virtual production
stage, deepening ties to Culver City. Los Angeles Times. Retrieved February 4, 2023,
from
https://www.latimes.com/entertainment-arts/business/story/2022-12-06/amazon-culver-
city-virtual-production-stage-candy-cane-lane-eddie-murphy
Schaefer, S. (2022, May 24). The LED Screens Used In Our Flag Means Death Were So
Realistic They Made The Crew Seasick. Slash Film.
https://www.slashfilm.com/873215/the-led-screens-used-in-our-flag-means-death-were-
so-realistic-they-made-the-crew-seasick/
Seymour, M. (2019, September 2). How virtual production worked on-set of the lion king.
fxguide. Retrieved January 28, 2023, from
https://www.fxguide.com/fxfeatured/how-virtual-production-worked-on-set-of-the-lion-
king/
Sony. (2022, October 25). Press releases. Sony Pictures Entertainment Announces The
Acquisition Of Pixomondo, An Oscar And Emmy® Award Winning Virtual
Production, Visualization And VFX Company | Sony Pictures Entertainment. Retrieved
February 4, 2023, from https://www.sonypictures.com/corp/press_releases/2022/1031
Sony. (2023). Divisions. DIVISIONS | Sony Pictures Entertainment. Retrieved January 28,
2023, from https://www.sonypictures.com/corp/divisions.html
Sustainable Production Alliance. (2021, March). CARBON EMISSIONS OF FILM AND
TELEVISION PRODUCTION. Retrieved April 7, 2023, from
https://www.greenproductionguide.com/wp-content/uploads/2021/04/SPA-Carbon-Emi
ssions-Report.pdf
Sustainable Production Alliance. (2022, May). SPA SOUNDSTAGE FACILITY SURVEY
KEY TAKEAWAYS. Retrieved April 7, 2023, from
https://www.greenproductionguide.com/wp-content/uploads/2022/08/SPA-Report_05.2
5_FINAL_with_infographic.pdf
Tangcay, J. (2022, May 10). How virtual production is helping to cut costs and reduce carbon
footprint. Variety. Retrieved April 8, 2023, from
https://variety.com/2022/artisans/news/virtual-production-small-budget-1235236717/amp/
Tangcay, J. (2022, October 25). Sony Pictures Entertainment acquires VFX company
Pixomondo (exclusive). Variety. Retrieved April 6, 2023, from
https://variety.com/2022/artisans/news/sony-pictures-entertainment-acquires-vfx-compa
ny-pixomondo-1235413544/
Visual Effects Society. (2023, January 25). The virtual production glossary. The Virtual
Production Glossary. Retrieved February 4, 2023, from
https://www.vpglossary.com/#a-z-listing-letter-G-1