Figure 1: How Deepfake technology is used. Source: Deeptrace Lab.
There are currently over 14 thousand Deepfake videos circulating online as of September 2019—almost a 100% increase over a period of one year. Ninety-nine percent of the circulated videos are in the entertainment industry, mainly in the form of pornography. The use of this technology is undoubtedly expanding. Despite its common inappropriate use, Deepfake technology can surely benefit the entertainment industry, specifically film production. This comes with further consequences to be considered by various parties.
Before going into the potential use of the technology, a simple explanation and background of Deepfake technology is important as a foundation.
What is Deepfake?
Deepfake is a merge between 2 terms which are Deep Learning and Fake. It is a technology that includes teaching software to memorize faces, expressions, movements and even voices of a person so that the machine can later project that information onto another person. A popular use of Deepfake are applications and websites that allow one to manipulate the facial expression of a photo, or the popular social media feature face swap, where users can swap their faces with one another.
Figure 2: Face Capture and Reenactment of RGB Video. Source: Face2Face.
Though Deepfake seems relatively new—the term coming into public awareness only 2 years ago—the idea behind Deepfake is not new at all. In fact, researchers have been using this idea of video manipulation since 2016, without the assigned name to the technology. The story of Deepfake itself began in September 2017 when a user called ‘Deepfakes’ posted a computer-generated pornography video of a famous actress on Reddit, an online community. The post gained popularity, and eventually led to a dedicated subreddit r/deepfakes where users could post fake videos of celebrities’ faces mapped on porn stars. A script for how to produce such videos was also published in a forum where users exchanged the latest know-hows and potential applications of the technology. Two months after its launch, the forum gained 25,000 members and sparked controversial debates about the use of such technologies. Once the technology raised concerns and threats in wider communities, Reddit decided to shut down the forum. But that was just the beginning. Deepfake has since gained wider recognition and been further developed by different users. There are still various Deepfake-related repositories on GitHub, an online software development platform where users can share the know-how of their projects. Apart from that, there are over 20 Deepfake creation communities and online forums. 13 out of these communities have disclosed the number of their users, which is close to 100,000 users.
Another notorious case concerning Deepfake was the release of an application called DeepNude, which allows users to upload photos of women for an algorithm to remove all their clothing. Though the anonymous developer shut DeepNude down within a day of its release, it was enough to show the public its capabilities. Deepfake technology is much more accessible today and users without any computer engineering background can create a fake video within seconds.
The main requirement for creating a realistic video using Deepfake technology is a huge amount of input to train the machine on a person’s expressions and movements. The more information the machine can learn from, the more realistic the result will be. For example, an impersonator acting as several famous Hollywood actors used Deepfake technology to map out their faces onto his face in order to create a three-minute video. The creator claimed that he used 1,200 hours of footage, 300,000 images and 250 hours of work to achieve a realistic result.
Some are attempting to develop Deepfake videos using only a single image as well. The idea is to teach the machine to study general knowledge about human heads and facial expressions for it to use as a guideline and foundation. Once the machine is familiarized with the basic information, a single photo is enough for it to generate the output. This has increased the fear of not only celebrities, but also normal people whose identity is not as ample as that of a celebrity. Though this method is still far from being realized, it demonstrates potential threats of this technology in the future.
This also shows the power of machine learning. The more training a machine has, the better it gets. The more footage and photos, the more realistic Deepfake videos become. It is understandable why celebrities and public figures can be the perfect input sources for this technology since there are plenty of pictures and videos of them available online due to the nature of their career. This will likely lead to potential uses of Deepfake in the entertainment industry’s production process.
The key element in creating realistic output from Deepfake technology is GANs, or Generative Adversarial Networks, which is the idea of using two AI systems fighting against each other to improve the quality of the result. One system functions as the forger, while the other one works as a detector. Together the two systems continuously try to overcome the other resulting in a more and more realistic result. By using this principal, Deepfake can create and self-detect their results over and over to come up with human-like effects. GANs is the key element in altering and creating images based on the previous input. This principal makes Deepfake a beneficial tool in the entertainment business.
To begin with, Deepfake can be useful at the very initial stage to improve the quality of a video, turning amateur videos into more professional-like videos. This also includes turning black and white films into color. Deepfake is usually regarded as a visualization tool, but the technology can also be used with audio. In training a machine to study the voice of an actor, it can generate the voice in case the said actor loses his vocal ability due to sickness or disease. The technology can even make the movement of lips and facial expressions match the lines the actor is saying. This is useful in dealing with usual problems like censorship or when there is a problem in sound recording during production. Furthermore, Deepfake can be used for multiple language dubbing, reducing the cost of hiring local voice actors and creating a more realistic experience for international audiences.
Another possible use of Deepfake technology is for special effects and advanced face editing in post-production. In this case, it can be used to reverse aging to create a younger version of an actor, or to edit scenes where an actor is not able to participate due to scheduling conflicts or an unfortunate tragedy. This is not new as many films have been using computer generated footage for scenes where original characters were portrayed by deceased actors. Hollywood digital effect studios like Industrial Light & Magic and Animal Logic are convinced that Deepfake technology would save time and production costs. Furthermore, with a trigger created by the DeepNude application developer, Deepfake technology might also be used for nudity scenes without the use of body doubles, though it is doubtful this would be cost effective.
The next step would be replacing actors completely with Deepfake technology using an impersonator, further cutting studio costs. This has potential to even create new actors, where producers can choose elements they like from multiple actors, such as the body movement of one, facial expressions of another, and the voice from another actor, creating an actor that is completely computer generated. This idea could fulfill any directors’ dream, while eliminating the high cost of established actors.
Figure 3: A younger version of actor Robert De Niro next to his current picture. Source: The CineRanter.
Deepfake technology could threaten jobs for current actors while making it more difficult for immerging actors to gain success, as legendary actors can now be regenerated after their death. Though the practice is still controversial, it has already happened in the franchise movie Star Wars. A more recent and controversial case is the plan to bring James Dean back to the screen as a new character in 2020. The movie is called Finding Jack, which is an adaptation of Gareth Crocker’s novel. The story takes place in wartime Vietnam and follows a group of soldiers who refuse to leave their dogs behind. Dean will play a character named Rogan,the second lead of the story. His character will be crafted using Dean’s photographs and footage from old movies. The idea has received negative feedback and caused fear within the entertainment industry. The case involves many layers of appropriations. First, the production is surely not consensual, at least not from Dean himself. (His family already gave the approval to the production.) Second, it is not about reviving the roles he portrayed, but is about casting him in a completely new role as if he is still acting. Despite the argument the producers give that Dean is the perfect choice for the role, one of the controversies is the necessity of casting him instead of using a living actor. This, among other potential uses, involved many legal and moral consequences.
Detection and Protection
In its early stage, Deepfake videos were detectable by humans. Some small abnormalities like eye-blinking or glitches in the produced videos often gave away the fact that the subjects were not real humans. However, the technology is constantly developed which makes it harder for humans to detect it without the help of the technology that created it in the first place, like Artificial Intelligent (AI). Some giant tech companies like Google have been aware of Deepfake and done research on it. Google has stated that they are committed to developing AI best practices to mitigate the potential for harm and abuse, acknowledging the potential threats and harms Deepfake might create to individuals or society. Google has been providing Deepfake videos to academic and corporate researchers to use for the development of training tools to detect Deepfake videos. Adobe is another company that has developed an AI-enabled tool that can spot any small alterations of an image. In addition to that, the company also plans to release an authentication tool on their software in 2020 to allow users to attach information to their work such as when and where the images were taken. Furthermore, a Dutch start-up company called Deeptrace is also using AI to collect data, and recently published a report raising concerns about the rapid expansion of Deepfake technology.
Figure 4: An example of Adobe’s ability to detect photo manipulation. Source: Adobe.
The Digital Millennium Copyright Act allows victims to file lawsuits against online platforms using Deepfake inappropriately so that involuntary pornography will be removed. Websites where Deepfake originally started, like Reddit or Pornhub, have banned and removed content related to Deepfake technology altogether. U.S. representatives have become concerned over the threat the technology can pose after Deepfake affected the political field in 2018. Though it was made to raise the awareness of how far fake news can go under the age of Deepfake, a video of AI altered Barack Obama insulting Donald Trump raised fears over how the technology could be used as a political tool to disseminate false information. The idea of having legal rights over one’s likeness, even past death, has caused many to argue for protection over any involuntary digital replica of an individual’s image.
In the entertainment industry, as in other industries, the law is still trying to keep up with the technology. Defamation Law is one of the first laws dealing with Deepfake videos, especially ones without consent. It is the idea that Deepfake is related to fake information that might harm someone’s reputation. However, the interpretation of the law might not fully cover the consequences of the use of Deepfake. The law is interpreted so that if a person’s liking is used for a Deepfake video and that video is perceived to be a computer-generated video by the public, no matter how realistic the video might seem, the law does not cover it. Another aspect of this law is that the victim of Deepfake needs to prove there was malice in the false information. As for the case of film making, if a producer or a film studio decides to use the technology after acquiring copyright footage and images as an input, and publicly announce that they are creating the generated version of a certain actor, then the actor might not be able to file a defamation lawsuit against the company.
Figure 5: An example of a falsely reported article using Clint Eastwood’s image. Source: The National Enquirer.
Despite the lack of coverage in the Defamation law, Publicity law can be used for this case. Though only a handful of states—California being one of them—currently recognize publicity both by statute and Common Law, the idea seems to be a good foundation for further recognition in other states and countries. According to the California law, by the statutory right, ‘any person who knowingly uses another's name, voice, signature, photograph [either still pictures or video] or likeness, in any manner on or in products, merchandise, or goods, or for purposes of advertising or selling … goods or services, without such person's prior consent … shall be liable for any damages sustained by the person injured as a result thereof.’ This means a lawsuit can be filed when a person’s likeness is used without their consent. Similar conditions go with the Common Law, as seen in the case of Clint Eastwood suing the tabloid magazine National Enquirer. The actor claimed that the tabloid magazine used his photo in a falsely reported article about him being involved in a love triangle with other celebrities. The court states that a plaintiff can take an action when his/her identity, name and likeness is used without consent to create benefit for the creator and result in injury. These two frameworks provide opportunities for actors to fight back against the use of Deepfake. California’s law also allows transferable publicity where heirs can determine further use of their ancestor’s publicity, including images, name and likeness.
New York has also discussed modifying their Publicity Law to make the publicity transferable as well. In June 2018, a bill passed in the state assembly declaring it fraud to create a digital replica of someone without their consent, subject to damages and/or an injunction. However, this raises concerns because once publicity is considered a transferable property, it can be transferred in the case of bankruptcy and divorce. Another conflict raised by the opposing parties is the First Amendment right of the U.S., where freedom of speech and expression are protected. Disney and NBCUniversal were opposed to this bill, claiming that creativity and storytelling could be compromised. This could be especially true in sequels of classic movies where actors in the prequel produced decades ago have since passed away. By prohibiting the use of Deepfake technology, the studios might need to replace the original cast or kill off characters instead.
Figure 6: Robin Williams as the Genie in Aladdin. The use of Williams’ voice and image is prohibited until 2039. Source: Filmdumpster.
The case of James Dean mentioned earlier, brings up ethical problems rather than legal ones. He died as a California’s resident so if the movie director obtains permission from his heir, recreating Dean in a movie is completely legal. The simplest way an actor can protect the usage of their likeness after their death is to specify their wishes in a will. As previously discussed, the main input for Deepfake technology is the ample amount of images and footage of celebrities. That is not all producers need, however, as they also require consent from the individual or their estate. Under copyright provisions in multiple jurisdictions, the copyright owner holds rights over the reproduction and distribution of a copyrighted work; they also have ‘moral rights’, or a right to the integrity of the work. The holders of copyright for photographs used to face swap, and the video into which the face is swapped, may be able to seek damages and to have copies of the fake video destroyed (Black, et al., 2018). There are increasing numbers of celebrities seeking post-mortem wills. In a state like California, such a will can prohibit the use of a celebrity’s images for any commercial use in any form.
An example of this preventive method is the case of the late actor Robin Williams. He simply passed his rights to a foundation and prohibits any use of his voice, likeness or even his signature for any commercial purposes until 2039—25 years after his death. This kind of action will likely limit Deepfake for regenerating actors of our age. It will not stop producers from attempting to regenerate actors who passed away long before this idea was born, such as actors like James Dean, Grace Kelly, Marilyn Monroe and Audrey Hepburn. In dealing with new law, precision is the key element. If the law prohibits Deepfake due to its likeness, this might also prohibit make-up and special effect artists’ ability to replicate certain actors in their biographic films as well.
Conclusion
Despite the many concerns Deepfake has brought in the past couple of years, the technology still has great potential in the entertainment industry. The problem is how to fully benefit from the technology without violating anyone’s intellectual property and privacy. Regarding the laws and tools we have nowadays, it is undeniably true that we are still a step behind the technology. However, when it comes to technology, it is either now or never. As said by the anonymous developer of the notorious DeepNude application “The technology is ready (within everyone’s reach). So if someone has bad intentions, having DeepNude doesn’t change much. ... If I don’t do it, someone else will do it in a year.” (Anonymous, 2019)
The idea of video manipulation started back in 2016. The idea of GANs even dates back further. Whether society is ready or not, the technology already exists. The only thing we can do is maximize the use of it while being careful not to abuse others at the same time. Therefore developing a legal framework is crucial at this time. Before doing that, it is important to understand the different impacts Deepfake can have in various industries. In the field of politics, Deepfake is considered a threat to reputation and can be used as a political tool to spread convincing fake news. However, in the entertainment industry, Deepfake can create both positive and negative impacts, and aid in the creative process. Lawmakers need to consider all these factors to realize this is not a zero-sum game, and a solution can exist that will benefit all parties.
Resources
Ajder, Henry, Giorgio Patrini, Francesco Cavalli, Laurence Cullen. “The State of Deepfakes: Landscape, Threats, and Impact”. Deeptrace Lab. September 2019.
Alt, Eric. “The Controversial Tech Driving James Dean's Return to the Big Screen.” Popular Science. November 12, 2019. https://www.popsci.com/story/technology/digital-actors-james-dean-resurrection-hollywood/.
Black, Ryan, Pablo Tseng, and Sally Wong. “What Can The Law Do About Deepfake”. McMillan Litigation and Intellectual Property Bulletin. March 2018. https://www.mcmillan.ca/What-Can-The-Law-Do-About-Deepfake.
Burkell, Jacquelyn, and Chandell Gosse. “Nothing New Here: Emphasizing the Social and Cultural Context of Deepfakes.” First Monday. December 2, 2019. https://firstmonday.org/ojs/index.php/fm/article/view/10287/8297.
Chawla, Ronit. "Deepfakes: How a pervert shook the world." International Journal of Advance Research and Development 4, no. 6 (2019): 4-8.
Farish, Kelsey. "Do deepfakes pose a golden opportunity? Considering whether English law should adopt California's publicity right in the age of the deepfake." Journal of Intellectual Property Law & Practice (2019).
Fisher, Christine. “These Deepfake Celebrity Impressions Are Equally Amazing and Alarming.” Engadget. October 11, 2019. https://www.engadget.com/2019/10/11/deepfake-celebrity-impresonations/
Gardner, Eriq. “Deepfakes Pose Increasing Legal and Ethical Issues for Hollywood.” The Hollywood Reporter. November 23, 2019. https://www.hollywoodreporter.com/thr-esq/deepfakes-pose-increasing-legal-ethical-issues-hollywood-1222978.
Hall, Holly Kathleen. "Deepfake Videos: When Seeing Isn't Believing." Cath. UJL & Tech 27 (2018): 51.
Jahner, Kyle. “Dead Actors' Fortunes May Hinge on Where They Die in a CGI World.” Bloomberg BNA News. Accessed December 11, 2019. https://news.bloomberglaw.com/ip-law/dead-actors-fortunes-may-hinge-on-where-they-die-in-a-cgi-world.
Javed, Farah. “Actors Debate over James Dean Posthumously Cast Using CGI.” The Ticker. November 18, 2019. https://theticker.org/ticker/2019/11/18/actors-debate-over-james-dean-posthumously-cast-using-cgi.
Kemp, Luke. “In the Age of Deepfakes, Could Virtual Actors Put Humans out of Business?” The Guardian. July 8, 2019. https://www.theguardian.com/film/2019/jul/03/in-the-age-of-deepfakes-could-virtual-actors-put-humans-out-of-business.
Marr, Bernard. “Artificial Intelligence Explained: What Are Generative Adversarial Networks (GANs)?” Bernard Marr. Accessed December 11, 2019. https://bernardmarr.com/default.asp?contentID=1901.
Marr, Bernard. “The Best (And Scariest) Examples Of AI-Enabled Deepfakes.” Forbes Magazine. July 22, 2019. https://www.forbes.com/sites/bernardmarr/2019/07/22/the-best-and-scariest-examples-of-ai-enabled-deepfakes/#5a6398512eaf.
Morris, Betsy. “Tech Companies Step Up Fight Against 'Deepfakes'.” The Wall Street Journal. Dow Jones & Company. November 22, 2019. https://www.wsj.com/articles/tech-companies-step-up-fight-against-deepfakes-11574427345.
Murray, John. “Why The Danger of Deepfakes Is No Danger At All.” Medium. Primalbase. July 15, 2019. https://medium.com/primalbase/why-the-danger-of-deepfakes-is-no-danger-at-all-82c21366e6c6.
Pomeroy, Robin. “This Iconic Filmstar Will Star in a New Movie - from beyond the Grave.” World Economic Forum. Accessed December 11, 2019. https://www.weforum.org/agenda/2019/11/james-dean-cgi-deepfakes/.
Richwine, Lisa, and Jill Serjeant. “Actors Seek Posthumous Protections after Big-Screen Resurrections.” Reuters. December 19, 2017. https://www.reuters.com/article/us-film-resurrections-analysis/actors-seek-posthumous-protections-after-big-screen-resurrections-idUSKBN14J1TU.
Robinson, Joanna. “Robin Williams Posthumously Blocks a New Aladdin Sequel Featuring His Voice.” Vanity Fair. November 9, 2015. https://www.vanityfair.com/hollywood/2015/11/robin-williams-aladdin-sequel-will.
Romano, Aja. “Reddit Finally Bans Its Forum for Creepy Fake Celebrity Porn.” Vox. February 8, 2018. https://www.vox.com/culture/2018/2/8/16987098/reddit-bans-deepfakes-celebrity-face-swapping-porn.
Romano, Aja. “Why Reddit's Face-Swapping Celebrity Porn Craze Is a Harbinger of Dystopia.” Vox. February 7, 2018. https://www.vox.com/2018/1/31/16932264/reddit-celebrity-porn-face-swapping-dystopia
Samuel, Sigal. “A Guy Made a Deepfake App to Turn Photos of Women into Nudes. It Didn't Go Well.” Vox, June 27, 2019. https://www.vox.com/2019/6/27/18761639/ai-deepfake-deepnude-app-nude-women-porn.
Shanley, Patrick, and Katie Kilkenny. “Deepfake Tech Eyed by Hollywood VFX Studios.” The Hollywood Reporter. May 4, 2018. https://www.hollywoodreporter.com/news/deepfake-tech-eyed-by-hollywood-vfx-studios-1087075.
Singer, David, and Camila Connolly. “How Hollywood Can (and Can't) Fight Back Against Deepfake Videos (Guest Column).” The Hollywood Reporter. December 11, 2019. https://www.hollywoodreporter.com/thr-esq/how-hollywood-can-can-t-fight-back-deepfake-videos-guest-column-1237685.
Thies, Justus, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. "Face2face: Real-time face capture and reenactment of rgb videos." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387-2395. 2016.
Trost, Brette. “Review: New York Right of Publicity Law: Reimagining Privacy and the First Amendment in the Digital Age.” JIPEL Blog. Accessed December 11, 2019. https://blog.jipel.law.nyu.edu/2018/04/review-new-york-right-of-publicity-law-reimagining-privacy-and-the-first-amendment-in-the-digital-age/.
Vincent, James. “New AI Deepfake App Creates Nude Images of Women in Seconds.” The Verge. June 27, 2019. https://www.theverge.com/2019/6/27/18760896/deepfake-nude-ai-app- women-deepnude-non-consensual-pornography.
Vyas, Kashyap. “Generative Adversarial Networks: The Tech Behind DeepFake and FaceApp.” Interesting Engineering. August 20, 2019. https://interestingengineering.com/generative-adversarial-networks-the-tech-behind-deepfake-and-faceapp.
Westerlund, Mika. “The Emergence of Deepfake Technology: A Review.” Technology Innovation Management Review. November 2019. https://timreview.ca/article/1282.