AR for production and installation of exhibitions, stage design, and public art

Since its first iteration, Augmented Reality (AR) has been disrupting education, health, entertainment, and many other fields. By enhancing the senses and abilities it has delighted but also aided in solving difficult problems. In arts and culture, AR has transformed static museum displays and provided special effects for stage productions. But the use of AR hasn’t gone much beyond support in storytelling, and the adoption of such a versatile technology as a tool in the production processes has been tentatively explored. In assessing the potential of AR as a tool, art institutions can take a cue from the field of architecture and construction where AR has been successfully used on site for many years. The question is if adjusting the technology used to the scale and needs of a particular area of arts and culture, such as the production of visual arts exhibitions, stage design, and creation and installation of public art, would it truly disrupt the field?

Understanding AR

Before delving into the evaluation proposed, a short introduction into AR is needed. In 1997, Ronald Azuma, a researcher at Hughes Research Laboratories, wrote one of the first extensive papers on AR. At the time, AR was an emerging technology which had advanced rapidly in a short time, as more researchers in technology centers and academia understood the potential it could have in the medical field or manufacturing as annotation and visualization of procedures and parts, or in entertainment, robot path-planning and military aircraft. In his paper “A Survey of Augmented Reality” he presented AR as a technology that “allows the user to see the real world, with virtual objects superimposed upon or composited with the real world” (Azuma, 1997). Azuma used as an example the experiment made by ERCR in 1994 were two 3D virtual chairs and a lamp were superimposed on a room containing a table and a telephone. The revolutionary aspect was the way the virtual objects blended in with the real objects, so the table covered the chairs, and the lamp covered the table. The main three characteristics of AR were its ability to combine the real space and virtual objects, to be “interactive in real time,” and “registered in 3-D” (Azuma, 1997). Since then there have been many changes, but these characteristics remained.

20 years later, AR has developed beyond the superimposed 3D virtual objects to a much wider understanding of what augmented reality can be, whether it’s using audio, haptics, and even smell. In this second wave, Helen Papagiannis, author and AR expert, described AR as technology were “the focus will be on delivering a meaningful and compelling experience that enhances your reality” (Papagiannis, “Augmented Human”), not just certain senses. There are two examples that can illustrate the future of AR: to communicate and as an extension of ourselves. The first example is Touch, developed by Smartstones, a device shaped like a stone which uses vibrations and LED combinations to communicate brief messages between devices. This technology enhances the ability to communicated for those with medical problems such as seniors with a neurological illness or children with autism. The second is more radical, AR is used with AI to create avatars or intelligent agents that can extend a person’s existence even after demise, as in the case of the “ultimate selfie”. Eterni.me is one of the companies that offers this technology. It creates an avatar using knowledge taught to an AI machine while the person is still alive. After their demise, interactions with the avatar can be helpful in dealing with grief and keeping one’s memory alive. These are just two of the many projects developed around AR which push the boundaries of our understanding and use of augmented reality in solving real problems.

How AR works

In every industry in which it is used--be it health, education, business, entertainment, even public service or military--AR requires the same core components for hardware: “a computer, a monitor or display screen, a camera, a tracking and sensing system (GPS, compass, accelerometer), a network infrastructure, and a marker” (Rampolla, Kipper, ”Augmented Reality”. For software, it uses “an app or program running locally, a Web service, and a content server” (Rampolla, Kipper).

The process behind the most basic AR experience, that of the 3D object superimposed upon a real environment, starts with a live video from a camera, showing for example a room. The image is digitized, and a marker is identified through boarder detection and binary encoded patterns. Markers could be anything from floors, or other objects and can be identified using the pattern, outline, location, or surface of real objects. Based on the marker, the program/app positions and orients the 3D object, and the symbol inside the marker is matched to the assigned digital content. The last part is rendering the virtual object into the video stream, making it viewable on the display screen, a smartphone, a tablet, or a wearable display.

Two important aspects regarding AR are the display and interface. The displays can be placed into 3 categories according to Rampolla and Kipper: mobile handheld displays, video spatial displays and Spatial Augmented Reality (SAR), and wearable displays. The most accessible are the mobile handheld displays, which are the everyday smartphone and tablet. Video spatial displays use a web camera to identify the marker and render it on a separate screen.  SAR takes this a step further, the technology doesn’t require a separate screen, it displays the rendered video on other surfaces through video projectors, holograms or other similar technologies, making the experience more integrated into the surrounding environment. The last category--the wearable displays-- is becoming more available as technology evolves, and industries are more open to using AR/VR thus increasing demand of this type of equipment. The wearable displays can be binocular augmented displays or monocular augmented displays depending on the complexity of the information and the context in which it is used. Binocular augmented displays like the Epson Moverio BT-300, one of the earliest such displays to the Microsoft HoloLense, Osterhout Design Group R-7 Smartglasses, DAQRI Smart Helmet, or AtheerAiR (Augmented interactive Reality) Glasses, are the major type of displays that are entering the market, mostly for business, industrial use and even military and defense. The monocular augmented displays are used in other areas like health, because of limited need of scene analysis with less complex information shared. Wearables like Vuzix M300 Smart Glasses and Google Glasses are used mostly to display text, images or icons.

The second important aspect is the interface in AR, which is shaped by the type of technology used in addition to AR. The tangible AR interface uses haptics to make the experience more real, while hybrid surfaces use more than one display with the goal of serving as a flexible platform. The same goal applies to multimodal AR interfaces that combine different methods to interact with the system, from speech to gestures and even gaze. The last is the collaborative AR interface which could be the most useful in large scale projects or global collaborations, as it supports remote sharing and interaction of multiple users, over different applications and platforms.

A lot of the elements that create an AR experience, the different displays and ways to interact with the user and environment, could be implemented in the production of exhibitions, stage design or installation of public art. In addition to these elements there are certain specific aspects in the use of AR in architecture and construction that could also be considered as these two fields have commonalities with exhibition and stage design in terms of working with similar materials and techniques, having limitations due to site specificities which require prior 3D visualization, and being the result of a coordinated collaborative effort .

AR in construction and architecture

In construction and architecture AR is used mostly as 3D objects superimposed onto real spaces using specific markers provided in the architectural plans. It uses a collaborative interface with handheld or wearable displays. In this field there are two specific developments that can be applied to production in arts and culture, and these are the use of Building Information Modeling (BIM) with AR and Fologram.

Building Information Modeling (BIM)

BIM is an intelligent 3D model that gathers all the documents needed for a construction or architecture project and facilitates, for all involved in the project, the “management, coordination and simulation during the entire lifecycle of a project (plan, design, build, operation and maintenance)” (Autodesk). The model allows for the analysis of the whole project to be made as all the elements (architectural plans, MEP – mechanical, electrical, and plumbing, and other design plans) can be visualized in the same 3D rendering, before it is built. The visualization helps to see whether there are issues in the overlaying of the different plants, and tests solutions in the virtual model, at no cost. Also, the model can generate all the design documentation needed, as well as serve for future maintenance work. The process in BIM starts with the Plan phase, when reality capture and data help build a model of the actual environment. In the Design phase, the actual design, analysis and documentation are made. The information is also used to create scheduling and logistics for the project which will be shared with contractors and trades in the Build phase, when fabrication begins. The last phase is Operate, where the BIM is used for maintenance, but also future work or deconstruction. The idea and technology behind the 3D model aren’t new, it has been developing for the last 30 years, being influenced most recently by Human Computer Interaction, Generative Design, Cloud Computing, and even AR.

The use of BIM and AR together is becoming more frequent, noticeably by the many apps and software developed in the past years. Apps like Dalux Field, AR Mapper, Gamma AR show there is a demand, as well as enough familiarity with the technology that allows for more developers to combine the two technologies. The prices for the apps, handheld displays, or wearables required varies. Some apps are offered on a monthly basis starting from 49 euros or a one-time fee for a number of months, like 150 euros for 3 months. Each app or software can be used with either handheld devices or wearables which can add to the final cost. Two wearables used on constructions sites that are certified as protection glasses are Microsoft HoloLense and DAQRI Smart Helmet. However, the prices for these are still high, with HoloLense 2 selling at $3,500 and Smart Helmet at $15,000, which can add considerably to the costs of an AR-BIM system.

Fologram

Another development that uses AR in architecture and construction as well as design is the software platform Fologram. It “allows Mixed Reality hardware to talk to desktop design software” (Fologram,), so users can design the geometry and plans in their design software of choice and stream it to their wearable or handheld display, thus allowing them to make changes in real time. The advantage of this platform is that by designing through AR, a connection can be made between the object designed and the spatial information. Furthermore, Fologram allows sharing the interactive design with other users on different platforms and devices. Apart from being able to create with AR 3D designs, either as a single user or a collaborative project, Fologram has also been used on construction sites to instruct workers on how to assemble or built certain parts of the project. This particular use provides more precision and efficiency in the work done, requiring less time and resources to complete the project.

Can AR improve production for arts institutions?

There are practical uses of AR in the field of construction and architecture, but can these be as successfully utilized in the production of exhibitions, stage design, and creation and installation of public art? There are some similarities between the two fields in terms of materials used, the fact that both are limited by site specifications and are the result of a collaborative process. But can AR be applied in the same way to arts and culture as it had in construction and architecture?

There is some reluctancy from arts and culture institutions to implement these technologies, as there are very few cases where AR has been used for this particular activity. But, a general aversion towards technology isn’t a factor as 3D visualization have been used before installing exhibitions, and the idea of a virtual museum or exhibition explored in VR has already been coopted in many well-knows cultural institutions. Furthermore, it can’t be a particular dislike of AR as museums have already adopted this technology to make their displays interactive, such as the Cleveland Museum of Art, and theaters have used this for special effects, like in the 2018 version of “Gulliver” at the Gesher Theatre, or as an aid, as The National Theatre in London used  smart glasses to display captions for people with hearing loss. In terms of the use of AR in production activities there are very few examples. These cases reveal that there are still technical limitations in using AR in production as well as high costs compared to traditional ways of designing, producing, and installing exhibition and stage productions. A very publicized case is that of Cirque du Soleil, in 2017, the Cirque du Soleil design team demonstrated the creation of the set and choreography for their next production, using AR and Microsoft HoloLense. The demo, made for the Microsoft Build 2017 developer’s conference, showed how this technology can aid by allowing them to see and interact with the real size set, make changes virtually if needed, and work remotely on this collaborative project. However, the demo was partially set-up and not a fully real demonstration of the technology in real time due to the technical limitations of the Microsoft HoloLense as it couldn’t cover the whole area of the stage. Adding the high costs of implementing this technology, which include wearable displays, the monthly fees for apps, and an extra person specialized in building the 3D model, it becomes clearer why arts and culture institutions are so hesitant to use AR for the production and installation of art exhibition, stage design, and public art.

Still, the technology behind AR is evolving, and there are some elements, from the tools presented earlier, that can be used either as they are or modified to meet the different needs of art museums and theaters. For example, a simplified version of BIM could be used for museums or theaters, as the floor plan could be done only once in the beginning, with new content created and fitted to the plan for each performance or exhibition. Furthermore, it could be used for project management to coordinate different teams on how and where to install certain artworks, or elements of the set. Both BIM and Fologram facilitate remote participation in collaborative projects, which is a feature that can be used for travelling shows, or off-site coordination of an exhibition installation, if the curator can’t be present. Many of these features would benefit larger theaters and museums, as the complexity of the sets and exhibitions can be visualized and then corrected, and greater coordination can be provided during its production and installation. If the display screen is a handheld device instead of a wearable display, that would considerably lower the costs so the use of Fologram or an AR-BIM system would be feasible financially. A relevant example is the AR Stagecraft, an app still in development at Husson University, that seems to offer a simple solution to AR aided stage design, using CAD to design and visualize the stage design and handheld displays such as tablets or phones to see the virtual design superimposed on the real stage.

For smaller spaces and public art, utilizing Fologram, or a similar program, to install exhibitions or public artworks would also be solution. Instead of creating and using the map of the whole environment in an AR-BIM system, which is a rather complex and time-consuming process, another solution would be to create virtual 3D versions of the artworks that can be placed in a real space. Using handheld displays with the 3D object, a curator can easily move around an artwork in a small space in order to find the right placement for the artwork, a position that allows most visibility for different types of viewers and that doesn’t obstruct access to other elements that exist in the real space, such as doors or desks.  Utilizing Fologram and handheld devices could be a solution that doesn’t require highly trained staff and expensive technology and can reduce time and labor costs for small exhibitions and installation of public work.

Considering the success of AR in architecture and construction and the similarities in terms of logistical needs to the installing of art exhibitions or designing stage sets, it is evident that AR has the potential to disrupt the production processes in arts and culture. It also has the potential to better the traditional processes for producing art exhibitions and theatre sets as it can reduce production time, labor costs, and material waste.

Conclusion

Things are changing rapidly for these technologies and the initial disruption is fading as they blend more and more into our daily lives. Arts and culture institutions, unlike the creators and artists they show, have always been reluctant to change, especially when it involves technology. In the next 5 to 10 years, probably there will be more technological developments for the display devices which would allow platforms like Fologram to be used more frequently and at lower costs. Easier access could also lead to cultural changes and make art institutions more open to the use of AR not just for display but also in the production process. Looking back, the printing press and the camera faced the same reluctancy in the beginning, adopted faster by artists as mediums than by museums as storytelling tools or for practical uses. However, AR is far more versatile than the printing press or the camera, and hopefully its transition into a regular tool for exhibition production and stage design will happen a lot faster.

Resources

ARShow, “ARShow – Staging Dreams”, Accessed February 3, 2020, http://www.arshowpro.com/

Aukstakalnis, Steve , “Practical Augmented Reality: A Guide to the Technologies, Applications, and Human Factors for AR and VR”, Addison-Wesley Professional, 2016

Autodesk “Benefits of BIM” Accessed December 10, 2019. https://www.autodesk.com/solutions/bim/benefits-of-bim

Autodesk “What is BIM?”. Accessed December 10, 2019. https://www.autodesk.com/solutions/bim

Azuma, Ronald, “A Survey of Augmented Reality”, Presence: Teleoperators and Virtual Environments 6, 4 (August 1997): 355-385. https://www.cs.unc.edu/~azuma/ARpresence.pdf

Dalux, “Dalux BIM Viewer”, Accessed December 10, 2019. https://www.dalux.com/daluxbimviewer/Whitaker, Ross; Crampton, Chris; Breen, David; Tuceryan, Mihran and Rose, Eric, “Object Calibration for Augmented Reality”, Technical report ECRC-95-04, 1995, https://www.cs.iupui.edu/~tuceryan/tech-reports/ECRC-95-04.pdf

Fologram, “How does Fologram work”, Accessed December 10, 2019. https://learn.fologram.com/hc/en-us/articles/360008563434-How-does-Fologram-work-

Fologram, “What is Fologram”, Accessed December 10, 2019. https://learn.fologram.com/hc/en-us/articles/360008722913-What-is-Fologram-

Futurism Creative, “The Cleveland Museum of Art Jumps Into Augmented Reality”, Video 0:58. Accessed December 10, 2019 https://futurism.com/videos/the-cleveland-museum-of-art-jumps-into-augmented-reality

Gesher Theatre, “Gulliver”, Accessed February 3, 2020http://www.gesher-theatre.co.il/en/PRODUCTIONS/SHOWS/Gulliver

Ikono Space, “Build, Design and Collaborate Directly in 3D”, Accessed December 10, 2019.  https://www.ikonospace.com/

Immotef, “AR mapper”, Accessed December 10, 2019. https://www.immotef.com/ar-mapper

National Theatre, “Caption glasses”, Accessed February 3, 2020,https://www.nationaltheatre.org.uk/your-visit/access/caption-glasses

O’Kane, Shaun, “Cirque du Soleil will use HoloLens to design sets and plan shows”, The Verge, May 11, 2017. Accessed December 10, 2019. https://www.theverge.com/2017/5/11/15625724/cirque-du-soleil-microsoft-hololens-build-2017

Papagiannis, Hellen “Augmented Human”, O’Reilley Media Inc, 2017

Portal Gamma AR, “The Solution for BIM and Augmented Reality on your Construction Site”, Accessed December 10, 2019. https://portal.gamma-ar.com/

Quirk, Vanessa, “A Brief History of BIM”, Archdaily, December 7, 2012. Accessed December 10, 2019. https://www.archdaily.com/302490/a-brief-history-of-bim

Rampolla, Joseph and Kipper, Greg, “Augmented Reality”, Syngress, 2012

Shavel, Tatyana, “When augmented virtual reality hits theather” Datafloq, February 11, 2019, Accessed February 3, 2020. https://datafloq.com/read/when-augmented-virtual-reality-hit-theater/6046

Smart Reality, “Smart Reality”, Accessed December 10, 2019. https://smartreality.co/

Souza, Eduardo, “9 Augmented Reality Technologies for Architecture and Construction”, April 14, 2019. ArchDaily. Accessed December 10, 2019, https://www.archdaily.com/914501/9-augmented-reality-technologies-for-architecture-and-construction

Stein, Scott, “Microsoft's latest HoloLens partner: Cirque du Soleil”,Cnet, May 11, 2017. Accessed December 10, 2019. https://www.cnet.com/news/microsofts-latest-hololens-partner-cirque-du-soleil/

The Creators Project, “The Royal Shakespeare Company Uses Real Time Effects to Create a New Version of ‘The Tempest’”, Vice, December 14, 2016. Accessed December 10, 2019 https://www.vice.com/en_us/article/bmy9ev/royal-shakespeare-company-uses-real-time-effects-to-create-the-tempest

Williams, Brave, “Husson to demonstrate new Augmented Reality app for theatrical set designers”, Bangor Daily News, April 30, 2019. Accessed December 10, 2019. https://bangordailynews.com/bdn-maine/event/2019/04/30/husson-university-to-demonstrate-new-augmented-reality-app-for-theatrical-set-designers/

Zara, Christopher, “Meet the AR start-up that wants to give live theatre an augmented make-over” Financial Times,  July 26, 2018, Accessed February 3, 2020. https://www.fastcompany.com/40587203/meet-the-ar-startup-that-wants-to-give-live-theater-an-augmented-makeover