Written by Katelyn Clement

What is augmented reality?

Augmented reality (AR) places digital objects and sounds into the physical world. Computer vision tracks what is in the real world through a camera feed, and digital content is then displayed realistically through a screen. This differs from virtual reality, which transports users into a digital world. AR is used in a variety of contexts, such as public safety, gaming, education, tourism, marketing, and equipment maintenance. In the past few years, both Pokémon GO and Snapchat brought AR into the limelight, but there are many more applications of the technology.

In the arts realm, AR has been used in both the classical music and popular music realms to cultivate audience engagement with visual components to the listening experience. This article provides an overview of AR as a technology and its usage in the music industry.

History of augmented reality in everyday life

• 1968: Sword of Damocles

  The first head-mounted display was called the "Sword of Damocles.” With it, users could see computer-generated graphics through a pair of googles.

• 1974: Videoplace

  The University of Connecticut and Myron Krueger built an AR lab called "Videoplace" that projected silhouettes that surrounded and interacted with each other. Two people in separate rooms each with a projection and screen and video camera were able to communicate through their projected images.

• 1990: Boeing

  Boeing researchers Tom Caudell and David Mizell coined the term "augmented reality.”

• 1994: Dancing in Cyberspace

  Julie Martin brought AR to the entertainment industry with a production called "Dancing in Cyberspace” where acrobats danced alongside virtual objects.

• 1998: National Football League

  The NFL added a yellow line marker to broadcasted games so that fans could see where and how fast players moved across the field. It is still used today, although the technology has upgraded. Most people are so used to it that they don't realize it's augmented reality.

• 2009: Esquire Magazine

  Esquire Magazine created the first AR magazine cover where users could scan the cover using their computer scanners and watch a feature by Robert Downy Jr. who "popped out" of the cover.

• 2013: Volkswagen

  Volkswagen released the MARTA app, which overlaid car repair instructions for technicians.

• 2017: Ikea

  Ikea released "IKEA Place,” which allowed consumers to preview IKEA decor in their own homes before purchasing items.

How does augmented reality work?

Augmented reality seemingly places virtual elements into the real world through a variety of elements, including computer vision and registration tools. A depth-sensing camera records visual information to add to an existing object or place. For example, Ikea first records an image of a couch before placing it into a consumer's room. Computer vision can take one image in the user's room and reference it with similar-looking objects so that the objects appear to be three-dimensional. Registration tools, such as accelerometers and motion sensors, help the computer define the space to superimpose the objects that should be around the AR user. Users then view the augmented reality content, such as a video or image, on a phone, tablet, goggles, or computer monitor.

Types of augmented reality

MARKER-BASED AR

Marker-based augmented reality needs a distinct marker or image, such as a QR code or custom artwork, in order for the computer to generate information. This type of AR is inexpensive and easy to implement. It is commonly used in album covers, but users need to download specific apps to recognize the image.

MARKERLESS AR

Markerless augmented reality does not require anything specific in the real world. The camera can identify any physical space and augment it with an image. An example would be trying on virtual glasses.

LOCATION-BASED AR

Location-based augmented reality takes data from GPS and places objects in the real world based on users’ locations. A great example of this is Pokémon GO.

PROJECTOR-BASED AR

Projector-based augmented reality uses advanced projection technology to project onto specific objects. It is most commonly used to provide manufacturing instructions.

AR in the music industry

In 2016, streaming apps such as Spotify and Apple Music transformed the music industry and brought millions of songs to users’ fingertips in a legal and convenient way. Even though the music itself is much more accessible, artists struggle to personalize the listening experience. Streaming has taken away the need for fans to buy physical albums. While people used to connect with artists by buying CDs and records, they are now missing that step between streaming a song and attending a live performance. Artists are beginning to use augmented reality to increase audience engagement and make listening experiences more personal. Augmented reality can provide the bridge between the physical and digital world.

Album artwork

Streaming has taken away the need for fans to buy physical albums, so artists need to find new ways to promote sales and engage with their fans. The band Haim’s album "Women in Music Pt. III" created a sonic augmented reality experience when fans pointed their Instagram camera at the album cover. The AR experience altered both the album artwork and music at the same time. By using Instagram, Haim eliminated the need to download an external app and place more work on the listener.

Music videos

The Lego Vidiyo app and products allow children to explore legos and augmented reality by creating, directing, and sharing their own music videos. They can buy Lego Vidiyo products, build figurines and buildings, and then animate their creations via the Vidiyo app. 

Startup company Wavy helps musicians enhance their music videos with augmented reality. Its goal is to enhance music listening experiences by adding three-dimensional animation and storytelling.

Music marketing

Artists can use augmented reality to make their posters more interactive.

In October 2020, Sam Smith collaborated with Spotify and created a 3D augmented reality music video to their single "Diamonds," viewable when users scanned the album art with another device. 

In 2017, Michael Jackson's album "Scream" came with a bundle that featured a scannable poster. 

Live shows

Adding augmented reality to live performances can personalize audience experiences as well. 

The 2019 Coachella music festival transformed one of its stages into an AR experience. People could see completely new visuals by pointing their phones at the screen. 

AR in classical music

Using augmented reality in classical music can help attract new audiences and keep retaining audiences more engaged. It can be used in a variety of arenas, including education, composition, acoustics, and performances.

Acoustics

Augmented reality concert halls have microphones that can adapt to the medium being performed. Stage managers can dial in acoustics for a solo performance or large orchestra, and the room will adapt to the ensemble. These halls are typically dry-sounding without the microphones on, but with the microphones, they can use acoustic samples from halls from around the world. "Living Pianos" brought the concept to light and discussed a similar feature in the Indianapolis Hilbert Circle Theater.

Opera

Opera companies can use augmented reality to help audience members connect with storylines and characters. The Welsh National Opera developed an AR experience, "A Vixen's Tale," which combines a physical tunnel book stage with an augmented reality virtual fox that takes audience members down the tale of "The Cunning Little Vixen.”

The English National Opera suggests that augmented reality apps can be used to help opera singers and theatrical makeup artists using virtual makeup try-on apps. The apps can help vocalists transition into their roles and help makeup artists develop a look for each character. Audience members could also test out the apps to see how they would look if they were also opera characters. 

Notation

In 2013, Google Glass wanted to replace music stands and sheet music with scrolling notes through the glasses. They discussed changing traditional music notation to one with colors and shapes instead of a staff, which could be extremely problematic for musicians. The app would also need to adjust to the conductor. They decided that this technology may take a long time to develop. 

In 2019, the National Theater in London introduced "smart caption glasses" that displayed dialogue as the actors spoke. The glasses allowed audience members to read the captions and watch the stage at the same time instead of switching between reading captions above or below the stage or on a caption reader and the stage. I could see this technology adapting to operas in the near future.

Education

Augmented Reality can be used to teach a variety of instruments. At the moment, most research has been done using piano and stringed instruments. 

A Pittsburgh-based start-up Music Everywhere is currently developing a program in which students can learn to play the piano by putting on a Microsoft HoloLens. The app then overlays icons onto piano keys that tells the students where to place their fingers and for how long to hold the notes. 

An app called Instant Musician allows users to point a phone camera onto a piano. Virtual moving blocks, such as the ones used in Guitar Hero, Rock Band, and Dance Dance Revolution, show up on the phone's screen and tell users which notes to play and when. 

Performances

Augmented reality can put classical music concerts in audience members’ living rooms. Spacial Orchestra lets users walk through their living room and hear and see how an orchestra would be placed in real life. ​

The Piano Guys partnered with Massive Technologies to bring Jon Schmidt to users’ home pianos or virtual ones. Users can buy an app, AR Pianist, and pay $2.99 per piece or $59.99 annually to put The Piano Guys in any location. ​

Strings Pavilion developed an app in which users can purchase a pass to view 360-degree concerts on their mobile devices in a variety of venues.