Written by Gabrielle Di Raddo
Source: Pittsburgh New Music Ensemble
Binaural Audio for the live arts audience
Emerging from COVID-19 lockdowns, being able to attend performing arts performances in person has felt like a sensory treat. Feeling the energy of the collective audience, the soft whispering and rustling in seats, the sounds of footsteps as performers take the stage, the low hum of stage lights.
Despite numerous advances in the digital landscape over the last two and a half years, experiencing performance in person heightens the senses in a way that its virtual counterpart cannot.
Enter the Neumann KU 100 microphone, or “Michel” as Kevin Noe, Artistic Director of Pittsburgh New Music Ensemble (PNME), affectionately calls the humanoid microphone that sits at the center of Ensemble’s 2022 season finale show, literally and figuratively. The microphone is a feat in binaural audio, engineered to replicate how sound travels through the human ear. When heard through headphones, it’s as if the listener’s head is placed right where the microphone is placed. For PNME’s performance, this meant that the audience’s ears were placed on stage between a cello and clarinet, surrounded by an ensemble of musicians.
Yes, PNME asked the audience to wear headphones at a live music performance. Through some clever choices, the sounds from the venue bled through (even the soft gasps and chuckles from the audience), creating a unique and immersive soundscape. With an original composition in tow, PNME used this technology to enhance the audience’s sensory experience, skillfully layering techniques to develop an intimate and expressive show, and making a live performance an audio exploration.
Recently, arts and entertainment organizations have relied on binaural technology to develop immersive, three-dimensional audio for digital experiences - largely AR/VR, film, and the metaverse - but can live arts co-opt the technology for their own immersive experiences? Can this technology live on the stage for in-person audiences, too?
Binaural Audio
Audio is most commonly recorded using monophonic recording, one microphone and one audio channel, or stereophonic recording, two microphones spaced away from each other and two audio channels. The latter gives the recording artist control over the audio that is transferred to the listener’s left and right hearing, a common technique to immerse the listener by tricking the brain into believing that the sound is happening between their ears. Surround sound takes stereophonic a step further by increasing the number of microphones to five to create a more robust sound field. We typically experience surround sound as if the sound is happening all around us, like at a movie theatre. Some effects may be directionally detected, but overall, we hear a harmonious blending of sound.
While stereophonic is limited to left and right, binaural audio gives the listener a three dimensional experience. Instead of hearing a balance of instruments like in a symphony hall, binaural audio places the listener in the center of the symphony’s stage.
Experience it yourself - put on a pair of headphones and play the video below.
Developing binaural audio technology required sound engineers to understand how the human brain interprets sound as it travels through the ear. The brain relies, in part, on sound wave timing to decipher where a sound is coming from. Stereophonic audio, with left and right channels, tricks the brain into believing that sound is coming from one side by delivering sound to one ear faster than it delivers sound to the other. Binaural audio uses a similar method, but with more precision.
By placing two microphones where one’s eardrums would be in a dummy head, the sound that the microphone picks up mimics the density and timbre that human ears experience. Replicating human anatomy in this way also allows the microphone to mimic sound wave timing of the environment, too. We never hear audio completely from the source, rather from sound waves bouncing off of other surfaces like walls, floors, and fabrics before they register in our ears. This effect gives the listener an experience almost identical to what they would hear with their own ears at the recording position.
Figure 2: Inside the Neumann KU 100 Microphone. Source: Neumann KU 100
Immersive Audio for the Art Audience
The Neumann KU 100 - PNME’s choice of binaural microphone - is a leader in this space. Neumann’s parent company, Sennheiser, stated earlier this year that they intend to pivot to professional audio sales only. Given their success in the most recent season of Stranger Things, VR recordings, and a pre-COVID Pink Floyd museum exhibition, they seem to be betting on an emerging trend across entertainment markets to move toward immersive audio. The art world has joined the trend, too, using immersive audio to enhance museum exhibitions with soundscapes to meet audiences' growing demand for sensorily immersive experiences.
The Binaural Audio Experience at Pittsburgh New Music Ensemble
For PNME’s live performance, incorporating a binaural microphone into the experience took a fair bit of equipment and setup. Every seat in the house required a working, hardwired headset. Opting for hardwired meant the headsets worked more reliably than bluetooth and the sound quality was better. This also meant that audience members could easily disconnect cables on the way to their seats.
Figure 3: PNME’s headset setup at Pittsburgh’s City Theatre. Source: author.
To ensure the headsets were in working order before the show began, PNME’s Artistic Director Kevin Noe came out to demo ‘Michel’ the microphone for the audience. By providing an audio-tactile introduction, he verified cables were all attached and gave a sneak peek of the binaural technology. A whisper behind Michel’s left ear felt like Kevin Noe was standing right behind my seat whispering a private ASMR experience in my own ear. The demo was enough to imagine my own head where Michel’s dummy head was placed on stage and guaranteed the audience was prepared for the show.
The test also demonstrated the advantages of using open-back headphones with binaural audio. Whereas closed-back headphones seal out noise from the outside world, open-back headphones let the sound from the outside world seep through - in this case, the audience and performers. For PNME’s binaural audio application, this created the opportunity to layer tracks pre-recorded on the binaural microphone with music performed live in the theatre along with any sounds Michel the microphone was picking up.
PNME’s show was composed for the binaural microphone. Composer Stephen Bryant intentionally placed different instruments and singers in specific locations around Michel to elicit different effects for the audience. The ensemble’s final show of their 2022 season, The Treachery of Sounds, explored the creative capabilities of the binaural microphone by layering the ensemble’s music with several mediums - Magritte’s surrealist paintings, theatrical sidebars, storytelling and poetry set to lyrics by Kevin Noe and Lindsay Kesselman, and dance with original choreography by Deeply Rooted’s Gary Abbott. Each medium explored how the binaural microphone could enhance the work.
Figure 4: Magritte’s Golconda. Source: Rene Magritte
The most successful applications exploited the headset’s capabilities of layering prerecorded tracks with live sound. The very first work, with an animated version of Magritte’s Golconda as a backdrop, ended by hearing the familiar trickle of rain through the headphones, at first imperceptible layered underneath the cello heard both on the stage and behind ones head. In another piece, opera was layered over a prerecorded track of the very same opera performance performed live behind Michel’s head, allowing the singer to echo over and over in the audience’s ears. Yet another application played with the audience’s perception of spoken word poetry as it was whispered behind Michel’s ear as musicians played on stage. It wasn’t until this well-timed piece that I noticed that the musicians, too, had headphones on for the experience.
The overall effect was subtle and intimate. As an audience member, I could imagine myself in many places across the stage. I was at once in my seat, on the stage, and in a painting. The success of this immersive experience relied on extensive work to test out what subtle audio expressions translated well through the microphone. PNME workshopped the commission with their audience to better understand their experience with immersive headphones. After some testing, feedback included: “Hyperhuman”, “Intimately connected”, “I’ve never experienced music so deeply”, “My perspective is completely different.” "I feel like I can finally hear all the PARTS!"
While the experience was well worth it as an audience member, it’s more difficult to say if the investment is worthwhile from an organizational perspective. The microphone is a nine thousand dollar investment, not to mention the headphones needed for every audience member. From a resource perspective, this endeavor takes considerable setup time. According to PNME, touring with Michel was particularly difficult when a variety of theaters needed to be hardwired quickly.
While this live application was a delight, the microphone has been used by other organizations to make their digital works more immersive. Arts organizations emerging from pandemic lockdowns have had to make choices about whether or not to continue offering digital or hybrid performance options. The Neumann KU 100 offers an investment that enhances the work for both in-person and at-home audiences if used thoughtfully - placing the audience’s ears on stage whether they are in the theatre or on the couch.
Still, the microphone relies on an engaged audience who are willing to take a leap of faith and don headphones at a live performance to listen in an entirely new way. As an avid arts attendee, I was primed for and invested in this atypical experience. Others who came with me said plainly that it was a fine addition, but not nearly as moving as the intimate and sensory explosion I felt. Still, Michel gave all audience members a lot to talk about, and the performance’s most successful moments were still universally felt.
Whether that’s worth the tangled web of cords for your organization’s tech crew and the hefty price tag is up to you.