In this episode of the Arts Management and Technology Podcast: Tech in the Arts, Luna Lu speaks with Erin Reilly about the future of immersive storytelling, the transition from the mobile era to the spatial era, and the evolving relationship between AI, audience engagement, and world-building. Drawing from her work at the Texas Immersive Institute, Erin discusses how immersive media is reshaping communication, education, and entertainment through VR, AR, mixed reality, and multi-sensory experiences. She also reflects on ethical questions surrounding AI, participatory storytelling, and how emerging creators can design more inclusive and audience-centered immersive worlds.
Show Notes
Transcript
Erin Reilly
I really feel like immersive media is moving us from the mobile era to the spatial era. In fact, we are right now kind of still at the end of the mobile era, but attention right now is at a deficit. In fact, most people swipe, scroll and forget within eight seconds, which is less than the attention of a goldfish.
Luna Lu
Welcome to another episode of Tech in the Arts, the podcast series from the Arts Management and Technology Laboratory at Carnegie Mellon University. My name is Luna Lu, the technology manager and podcast engineer at AMT Lab. Today I'm excited to be joined by creator, educator and strategist Erin Reilly as part of the SONA Immersive Storytelling Festival.
SONA is the annual festival at Carnegie Mellon that brings together filmmakers, artists, researchers, and industry professionals to explore the future of storytelling through immersive media like VR, AR, interactive experiences. Before we begin, I like to share a little background for our listeners. Erin is a founding director of the Texas Immersive Institute and a Professor of practice in the School of Advertising and Public Relations at Moody College of Communication at the University of Texas at Austin.
She also serves as a board chair of the Immersive Experience Alliance, where she helps support the growth of the global immersive media community. Our work focuses on immersive storytelling, emerging technologies, and building pathways for the next generation of creators to work across media technology innovation.
Thank you so much for being here today. I'm really looking forward to talking about your journey, your work in immersive media, and how you see emerging technology help shaping the future of storytelling and creative careers. So just to get us started, you've worked at the intersection of immersive media education and industry innovations, so what originally drew you to the space?
Erin Reilly
I think my curiosity always, I have always been super curious and a lifelong learner with a growth mindset, so I'm probably one of those early adopters at any time. A new emerging technology comes out, I'm the first to get my hands on it. Not really play with it because I'm a technologist, but play with it because is this really gonna serve an audience, right? And so I see a lot of pattern recognition that not very many people do, and often look at it from a different lens and try to identify different groups, especially fam, communities that might wanna adopt it. While at the same time trying to look to see if, uh, who's being left out of the equation.
Luna Lu
Thank you. Just talking about that, where do you see immersive media having the biggest impact outside of the arts and entertainment and industry that we're in right now? How do you think that's impacting the audience?
Erin Reilly
Yeah. I really feel like immersive media is moving us from the mobile era to the spatial era. In fact, we are right now kind of still at the end of the mobile era. But attention right now is at a deficit. In fact, most people swipe, scroll and forget within eight seconds, which is less than the attention of a goldfish. So when I think about going from the mobile era to the spatial era, I think that the mobile device will stay in our pocket, and mobile would become an action, not an object. It'll become a verb again and not an object, and we will step into the spatial era, step into our story, start living and embodying them.
Luna Lu
I was just talking with another guest speaker at SONA yesterday about the relationships between Story and Medium. From your perspective, how can investing in new approaches, products or experiences through immersive technology reshape storytelling and audience engagement in general?
Erin Reilly
I think actually we always start with an audience, and I believe that immersive media is going to impact more than the entertainment industry. In fact, I get a majority of people coming to me wanting it to shape new forms of training for the workforce. The workforce is going through huge disruptions right now with the rise of AI, and a lot of people are wondering, will we have an universal salary for people. And I always think, God, wouldn't that be nice? We could all then focus on our passions and the stories we wanna do, but I think that to do that, to even get to that point, we need to really start thinking about how, uh, these tools are being used kind of in a more public way.
Often we have a really big distribution issue right now with immersive media. They're shown at festivals. They're often one-offs. They're hard to actually transport and move. We need to really think through new ways to localize it and to distribute it to the many in order to have the impact we want.
Luna Lu
Well, thank you for your insight. And you mentioned about AI, you wrote a book chapter called The Remix in the Age of AI. How are you integrating AI into your immersive storytelling or your workflow, and where do you see AI go in the future?
Erin Reilly
Yeah, I'm kind of fascinated by AI, but I feel like I take a critical eye to AI, no pun intended. And the way I do that is I always feel like you can't be a naysayer unless you've tried it, right? So you have to play with it and tinker with it. The ways I've played with it, I've been using it for quite some time. In fact, the early development was when I was doing my leveraging engagement framework, where I was identifying the motivations to engage in our passions.
I was working with IBM at the time, and we had validated eight core motivations that fans really spoke through natural language processing, and that model then was put into Watson. We started to do a lot of tests on, by the way, people were talking on social media, could we see churn? Could I actually use it for, for example, cable TV or streaming and make recommendations based on scripts on what would motivate a fan to watch the next type of show similar to the shows they were watching.
And at the same time, I was like, hmm, this is interesting, but I also am not sure I want to be an algorithm. Right. So I'm hesitant at the same time of like, do I really want to reveal all of my data and not own my data and have these big five companies own all my data in order to personalize something so much and.
I wrote an article about that called Fan Favorites, and it was when I was studying global sports soccer specifically, and all of a sudden I started getting all these ads on soccer. I was just studying soccer fans, so it started to really mess up my algorithm because of the data I was collecting. Going forward, I've explored generative AI in new ways. In fact, I created a model of all of my own research and my classes and the way I talk and my books and publications, and I created a digital twin of myself called Professor Moxie. She's a 3D avatar and she kinda looks like me. We created her where she's a little over the top, uses her hands very much like I do when I speak.
And I tested her in my creativity and AI class where I had the students actually talk to Professor Moxie. At first, they were like, oh, this is really cool. How did you build that character? She kind of sounds like you because I used 11 labs, Voice Changer. So she sounded like me with through that API, and she was like, you know, she kind of talks like you.
And I was like, okay, great. She's gonna teach Thursday's classes. I'll teach Tuesday, she'll teach Thursday. The students were like, no way. So it really sparked a good ethical discussion on like generative AI and their process in learning and when do they fit in, when you're perhaps practicing something or memorization.
But they did not want Professor Moxie to teach my class because they were like, well, we're paying you, you are a human. Besides, you run your class like a DJ, you're fun. And you know, she's kind of weird. It was a really good discussion like, okay, I'm not gonna be out of a job. There's still that human connection that no matter how much I can personalize and make her sound like me, make her talk like me, make her have the same type of quirks, she can't replace that connection of me being able to really riff off of someone and understand someone and smile at them.
Luna Lu
Wow, I wish all the audiences could see your hand movements and just how passionate you are about this, and I love seeing the human connection between you and your students. Now bring everything back to Texas. Can you tell us a little bit about what Texas Immersive Institute is and what gap were you trying to fill in when you found it?
Erin Reilly
So when I started, I was recruited to Moody College of Communication to be the inaugural director of innovation and entrepreneurship. So I was in the dean's office trying to really think about the interdisciplinary efforts across the seven majors in the College of Communication, and then I started looking outward to other colleges and how they were starting to apply or connect with emerging technology.
At the time, there was a research grant from the provost, and the provost asked me to really put a working group together. Around the different colleges that were starting to tap into emerging technology. And together we did three working group sessions and wrote a white paper for our university called Immersive in a university setting.
And we found things like what a SU was doing in immersive media, what Michigan State was doing in a university, and how we were kind of disparate across our campus. That perhaps there was an opportunity to really create connections and to really foster a new type of institute. So I shared that with my dean and the dean thought, Hey, why don't you step off of being the director of Innovation and entrepreneurship and Launch Texas Immersive Institute?
I stepped down and got to focus again on my area of interest instead of kind of more writ large, and began with really thinking about what was the core. Literacies students needed to acquire as we start moving into the spatial era, and that goes from moving from new media literacies to digital literacies, but really thinking about AI literacies and spatial literacies
Luna Lu
After spending time at SONA, does it give you any optimistic views about the next generation of creators, or does it not?
Erin Reilly
I'm really excited to be here at Carnegie Mellon University. I've known people over at the ETC for decades now. And we've worked in consortiums together, so it was nice to get a visit there and see the messy creative technology sandbox.
That feels like home to me. I think what Stephan Casper is doing with the Sona Festival is amazing. We met years ago when we both brought our students to the Infinity Festival in Los Angeles, and I love that he was inspired with what we both saw in the entertainment business. To bring it back to kind of a broader conversation about how does language and cultures and in society connect to these immersive storytelling?
And we're both have huge passions for storytelling at the center of these immersive media tools. I think that we've always been storytellers from like sitting around the campfire to cave paintings, to like broadcasting to many. And I think the storytelling that we're doing now through the examples I'm seeing at SONA Festival are really representing stories that are not told, and I think that's very common in extended reality is you see a lot of women in extended reality and you see a lot of people of different cultures and races and genders in XR. Often, we're pushed out of the traditional mediums, and we look for new platforms and new communities that really respect and want our voices heard. And you're seeing that in the VR examples that he is showcasing at this festival.
Luna Lu
Let's give a shout out to Stephan. Thank you for putting everything together right, and as a storyteller, Erin, what makes a story work well in immersive formats versus traditional media in your opinion?
Erin Reilly
I think the first thing you do when you are an immersive designer is begin with world building. So I had an opportunity when I was at USC to really learn from Henry Jenkins and Alex McDowell. Alex McDowell, I believe, coined the term world building. He did it with Minority Report. He was hired as a production designer before even the script writer was, and he was given the prompt. If the world was only 50 meters in diameter, what type of world would you build?
And so you see in the film Minority Report he built up, right? And you had different levels, kind of health, you know, uh, like Dante’s Inferno as you went higher and higher up. And I think that's interesting because when you think about world building first, you're thinking about the space and you're thinking about how people move through it.
Like going up instead of like going farther out. But they also think about like the politics and the language and the transportation and the everyday movement through it. And when you get to world building from there. You then have multiple stories, which goes to Henry Jenkins work on transmedia storytelling.
That's across media, and that often in a transmedia experience, you have multiple stories or multiple entry points when you world build in a spatial era. You have multiple entry points to get into that world and into following a particular story. What we're used to doing with plot and having a beginning, middle, and an end is really changing in immersive media because you're able to now actually jump in at any entry point and kind of in a sandbox way, find your way to the end of your choice.
That brings us to agency, right? The user is at the center of the story for an immersive media experience. That participation is a contract between the creator and the audience member, and to really gain that audience's trust, you can't just pretend to give them agency. You have to co-create with them. I think that's a different way of thinking that often is hard for students when they're first coming to immersive storytelling.
They often come with baggage of learning in English class, beginning, middle, and an end, right? And what a traditional plot is. And I think we have to break them out of that by first actually just imagining or watching what a space looks like. I always say go people watch in like an atrium and watch all the different stories that surround you, and then imagine what that world is if you took it into augmented reality or virtual reality or mixed reality, or even through projection mapping in that space.
It doesn't always have to be in a headset. Immersion to me is live performative theater. Immersive theater, all the way to the suitcase of acronyms of extended reality. These are the tools that give us a chance to really reimagine what our relationship is with our guests.
Luna Lu
Well, thank you for that. And just talking about any immersive experiences, the audience has more control over what they do and what they see.
Right. I can turn my head around if I'm in a headset and things like that. How do guide the attention without the traditional camera techniques?
Erin Reilly
Yeah. This is where my research on VR and empathy comes into play. Years ago, I think it was 2016, and Chris Milk, who was one of the co-founders of Within, was saying that VR is the empathy machine.
And I don't know, here's my critical hat. I was kind of like, really? Are you sure about that? I'm not so sure. So I did this study, it was the last study I did at USC’s Annenberg Innovation Lab, and it was looking at six different VR experiences that were all thematically around war. But I did some that were animated, some that were live action, some that like had graphics on top of it, some that were first person point of view or third person point of view.
And in the qualitative research that we gathered and we tested with a lot of college students going through these six different studies. First, and we had them try a couple of different studies. 'cause this was 2016. Some had never even been in a VR headset. So like let's level set first. Okay, now that you've got the all factor, let me show you a couple more and get you outta that moment, right?
And then once we did that, we found that audio was the number one medium that really fostered empathy in an immersive environment. Or draws your attention, right? We're not looking at stereo anymore, we're looking at, you know, five one or seven one or en BOUs sonic sound. And when I turn my head over here, you know, I'm going to draw my attention to the back, right?
Or like I can move myself around and sound more than sight is the one that will make you turn your head.
Luna Lu
I believe that 'cause when you snap your finger, that's where my attention went directly. And just to wrap us up, where do you see the immersive storytelling or immersive experiences in the next 5 to 10 years? Where do you see it go?
Erin Reilly
So what we're doing at Texas Immersive Institute is we're really trying to push into the breadth and depth of multi-sensory experiences and how they harmonize together. So I never think of a story just from one medium. I think about a combination of mediums. In concert really.
For example, in our Anchored Memories project, we are doing haptics and virtual reality, and we're using the sense of wind and rain on the person to really kind of enhance their experience beyond the sight and sound you can do in a 360 environment. And in our inner worlds lab, we're doing a heart rate monitor mixed with a mixed reality experience so you can play.
It's for neurodivergent young adults on emotional regulation and heartbeats. Their heart rate is about learning how to regulate their emotions. The mixed reality puts it into a comfortable space for them instead of completely removing them from their space, which would cause more anxiety and not make them feel comfortable.
So if we're going to help them practice their emotional regulation, mixed reality was a better call than say a virtual reality experience. And the heart rate gives them a tool for them to actually be more aware of their presence and embodiment.
Luna Lu
Interesting. Thank you for that take. Do you have anything else you would like to share with our audiences?
Erin Reilly
I would say it's interesting, immersive media and experiences are really being developed and studied and practiced across so many different disciplines. I've even had someone tell me, I'm not sure how immersive media fits into the College of Communication, and I've only been in College of Communications for 20 plus years.
And I think that's where we need to level set and let people know and understand that, uh, immersive media is the next wave of how we communicate, how we shape our lives, how we learn, how we entertain ourselves, uh, how we, uh, educate and it it's going to impact. Every vertical. So just like the television or the movie theater changed the way we saw our storytelling, immersive media is really now giving us a chance to live within our stories.
Luna Lu
Thank you so much for all your insights and for joining us today. Erin. It's been incredibly inspirational to hear about your work at the Texas Immersive Institute. And your insights on how immersive media is shaping the future of storytelling, education, and creative collaboration. For our listeners, please check out the links in our show notes to learn more about Aaron's work and the Texas Immersive Institute.
Dr. Brett Crawford
Thank you for listening to this episode of Tech in the Arts. If you found this episode to be informative, educational, or inspirational, be sure to check out our other episodes and send this to another arts or technology aficionado in your life. If you want to know more about arts management and technology, check out our website at amt-lab.org, or you can email us at info@amtlab.org.
You can follow us on Instagram at Tech in the Arts, or on Facebook and LinkedIn at our full name Arts Management and Technology Lab. Our season runs from September through May. So feel free to follow us on your preferred podcast app, be it iTunes, Spotify, or YouTube.