In this episode of Tech in the Arts, Sofia Akhmanaeva, AMT Lab’s social media manager, talks with Eric Williams and Jordan Herron, presenters at this year’s SONA Immersive Storytelling Festival.
Williams is a Professor at Ohio University and the director of the cinematic virtual reality (Cine-VR) initiatives at the Game Research and Immersive Design (GRID) Lab. His work spans award-winning film, television, and interactive media, including screenplays for Universal Studios, Fox Online, and American Movie Classics, as well as numerous published books and podcast series on storytelling.
Herron is an Immersive Media & Audio Producer at GRID Lab. He began working at GRID Lab in 2018 as a student employee and now specializes in audio storytelling, immersive sound design, and Spatial Audio Production.
At SONA, Williams and Herron presented a series of cinematic VR projects designed to support public service training. Using 360° filmmaking, they aimed to evoke empathy and create shared, memorable experiences.
Show Notes
Project Highlights:
Transcript
Sofia Akhmanaeva
Welcome to Tech and the Arts, the podcast of the Arts Management and Technology Lab at Carnegie Mellon University, or for short AMT Lab. I'm Sofia Akhmanaeva, marketing Manager at AMT Lab, and today I am joined by Eric Williams and Jordan Herron. Eric and Jordan's work was recently highlighted at the SONA Festival at Carnegie Mellon University, organized by the Department of Languages, Cultures, and Applied Linguistics. SONA is a forward-thinking festival that provides a platform for diverse voices and groundbreaking stories that redefine the boundaries of storytelling and technology, presenting the works of creators in XR, 360 video, and Interactive Media.
Very nice to have you here today. Tell me please about what you do with VR and how is it unique and different from what people usually do with VR?
Eric Williams
Our area of specialty is called cinematic virtual reality, and cinematic virtual reality uses 360-degree cameras and live actors, and so it's different from what most people conceptualize as VR. I think that most people imagine avatars that are created digitally in completely fictional and recreated worlds.
But all of our work is done on real locations with real people, and it's more of a linear experience in that you're watching a film. But the unique aspect using VR is that you're actually in the middle of it. It's 360 degrees around you, so you can choose to look anywhere at any time.
Jordan Herron
Yeah, to summarize that it's, it's like filmmaking, but in VR essentially.
And so yeah, a lot of people expect a simulated environment. But when they watch one of our scenarios, they're captivated by the fact that these are real world captured images.
Sofia Akhmanaeva
So it's called cine-VR, right?
Eric Williams
Cine for cinematic.
Sofia Akhmanaeva
Yeah. Great. cine-VR. And I would say among all what I saw in the SONA programming, this what you do is the most functional part of what people can do with VR. Could you speak a little bit more on what projects you are working on for public service simulations in VR and how it started?
Jordan Herron
So a main thing that is popular with the work that we do at the GRID Lab is currently making cinematic virtual reality trainings for public sectors. And currently one of the latest ones we've done was for the Ohio Peace Officers Training Academy.
And Eric, would you like to speak a little more on that?
Eric Williams
Sure. So that project is seen by 30,000 police officers, peace officers across the state of Ohio. But we're also working with healthcare providers to have them have a more empathetic relationship with their patients.
And we're just, actually this last month, we started working with the state of Ohio and doing projects for the Department of Health and the Department of Human Services. And so it's everywhere from human resource training to safety and security training, to… How would you describe the ODH stuff?
Jordan Herron
I would describe that almost as, trying to develop an understanding for workers to understand the clientele better, and understanding that individuals they're interacting with in the public sector have a life that surrounds them. And so I think it's very empathetic. The training we're...
Eric Williams
I was just going to use that word. That's kind of the thread throughout it, is how do you get one group of working professionals to have empathy for the people that they are working with? And I think that's kind of the thread throughout all of those different areas.
Sofia Akhmanaeva
I think this is very unique perspective. What people usually imagine, speaking of VR as training opportunity as something very functional in terms of like, oh, how do I do this particular thing? Or like operation with equipment or something like that. I experienced one of the simulations that you brought to SONA, and I can tell that it's really empathy evoking. Could you describe a little bit how this experience looked for those who did not experience it?
Jordan Herron
One way I try to summarize it is we allow for trying to create experiences that are almost a shared living experience that everybody can kind of use as a foundation for a particular area of training. And so, some of the experiences we've created might be used in, let's say a law enforcement piece might be used in a classroom setting to teach deescalation.
That same experience could be used in another classroom setting to teach report writing and I think there's something to the fact that without having a direct learning objective per experience, they're able to get a longevity out of taking the experiences and pulling it apart and seeing how your life kind of converges with that as well.
And I think that's something that the training allows is for individuals to see themselves in the training and maybe others they've worked with or that they've interacted with as well.
Eric Williams
Well, and I think the whole idea of a lived experience is really interesting too, on two different levels. One is, if every peace officer in your state has gone through these specific experiences, then they've all been in Chet's living room, right? They can all have a common reference point when they say, would you go this far? Would you go this far? Would you pull back at this point? Would you go forward? All 30,000 of them have the same lived experience that they can talk about.
But here's the thing that I think is even kind of more intriguing, and I want to be very clear that we're not doing research in this area. This is just our theories. I think that people remember these experiences differently than they remember watching a film because when you watch a movie, you remember being in the theater or being in the room watching the movie.
But we have found that people that have gone through these experiences not only remember them differently, they remember them longer, and they remember them in a way that they talk about. Remember when we were in Chet's living room and it's like, whoa, wait a minute. You weren't in Chet's living room.
Like imagine watching a movie and then talking about the movie. Oh yeah. Remember when you never talk about movies that way. So these stories are getting into people's heads in a way that they are remembering them as if, you know, they know they weren't there, but they're remembering them as if they were there.
Jordan Herron
They're remembering it like a lived experience.
Eric Williams
Yeah. Yeah. And I think that's incredibly powerful. We don't even understand how that's working yet. Mm-hmm.
Sofia Akhmanaeva
Yeah, that would be great to dive deeper into it in the future, I guess. And I can confirm, I experienced one of the cases with a mentally ill woman who was reported to police, because she just was sitting with the knife in the room.
And with this experience, you either experience the one who experienced it. Sitting right next to her, and it's so different from just being on a distance. As a police officer, I could feel the pressure in my body, like physically the tension of the situation and that's powerful.
Eric Williams
That's really great to hear because we spent a long time discussing exactly where we wanted that camera to be.
Mm-hmm. And we were trying to go for that. We were trying to get you to feel that tension. It's similar to the officer who was standing in the door from her perspective, but it's not exactly, beause you are on top of this woman.
Sofia Akhmanaeva
You're like right in the middle. Yeah. Between them.
Eric Williams
Yeah. And that's what we're going for. It's that visceral emotional memory. I'll be curious to think about how you remember this experience two weeks from now. I think it'll be interesting.
Sofia Akhmanaeva
Could you please tell about the story that you spoke of within your talk, how it all started? What was the first case for the healthcare system? I think it's really moving.
Eric Williams
Sure. So this was, this was pre-COVID. We were working with the Diabetes Institute, and they wanted to create experiences that informed healthcare providers of reasons why.
Some of their patients might not be making it to appointments on time or missing their medications. And so we did a day in the life of a woman named Lula Mae. Lula Mae is a woman in her sixties who was living below the poverty line who was suffering from diabetes. And so it was all the different reasons why couldn't she get to, you know, she couldn't get access to her car or things were breaking down, or she didn't have money to buy food for everybody. So she was buying food for her grandchildren rather than herself. There were all these reasons why, she wasn't doing certain things that made her healthcare providers kinda roll their eyes.
And we started doing those sorts of stories, for that client. And we did that for about a year and a half. And actually, Dr. Elizabeth Beverly was the one who started doing these experiments on the longevity of the memories. And she found that the healthcare providers who went through cine-VR training remembered the lessons longer than, and had more empathy than those who went through the traditional.
Training of the same knowledge, of the same information. And so from there, once we had made the Lula Mae and the Destiny Series, that series was seen by, a guy named John Bourne who was working at Ohio University in the Voinovich Center. John's background is in law enforcement.
He was the colonel of the state Highway Patrol for Ohio, and he recognized that Cinematic virtual reality would have an impact on police training. And so then he and his colleagues came up with the idea of creating the Appalachian Law Enforcement Initiative, which was using police officers from small towns rather than the big cities, to come up with training that they thought was important in the areas of deescalation and empathy.
And so that's when we created a pilot series and we did the Chet scenario and we did the Dion scenario or the Dion story and the Chet story. And then that's where it really caught on from there.
Sofia Akhmanaeva
This is very powerful application of VR, and I think it's breaking this narrative about VR is something that brings you farther from reality because this actually brings you closer to someone else's reality, and especially with all this societal tension that we have right now. Empathy is kind of key.
Let's switch to more technical aspects of your work. In your presentation, you were talking about the challenges that you have and how you solve them. Could you elaborate on this a little bit more, please? What challenges do you have with technologists and like for providing the most authentic experience?
Jordan Herron:
I would start with a main thing that we — I think we've done a good job of starting to figure this out — but a big thing that's a challenge with cinematic virtual reality filmmaking as compared to traditional filmmaking is there is no frame and the camera is seeing in all directions. And so it's a lot much larger task.
Trying to hide all of the crew and all of your lighting gear and all of those things have to be completely out of sight as opposed to in traditional film, you put all those things behind the camera.
Sofia Akhmanaeva
I didn't think about it, and now I'm thinking how is it even possible if it's 360?
Eric Williams
Well, so what we do is there's a thing called plating, and so we end up recording the scene multiple times, and so for instance, if you have a light up in the corner of your room. When you film the scene with all of the actors and the actors are being lit, that's the first time you record it. Then you get rid of all the actors. You get rid of the light, but you figure out how to light the room in a way that looks identical except the light isn't in the corner where it was. The light is somewhere else.
Then you record the scene a second time. That would allow you to create a plate. Around that corner of the room where there used to be a light, but now there isn't a light, and so you just have a little box, a little plate of that corner, and you pace that over your first scene with all the actors, and now your light has disappeared.
Sofia Akhmanaeva
That sounds crazy.
Jordan Herron
And that's another reason we try not to move the camera at all because it makes it very difficult to capture those plates after we've got a good performance. If the camera ends up getting bumped or moved, we have to consider do we start over? Do we try and capture a plate now before we continue doing new takes? Or if we need to change a camera battery, should we capture plates before we change the camera battery out? And so it causes a lot of considerations to be had when you're on set doing these sort of productions.
Eric Williams
And I think the other thing that kind of goes hand in glove with that which I think is more artistic, is we spend a lot of time trying to figure out how we know where people are looking.
Because everybody says, oh yeah, well, when you're in a 360 environment, the audience can look anywhere they want. But think about that in real life. I mean, when you're in a lecture hall. People aren't looking all over the place. You, you can pretty much guess two or three spots where people are probably looking and if you start paying attention or start influencing the location, you can start making people look in certain locations.
It's just like a magician, right? A magician knows exactly where people are looking at all times, even though you feel like you have the freedom to look wherever you want. And so what a magician does is gives you the illusion of freedom, but then is controlling what you're doing. And that's what we're trying to do using color, light, sound, movement.
There's all those different tricks. I think that's one of the biggest things we've been figuring out in the last two or three years.
Jordan Herron
And I would add to that, another really important thing that we try to focus on is doing that in a way. That goes unnoticed by the viewer. We don't want you to become completely aware. We're trying to pull your attention and draw your attention. We want you to be influenced and that makes the experience much more immersive and passive.
Because we're not breaking the immersion by suddenly making you do something that feels forced. We're trying to influence you to look in specific directions so it feels natural, but we know that it's not. Like a magician.
Sofia Akhmanaeva
Quick technical question. How long does it take to shoot a 10-minute experience?
Eric Williams
10 minutes? Is that what you mean? Or like from start to finish?
Sofia Akhmanaeva
No, the whole production process was like all the like plates, process that you just described and everything. To produce, I would say not to shoot but to produce a 10 minute piece.
Eric Williams
So if we were doing one 10 minute piece, let me do the pre-production and production and then maybe you can do the post. Absolutely. It will take about a month to write a script. We're typically working with content experts and so we do an outline with the content experts, and then we do three drafts. We do one draft per week, so take about a month to write. During the last two weeks of that process, we are casting location, scouting, we're doing costumes. I'd give that a month to kind of cast, write the scripts. We do typically a few online rehearsals to make sure that the actors know their lines. So this is going into month two. And then when we go into production, we will do one 10 minute scene even up to a 20 minute scene.
But we'll do one scene, we'll shoot it in one day. But the reason why we can shoot it in one day is we actually take the day before that and we rehearse for an entire day. So it's more like a theater piece. We want the actors to come on set for rehearsal with all their lines memorized, and we do all the blocking, and we really figure out exactly what's going to happen, and then we record it the second day.
And one of the reasons why we take an entire day, I mean, a 10-minute piece is going to be shot in its entirety. And so once you say action, I mean action is the beginning of the scene, you turn the camera on and you do the entire 10-minute scene. Then you say, cut. And then I go around as the director and ask, say, we have eight actors in that scene. I'll go around to each actor and kind of thumbs up or thumbs down. And if one actor thinks that the Performance wasn't spot on. If one actor gives a thumbs down, we do the entire piece again. We had one 14-minute scene or 12 minute scene that we did 19 times.
I mean, it took us, I mean, hour, it took us the whole day. So one day of rehearsal, one day of shooting, and then that's production, we're wrapped. So a month and five or six weeks total for production. Then we pass it off to post-production.
Jordan Herron
And then the post-production side, how long it takes is - are there graphics? So we have some pieces that have next to no graphics. The video will start, we'll have a title card, and then from there on out, there's no added graphics put into the experiences. Whereas we have some other experiences that are very heavily relying on the graphics.
Eric Williams
A quick example, we had one experience where an officer’s trying to talk a teenage girl off of a bridge. She's thinking about jumping and as he talks to her, there's a graphic behind her that shows her emotional state based on his questions. And so for the entire 15 minutes you hear her, but you can also see her emotions, based on if he's pissing her off or if he's calming her down. If he's empathizing with her. If he's trusting of her. And so that was heavy on graphics, just as an example,
Jordan Herron
A great example to use, in fact, because we had two versions of that scenario. We had one that didn't have the graphics, was the exact same video. And then the second version included the graphics and the whole part of that is it allows for a contrast to sort of watch it. And what did you think the emotional state was as opposed to now watch it and the graphics are gonna give you more insight as to what the emotional state is. And so if we were to look at that piece and say, how long will the post-production take?
And you remove the graphics as. Something that needs done. I would say that individual piece was roughly 10 to 12 minutes long. The video starts, it goes for 12 minutes and it cuts. There's not a lot of actual editing of the video will still do professional color grading and we'll still do professional sound design, but for a 10 minute piece like that, the post-production could be easily wrapped in a month's time, maybe six weeks tops. But once you introduce the graphics, now we have a lot more meetings where we're reviewing the videos with the client and the subject matter experts to be like, okay, the information being portrayed in the graphics, is it accurate? Is the timing of when it is portrayed accurate?
Okay, that's all good. Now, what does the design need to look like? How do we design the graphics so that what's trying to be portrayed to the audience actually makes sense? And then you might have a week of people trying to decide this emotion is this color, this emotion is this color. Well, what's the background that is the color underneath all of those colors.
And I think that's where things can get a little bit lengthier in post-production. But nonetheless, a piece like that would still be completed within, I'd say eight weeks to 10 weeks time tops. So two months roughly?
Eric Williams
Yeah. So three or four month process from start to finish.
Sofia Akhmanaeva
I'm going to ask a little bit later about the different difference in processes for normal movie shooting and this, but for now, is there some opportunity for interactivity in this experiences? Do you use it now or is it something for a longer term perspective. Kind of, you know, the scenario that like girl jumps or she doesn't jump, and it gets decided in the process.
Jordan Herron
With that idea in mind. I think a big thing of it is that the ideas that Eric and I might have are ahead of where the technology currently exists, of where we can implement those ideas. And so branching storylines is something that we would really like to. Try and accomplish however, current technology and the headsets and having software to support that amount of data and the playback.
That's where we're currently running into walls. And so as much as we want to do branching narratives and have that interactivity, we're sort of currently limited with what we can do with the distribution platforms that currently exist.
Eric Williams
Having said that, well, there's 2 thoughts that go along with that. The first is, there is technology to do branching narratives, but it only really works on your computer. And so you can do it with 360 and you can do it on your computer, but then you lose the whole empathetic memory power of actually doing it into the headset. So one of the things we did on the last series is…
Okay, let me back up for just one second. A big part of the process of this training is, it's not all just in the headset. And so the officers will come to a location, there might be 20 officers being trained, and they'll talk about it for half an hour, 45 minutes before they put the headset on. Right?
They might talk about domestic violence and different techniques. Then you put the headset on and you see the experience. Experience is 12, 15 minutes, take the headset off, and then there's an hour, hour and a half of conversation. And that's the process that they're using to teach and to learn.
And so one of the things we did with the last series is we created an experience where you put the headset on and you see the beginning of a story. There's a man who's having a psychotic break on a public school bus or public transit bus. City bus. And things go sideways and then it stops.
You take the headset off and you say, what would you do? And they talk about it and if they want to interview different people that were outside the bus, they can do that. And they can see 2D videos of those interviews. And then they say, okay, do you want to stick with officer Reagan, or do you want to move over to Officer Quinn to see who ends the story better?
And then they can choose, they put the headset back on and they choose that story. So we took the technology out and just made it a human choice. So we're, we're doing it. We're just not using the technology to do it.
Sofia Akhmanaeva
Yeah, we're going to get back to the topic of the distribution platforms and market, but coming back to the production process, this all sounds like the process of creating VR, senior VR experience takes the whole set of very different skills. It's dramaturgy, it's cinema production, it is also the whole other VR tech piece itself. Could you speak more to this? What set of skills should your team, or one team, have to create quality experiences of this kind.
And is it the whole industry of itself or it's kind of sub-branch of film production industry? Or where does it lie in all this continuum of creative industries race?
Eric Williams:
That's a really fascinating question. The technologies are based on film for the most part, don't you think? But we find that film students really approach storytelling in a very specific way that doesn't necessarily mesh with it. So the technologies are there, to me, it's more like theater plus video games.
And we find that people that come at it from a theater background or a video game background, they see it differently.
Jordan Herron
And to that, I would say a skill that's involved in that is very heavily storytelling. I mean, that is at the heart of any media you're trying to create.
And so to that note, when you're doing traditional film, we can draw attention to whatever we want because we can punch in on things and the door before it gets grabbed or do a panning shot to have the audience looking at very specific things, whereas theater and cinematic virtual reality, we have a full stage going on in front of you, and so we are more of trying to actually.
Draw your attention and influence your attention like we had spoke about earlier, we can't just expect that you're looking at the same place every time. And so we then have to look at it in a much more holistic manner as to how we keep the audience engaged, not just in the piece, but in the story itself, so that if we cut to a different scene or a different environment, they aren't completely thrown off because.
The whole world around them just changed as opposed to a 2D screen in front of you, cutting to a new scene.
Eric Williams:
And I find that theater people are more in tuned with that aspect.
Jordan Herron:
Which to that same note, we try and work with actors that have theater background
Eric Williams
Rather than film.
Jordan Herron
Yeah. And a thing of that is if somebody improvs a little bit or changes their line a little bit when they're theater actors working with other theater actors, we feel as though they're able to pick it up and still work and keep the essence of the piece without being like, “we’ve got to cut. That was not the right line.” Something to that extent. Yeah.
Eric Williams
And I'll tell you one of the things from a teacher's perspective, whenever I have film students in cine-VR classes, they're much more resistant to using the whole 360 space. They want to do everything out in front so they stage it like a play. And you keep saying, “no, no, no, you have to use the whole 360.” It just doesn't function in their brain. But if you take a video game student, they're used to stuff happening in every direction. And so even though they might not have the film background, they are open to the storytelling in that way.
Jordan Herron
Sort of open world storytelling to some degree. Yeah..
Sofia Akhmanaeva
What kind of people are working with you on script? Do you kind of outsource it or you have a whole set team that works on all of your projects?
Jordan Herron
So at the heart of every project that we do, we work with subject matter experts, and that is incredibly important because, as excellent as Eric and I are at the creative and making things look nice and sound nice, It doesn't matter how good it sounds or looks if it's not accurate.
And so that's a big part of not only making sure the information is accurate, but choosing stories to tell that are actually impactful within a specific industry. And so working with subject matter experts, they can sort of say what stories currently need to be told. And from there, individuals like Eric can pick the ball up and create that story.
Eric Williams
One of the things we're trying really hard to do is share the tools that we're coming up with. So Jordan and I have been working together for the last two or three years telling these stories and so we have phrases we'll be like, “oh, we have to puzzle chunk it.”
And so like, he and I know what puzzle chunking means.
Sofia Akhmanaeva
What does it mean?
Eric Williams
Okay. I'm glad you asked. Yes. So puzzle chunking is this one thing that we came up with. We had a story where. There's a police officer and he's come to this house and there's a guy in the basement and he wants to kill himself, but he won't unlock the door.
He'll only talk to you on the phone. So the story is the officer outside in the rain talking on his cell phone, talking to the guy in the basement with his guns, and we cut back and forth outside, inside, outside, inside. But then at a pivotal point, we split the 360 screen. And so if you're staring to the north, you can see the officer outside and if you stare to the south, you can see Tommy in his basement.
But then instead of having there just be a sharp line, we cut along key things. So the officer's car becomes the couch, the way that we've cut the lines. Mm-hmm. The flag on the ceiling becomes the neighbor's roof. And we did that by measuring the distance between the camera and different pieces. And just like a puzzle, we cut it out and we put the chunks together. So that's the puzzle chunk. Jordan designed this thing called the audio swing. And so that's how we can move people's attention from the left to the right, and then you can pair that with a visual swing. So Jordan and I can sit and we're writing, we're talking about scripts, and we're like, “okay, we're gonna puzzle chunk that part together.”
We're going to swing this out. Are we going to do a visual swing or an audio swing? Let's do a visual audio swing. Oh, what if, what if we swing them back visually, but swing them this way? And so we've come up with this creative lexicon to help us tell stories. The problem is, is when we collaborate with other people, they, they're like, what the hell is a puzzle chunk?
And so what we try to do is pull in students and collaborators on projects and get them into what we're doing and then kind of pass it off to them. But that's a slow process.
Sofia Akhmanaeva
Do you currently teach any courses in Ohio University?
Eric Williams
We do. Oh yes. Yeah. So we teach that. But sometimes it's hard to take a 20-year-old and have that 20-year-old then work with 60-year-old detectives who are telling really violent stories. And so there's this mix of youth and enthusiasm and experience and wisdom. And so yes, actually, that's a really good point. I mean, we're currently working with two great cinematographer and gaffer. And they were both former students of ours. Sammy and Elijah were students of ours three or four years ago and are now our…
Jordan Herron
I was gonna say, there's students of Eric's, I was in the same classes as them. Oh, that's true.
Eric Williams
Yeah. So Jordan's one of my students.
Jordan Herron
I did not teach you Sammy and Elijah, if you're listening to this.
Eric Williams
Yeah. That’s true. So yes, we try, but it, it takes a long time. I mean, there's some, there's a difference between teach teaching people and then actually having them be able to do it. Because a lot of times the students are still learning playwriting or screenwriting or, or film directing. And so it takes a little time for that knowledge, I think, to kind of season.
But yeah, so we use a lot of students that have graduated from our program that we weave back in. But I'm thinking more of like, I would love to be able to go and reach out to stage playwrights, and get seasoned playwrights to write for this, but they don't necessarily understand the technology or the tools because these are a little bit different than writing for the stage.
So it's getting them up to speed, I guess.
Sofia Akhmanaeva
Yeah. That's fascinating that it sounds like potentially the whole other specialization on the intersection of psychology, film production, and also VR and game technology. Could you tell more about where you came from personally to doing this kind of projects?
I know Eric has a huge background in film industry and entertainment. Jordan, you did sound design.
Jordan Herron
So I live not far from Ohio University, it’s where I had grown up. So Southeastern Ohio, and I came to OU. In the past I was a musician. I would call myself that, but you're not going to find my music anywhere or anything along those lines.
When I got to OU, I thought I wanted to be a music producer and work in studios. While at OU I discovered some courses that were involving film to moving image. I hadn't honestly, before that, ever considered that I could take my passion for sound and apply it to images on a screen. And that kind of took my creative endeavor to a whole other level, because, at that point, I actually had visuals to accompany any audio that I'm trying to work on. While in those classes, I had taken a virtual reality class that is, or was taught rather by Dr. Charles Linscott, and I found there was a audio position at the GRID Lab that they were hiring student employees.
I went, applied, I got the gig. I met Eric a week later who asked me if I wanted to go to San Francisco, and I said, I would love to go to San Francisco. And I've honestly been at the GRID Lab working on these projects ever since.
Eric Williams
Well, and I think that's an interesting example, right? I mean, Jordan's coming at it from a musical background, and then learning about cine-VR, but he's, he's kind of taking the audio knowledge and adapting it to it, rather than being said in like, I'm a music or I'm a sound recordist for films. He was just kind of figuring it out as he was finding the medium.
Jordan Herron
Yeah, because one of the first things I explored through the courses I took with Dr. Linscott was actually trying to make spatial music. And so a lot of the same software that I currently use for mixing the sound design for our cinematic virtual reality productions. I was originally using to try and take full songs and mix them in a spatial format, and from there I was like, oh, we could take this technology and then start trying to implement it into the virtual reality experiences.
Eric Williams
And I think it might be interesting to talk a little bit about the work we were doing in San Francisco, because the very first project that Jordan and I started working on was nonfiction. And so we were working with a group called The Better Lab at San Francisco General Hospital, and they were trying to figure out how to train emergency room teams to understand each other better. So we went into emergency rooms and dropped five different cameras around an emergency bed, where the gurney would be. One where the surgeon is, one where the head nurse, one where the anesthesiologist, one where the scribe, where all these different people are. And then for an entire weekend, we waited for real emergency room patients to come in and we rolled on every camera, real emergency room situations. And so it was car crashes and stabbings and shootings. I mean, just all these terrible things. Then we would put each of those in the headset so that the head nurse could understand what the anesthesiologist was seeing when somebody came in and they could say, oh, that's why she reaches across and grabs this gear at this particular time. We're trying to create empathy between teammates. And we did that for a while.
We went back and did that twice and we did that again in Columbus. But one of the things we found out with that particular case is they can also then start showing those to medical students before they get to to the emergency room. Because what they were finding was, you're a medical student and you're supposed to go and observe the emergency room. It takes you six weeks to get used to being in the emergency room. And those six weeks aren't used as well as if you were already accustomed to it. Well, they started sending the headsets to students so you could be at home before you even get to San Francisco, and you watch all that stuff in the headset.
It cut the preparation time from six weeks down to three weeks. So cut the preparation time in half because they were watching it all in cine-VR. And then they're like, oh, well, that door goes to, oh, I know what's in that drawer. Like, I've been in here, I've watched 30 hours of emergency room procedures.I know this room. Anyway, that's where we kind of started.
Sofia Akhmanaeva
That's great. Eric, could you tell more about your film experience before? And, why did you switch? How did you switch to this kind of stuff? Why didn't you stay in the entertainment, doing entertainment stuff and all these things?
Eric Williams:
I will tell you the short version. When I first got out of graduate school in the late nineties, I was interested mostly in writing. And so I was writing for Universal Studios. I was developing shows for American Movie Classics. And through a bunch of life experiences. I met my wife, I hiked the Appalachian Trail, and I decided that I didn't necessarily wanna live in Los Angeles or New York City, which is where I was bouncing around.
And so I was trying to figure out what I could do outside of those two coasts. I really fell in love with teaching. I started teaching at the college level, and I just found that being surrounded by creative people all the time, and being paid part of your salary, when you're a professor, is to do creative work. And I thought if I can get paid to teach creative minds and to do creative work, that'd be fantastic. And so I did screenwriting for many, many years. I've taught for 20 years, and it wasn't until about seven or eight years ago that the GRID lab invited me to change schools, and to work in this new medium.
And I said, “I'm not much of a tech person.” I said, “I don't know anything about these technologies.” And John Bowditch, who's the director of the GRID Lab, said, “there's this new film stuff, it's called 360 Video. You'll love it. It's like movie making on steroids.”
And I was like, “okay, I'll try that.” And then I just kind of found my way into it and I kept meeting different people like Jordan and John Bourne and working with John Bowditch and Chip Linscott, and Sammy and Elijah, and it just kind of grew from there.
Sofia Akhmanaeva
Great. It's interesting that teaching turned out to be more creative than working in actual creative industry.
Eric Williams
It's very true. Yeah.
Sofia Akhmanaeva
That's a pivot. Let's get back to the economic question of the industry and the market perspectives. What about the distribution network and the whole issue of getting the work out there?
Where do you see the perspective for these kind of works? In terms of moneymaking, disseminating it, maybe other applications beyond public services, but keeping the same principle of empathy evoking. Does it have any perspectives in entertainment? What are your thoughts on the future of this?
Eric Williams
I'll start. I have a very negative outlook on the entertainment industry. I think that a big part of entertainment is entertaining things with other people, right? And so I do not, I mean, for as much of this stuff as I make, I don't come home, put on my headset, flop on the couch at eight o'clock at night and hang out on a Wednesday night and watch cine-VR.
Like, I would rather be with my wife watching a movie because we can watch that together. I find that a headset is really isolating and I don't enjoy it.
Sofia Akhmanaeva
Isn't it the whole market of people who actually want to escape the reality to some different reality?
Eric Williams
I know. And also novels are the same thing. Like everyone reads a novel by themselves. I know, I know. I'm such a hypocrite, but I personally feel that that's one of the things that has hindered it taking off. I think when virtual reality and augmented reality start to overlap and you can start seeing things and you can start seeing the people, seeing real people that are in the room with you, I think that equation starts to change.
But I think with what we have right now, nobody goes home and watches cine-VR. They just don't, it has to be an assignment. They have to be being trained on something. I think it works incredibly well for training. I think if you're trying to learn about a certain location or a certain topic, doing a documentary, about something in 360, super helpful.
I mean, I think it's one of the most impactful media out there right now. I just don't think that it's entertaining yet. But having said that, I think that there's a tremendous amount of potential to make money in this market. It's just, it's technologically a heavy lift.
I actually think the technology's out there. There just hasn't been a company that's been willing to do the investment. Just think about, Netflix, but for 360.
Sofia Akhmanaeva
But it would be Netflix for training purposes.
Eric Williams
Yeah. And I would put documentary in that area as well, even though I know people watch documentaries as entertainment.
But yes. But for training purposes, I think a Netflix for nonfiction, informational distribution.
Jordan Herron
To add to that, I would say that there's just an overall supply and demand issue. There's not a lot of people currently demanding a film in cinematic virtual reality to watch, versus virtual reality simulated experiences. Right now, if the Netflix for cine VR existed, what is going to be on there? If there's no content to be on there, then even if the platform exists, it's not gonna get very far. If Eric and I got a deal today and we could put all our trainings on there, I don't imagine anybody would be entertained for very long after they got done with our pieces because there's, what else am I going to watch?
Not to mention also the public itself adopting the use of VR headsets and the accessibility to VR headsets is a whole other gap that still needs to be bridged before. Maybe people will become interested in making a lot more content for cinematic virtual reality because now more people have headsets to consume that content.
And so it's just this kind of ever longing thing that we're just waiting for enough content or the tech or something to somehow catch up to each other, and it's just doesn't seem like it's quite done that yet.
Eric Williams
And to make it more complicated. You need to create different file types for different headsets.
Yeah. It's not like you can just release a MP4 and anybody can play that on their computer. If you are using the Pico, you need one type of file format. If you're using the Quest, you need a different file format, and so it's not even…
Jordan Herron
There's not even a universally accepted format to deliver the content at this moment.
Sofia Akhmanaeva
Yeah, we were discussing with AMT Lab team, within the context of SONA and all the VR works presented that. Okay, I can see this kind of experiences usually at this kind of festivals, but what if I want to experience it beyond that, just on demand? Where do I go? And the answer seems to be like nowhere.
Eric Williams
Well, for VR there's, there's more outlets. But for 360 stuff, I mean, you can post it on YouTube and then you can access it through YouTube VR in your headset and watch it. But it's typically at a lower resolution. If you don't have a good streaming it might be jumpy. And the problem we run into with the police officers is most training academies don't want their technology hooked up to the internet. And so they wouldn't even do it streaming, if they could.
Sofia Akhmanaeva
It seems like more B2C model right now, but also technology is not there, right? Like we were talking about all this processing, like computational power, I guess the right term for it to, to do it through VR that it just can't handle that much data.
Yeah. Yeah. Okay.
Eric Williams
And so we literally will go, I mean, they have, 180 headsets around the state. They go around to each headset. And side load them.
Sofia Akhmanaeva
What do you mean?
Eric Williams
So we do a series every year, and the series is about an hour and a half's worth of footage, right? Six different episodes.
You can put that on a thumb drive and you can put the thumb drive into your headset and you just download the videos onto the headset that you have. We have to do that 180 times and that gets the new season out. You send out a handful of people to do that. Do it in a week.
Sofia Akhmanaeva
Yeah. It's, yeah. I guess we can hypothesize about the market development as much as we want, but technology is not there yet. Which is okay waiting and then we'll see.
Jordan Herron
You know, there are things like Meta Quest TV for anybody out there, you can put your things on YouTube or Meta Quest TV and you can access that content on, uh,Meta Quest headset and it'll work.
But again, we can't necessarily put the content that we're creating onto these platforms. And so even though in some cases you could say, “oh, I know where I could put my video so that people can access it and watch it,” we can't just post these videos to…
Eric Williams
Yeah, public service
Jordan Herron
Or public. Yeah.
Eric Williams
And then there's the whole question of how do you monetize it?
Sofia Akhmanaeva
Yeah, this is what I mentioned. B2C I am mistaking B2B, obviously. How many institutions at this point you cine-VR to your knowledge? Like ongoing constant basis for their training?
Eric Williams
What do you mean by - oh, for their training? Yeah.
Sofia Akhmanaeva
Is it like very unique in single cases, kind of institutions around the country, or is disseminated pretty well at this point? Or it takes some kind of special dedication from an institution to implement it.
Eric Williams
It takes an investment on two levels. First of all, it takes an organization, just the investment of getting the headsets and dedicating that they're going to train their people to use headsets. Once they do that, that's the first step. And then the second step is how are you going to create the content or obtain the content.
So what's happening in Ohio at the moment is the Ohio Peace Officers Training Academy made that dedication. They have headsets all over the state that's specifically used for training of peace officers. Now you're getting other organizations that are hearing about it and realizing that it's an effective use for training. So there's a group called the Forensic Nursing Network, and they want to do something, to train people about strangulation, and people being abused through strangulation, which is often invisible. They want to use it to train nurses, but that training will also be effective for police officers and emergency, EMS people. So now they're talking to OPATA, the Peace Officers Training Academy and saying, if we create the content, can we use your headsets? And it looks like they're starting to collaborate on that. So now you've got multiple agencies, EMS people, nursing and officers using the same kind of distribution infrastructure.
And so it seems to be sprouting up and then spreading. we're doing a lot of work, conversations with the state of Ohio. They have the need because they have a ton of people that they need to train, but they don't have the content and they don't have the headsets.
And so they're trying to figure out, you know, what does that look like? Do you move the headsets around, you buy everybody headsets?
Sofia Akhmanaeva
Do you work only with Ohio right now or there are some other places in the US where this kind of process gets implemented?
Jordan Herron
We did work with San Francisco, you know, a number of years ago. Current projects are mainly based in Ohio, I would say.
Eric Williams
Yep. I mean, we're starting to have conversations with people around the country, but nothing that we're working on yet. But I think it's because of that. Right? I mean, it's because people, they like the idea and that's like, oh, we have to buy hundreds of headsets.
Oh, we have to side load them all to figure out how to. And then we get people that are saying, well, yeah, we want to do what you did with the Lula Mae scenario. We wanna put it online. And it's like, okay, you can do that, but it's not going to be - it's not the same. So we're there, everyone's trying to figure it out, it feels like.
Sofia Akhmanaeva
And the last question, do you have any upcoming projects that you're excited about? Are you working on something new?
Jordan Herron
I think ongoing. I'm always excited for the hope of the next, what we're now calling seasons of the law enforcement trainings that we're making.
And aside from that, I don't know how much I can say, but we certainly have quite a number of clients that we're looking at trying to generate content for.
Eric Williams
I'm just going to say one phrase and I hope it strikes fear and excitement. Ebola outbreak. Yes.
Sofia Akhmanaeva
Well, let's finish at this point. Thank you for being with us today.
Jordan Herron
And helicopters.
Eric Williams
And helicopters.
Sofia Akhmanaeva
Oh wow. That's such an intrigue. Okay, we're going to look out for it, your next endeavors. Thank you very much.
Eric Williams
Awesome. Thank you. Yeah, this was great. Nice to talk to you. Yeah. Thank you very much.