In this episode of Tech in the Arts, AMT Lab’s Social Media Manager, Sofia Akhmanaeva sits down with Amber Johnson, an Associate Professor of Simulation and Game Development at Wake Tech Community College. A presenter at Carnegie Mellon’s SONA Immersive Storytelling Festival, Johnson shares about the technology and inspiration behind her work.
Johnson has worked in the game industry as an artist and level designer since 2007. She began teaching at Wake Tech in 2014, developing a curriculum that utilizes modern game asset creation techniques. She currently teaches courses in digital art, game programming, photogrammetry, tech art, and VR. On top of teaching, she does freelance work in photogrammetry and 3D renderings.
At SONA, Amber presented her project “Crafting the Unreal,” which reimagines Edgar Allan Poe’s Ligeia as an immersive surrealist stop-motion VR experience by combining the tactile, dreamlike aesthetics of traditional stop-motion animation with the interactivity of VR. Users begin by interacting with a physical cabinet containing stop-motion-inspired props, which they manipulate to unlock story elements. Through photogrammetry and game engine technologies, these physical objects transition seamlessly into the virtual space. In VR, users embody a character of the story, exploring surreal environments that reflect the eerie and tragic love story of Ligeia.
Artists and Works mentioned:
Sofia Akhmanaeva
Welcome to Tech and the Arts, the podcast of the Arts Management and Technology Lab at Carnegie Mellon University, or, for short, AMT Lab. I'm Sofia Akhmanaeva, marketing manager at AMT Lab, and today our guest is Amber Johnson. One of the artist participants of SONA Immersive Storytelling Festival at Carnegie Mellon University.
She's an interactive media creator, specializing in virtual environment simulation and game programming. She's also an associate professor at Wake Tech Community College in Rally North Carolina. She's an art director for Flip Axis Studio, and has worked as a lead artist and level designer for a number of game development studios. Today we're gonna talk about Amber's artistic path and career path, and also about the perspective for VR games, experiences in market in general. Hello, Amber.
Amber Johnson
Hello. Thanks for having me.
Sofia Akhmanaeva
Yeah, it's very nice to talk to you today. So, I would ask you first about your experience and the path. How did you come to work with VR and where did all it start?
Amber Johnson
Sure, it was a long and winding path. I think that's kind of common for lots of creative people. But I did my undergrad in media arts and animation and I took a few classes in creating game art in [VR] and I really enjoyed it. I've always been a gamer since I was really young, and being able to create those interactive experiences really felt more meaningful to me than just like a passive film. So after graduation I went into the industry working as an environment artist and a level designer at several different studios. VR wasn't quite a thing back then. You only saw it maybe at like Disney World at one of those specialty experiences because it was those big, bulky headsets and no one was really working on it for home use yet. So, I worked there for a while, in some studios and eventually I did end up with my teaching position. And one of the things I do enjoy about my teaching position is I get a lot of freedom to experiment and try new technologies.
So they actually purchased some - I think they were like the Rift back then, Like the really first Quest, before they were called Oculus Quests - for us to try to develop with so that we could teach it to our students because we always try to teach our students the most cutting edge stuff.
So I got to play around with it and, you know. It's interesting and I really enjoyed it. It is definitely a lot better than what I played with at Disney World. And, I ended up being able to also get contract work doing it. 'cause one of, one of my jobs was not with flip access. That's, you know, some, that's a really new one.
But previous studio was working on creating a VR remake of Colossal Cave Adventure, the original seventies adventure game, but in a VR experience. Because my original studio work was a lot, was mostly in mobile games and like, you know, for like the iPad and stuff like that. So the technology translates really well before, like the technical needs for that.
So I started with that and kind of kept going. And when I started my master's at NC State, I wanted to push that immersive experience with what I was doing. So it just kind of kept with the VR.
Sofia Akhmanaeva
Okay. So you transferred to VR experiences before you started your masters?
Amber Johnson
Yes. Got familiar with it as a professor at the community college. Took some, contract work, making VR games. And then when I started my master's about a year ago, when I was choosing my final project, I decided to do it using VR as the medium.
Sofia Akhmanaeva
Speaking of kind of work that you're doing right now, I just attended your “Crafting the Unreal” experience, for SONA Festival and it's definitely much more artistic than just some kind of games that are purely for entertainment. Could you speak a little bit more about why and how you switched to this kind of project from just commercial gaming?
Amber Johnson
Sure. So it's funny because I was actually not, I don't wanna say ranting to my, to like my classmates about it, but I'm usually the person that is like the tech art person.
Like I'm the one that figures out like the technology. I'm the one that has to be very, you know, do all that really hard stuff and like make the art do the stuff. And I'm actually a very artistic person, but I never get to express it. So, with my master's degree, part of like the idea behind it is to do experimentation with art.
So I decided to just like finally have an opportunity to go for it and be artistic. Because if I had free time, maybe after my master's, I would like to create more interesting artistic things with the game technology that I'm using currently.
Sofia Akhmanaeva
Okay. So you were kind of blending two worlds?
Amber Johnson
Yeah. But everyone, no one else can do the tough stuff but me, so I always get thrown the tough technical things rather than the fun art stuff.
Sofia Akhmanaeva
You have expertise among all the different things in photogrammetry that you particularly use for this work that you presented at SONA. Could you speak a little bit more about that and like what was particular challenge for this experience? Yeah, and maybe give some introduction to what this experience is for our listeners.
Amber Johnson
Okay, sure. So, if you don't know what photogrammetry is, it's a technology that's been around for actually quite a lot of long time, but it's kind of newer to games.
The idea behind it is you can digitize real world assets and places by taking hundreds of photos of it. The software, there's specialty software, that can take the photos you have taken and spit out a 3D model for you. It is really high end, like it's millions of polys. And if you know anything about games, you know that a game engine can't handle millions and millions of polys yet, especially VR. So you do have to clean it up.
Sofia Akhmanaeva
Excuse me. What is poly?
Amber Johnson
Polygons. I sometimes I forget to…Polygons like the triangles that make up 3D models. Usually you don't want to push it too hard, especially in VR. Like you can only maybe do a couple hundred thousand for the whole entire thing.
And then when a model is like 20 million polys, like that's not going to run very well in the VR headset, so you have to clean it up a little bit. But ultimately, what it does is it gives you a very realistic looking model, that you, it would take an extremely long time to achieve. If you try to make it the old fashioned 3D modeling way with pushing and pulling vertices and creating textures like it's, it's a significant amount of savings.
Sofia Akhmanaeva
Oh, wow. So people generally use it for VR experiences. It's like mainstream technology or not?
Amber Johnson
I wouldn't say for VR, but for video games in general, yes, it is very common to now use. Sometimes they're called photo scanned assets, or photogrammetry. If you are familiar with like the Unreal Engine. That's one of the major game engines out there.
They have a whole, a Quick Soul asset store that's just photogrammetry assets that you can purchase and put into your games that are like rocks and chairs and anything you can, you know, think of. So it's getting pretty common to use photogrammetry assets in games, but it can be tricky for VR, since VR has a lot less processing power than like a computer or like a PlayStation V.
Sofia Akhmanaeva
Let's get back to the work that you presented. Could you speak a bit about what it is, what it is about? Okay.
Amber Johnson
So, what I wanted to do is to showcase how games and game technology can be considered art. And one of the ways that this can be done is through remediation where you take an older art form and remediate into a newer art form.
So I am turning surrealist stop motion animation, taking the aesthetics of that, and putting it into a VR interactive experience. Specifically if you are familiar with surrealist stop motion animation, I am looking at like Brothers Quay and like Jan Švankmajer. For their aesthetics of what I built and captured to bring it over into that.
So I've been documenting my workflow of how I went through the whole pipeline of pre-production, production, capturing the assets, getting them into the game engine and like best practices because there was a lot of learning. I tried something one way and it's like, well that was a huge pain to do it that way. I probably shouldn't capture it that way again. Now that I finally have my assets, I'm turning it into telling the story of Poe’s Ligeia through some puppets that are digitized that are going to be animating, that you'll be able to, watch interact with each other, and you'll be able to do some slight interactions to help move the story along. But I, I'm, they're not quite in there yet. They'll be in there very soon.
Sofia Akhmanaeva:
Will this work be available anywhere on the web?
Amber Johnson
Yes, I have a dev blog I'm doing. You have to interact with the physical set, because part of it is, you have to physically interact with the set, because it's like a puppet set that you have to move around, interact with it.
But I do want to make a digital version where you don't need to do that. It'll kind of take away a little bit interaction because I have some interesting story ideas, but I think I'm gonna find a way to make it feel close to that. But I wanna put it up on a site like itch.io, which is free to download so that other people can try it out just for free.
Sofia Akhmanaeva
We'll put references for a lot of things that we're mentioning during this podcast, including, Amber's experience, “Crafting the Unreal.” You are gonna be able to check it out and be more aware of the context of the conversation.
Let's talk about the artistic inspiration of this work. You said that it's based on the, Edgar Allan Poe's novel. What was why this story and what was also behind the idea of merging stop motion technique, which is very real world connected, with something that is virtual reality.
Amber Johnson
Yeah. This was actually a really long thought process of, I needed like, a vehicle to do this. So like, what story should I do? What would be thematically appropriate for this style? So I kind of looked at the stories that the Brothers Quay and, Jan Švankmajer told, as well as other people who are inspired by them.
And it's very, 18th century and kind of gothic feeling. One of Jans Švankmajer's, more well-known movies is - he retold Alice from Alice in Wonderland, and the whole aesthetic is the sets, the puppets and everything are created from found objects. So it kind of brings, you know how there's always the joke that children play with the box, they don't play with the toy.
The whole like, idea behind this aesthetic is like, it almost brings like, a child nostalgia because it's stuff created from just junk. You find like you're a kid again. So it has like this real like appeal to people. So, one of the things that always also comes up with this is like, you know, again, like the 1800s and the idea of like a curio cabinet.
So, I built my thing into a shelving unit and where my next step is to actually build like doors so it can open up like a cabinet, so that it kind of feels like you're opening a cabinet of curiosities. I chose Poe’s Ligeia specifically because it is thematically, like the time period, accurate to how a lot of these stories are for the style.
And I like, if you read the story, it talks a lot about like the soul and like some weird, like metaphysical stuff. And that's a very common theme with surrealist art in general. So using this kind of story that goes into like those kind of ideas for a surrealist stop motion animation felt correct out of all the other stories that could come with, but yeah, it's like a long winding road. But I did put a lot of thought into like, this is the story that I need to tell.
Sofia Akhmanaeva
I can confirm that it feels very unique because you're sitting in front of this dresser, basically, which is a dollhouse too. And it actually provides this unique feeling of, “oh, I'm inside a doll house and this is something that you always, at least I wanted to do it when I was a kid and it was like, “God, it's so cute, and like, I want to be inside that.”
Amber Johnson:
Yeah. The idea is you get to move a, I call it the “wooden homunculus” because it's like an alchemy kind of theme to it too, and that your soul can go into the wooden homunculus, which just means that you can see in the VR headset what the wooden guy can see on the shelf. And there's stuff that you can't see in real life that he can see, like the puppets moving around and some other things like there's like a candle that's not actually lit in the real world because you can't do that, but there's like a lit candle and stuff like that.
Sofia Akhmanaeva
Yeah, and I think this is the unique part of this work because most of VR experiences, you just put headset on and something appears in front of you that is not connected in any way to the reality except for the space that you have to operate in still. But here you, you see the object then boom, and you're inside of it and it's altered and it's weirdly looking more real through VR than in life. Yeah. I'm going to keep digging deeper into the conceptual part. I'm just curious, what is the bigger context for this particular idea? Philosophically, almost, what is it about - merging and connecting VR world to the real world? Because usually people kind of use VR to escape the real world. Is there some like deeper meaning, emotional meaning?
Amber Johnson
I feel it is kind of weird, and this might be another weird, like roundabout answer. A lot of the world is becoming very virtual. You know, after COVID, especially, like we had virtual classes and everything's moving online and it's not like real.
And people I think innately miss some kind of real tactile thing that's in front of them that's not virtual. And I thought it would be interesting to bring the tactileness of this specific style of animation. Like if you ever watch like this, those artists, like, it's very tactile. You almost feel like you can touch the stuff because it's made of stuff that you encounter every day. And just kinda like bringing that like, hey, even if we're doing virtual stuff, we can still have like real tactile stuff to go with it, if that kind of makes sense. Because I think having that. It just kind of helps you feel more immersed in the experience.
Sofia Akhmanaeva
I know this work is also for your Master’s thesis project, but do you have something else in mind already?
Amber Johnson
I have to finish what I have by the end of the month and I don't think I'm going to have as much in there as I would be happy with. Because I would like to do more. While the story will be done, and you'll be able to go through the whole experience, I'd like to add smaller, almost like animated vignettes in there where it's not necessarily tied to the story, but it's kind of more of like an interesting little animation thing that happens.
Currently, the best example would be, in the first shelf there's a little wooden bowl and there's like a rusty nail that kind of swirls around and does weird animations in it. But I'd like to maybe do more of that, like, just to kind of have more stuff to experience in it. I want to try to push more of some of the electronics I have going on right now, it's just some simple lights that turn on based on how you interact with it. But I'd like to possibly add some more like lighting and more of the physical changes that happen when you like, turn a shelf on and stuff. But the whole thing is actually built in a way that if I decided I don't want to do this story anymore, I can take all of my shelves out of it and build new shelves and put them in, because they slide in and out. So I could always just use it as like a, you know, 3x story.
Sofia Akhmanaeva
You have some new projects from the gaming side. Is it connected somehow to the artistic part of your work or it's gaming again?
Amber Johnson
So as far as my non-teaching stuff, I'm working currently on a NDA project that I can't talk about, but they might have just talked a little bit about it at GDC, but I haven't talked to the director yet. But it's a VR game, I think I can say that at least. That one's kind of fun, but I'm definitely just like an art person on that one. Most of my work I get outside of teaching is actually doing photogrammetry. I've worked with the North Carolina Museum of Art, to help capture some of their art assets, to help them create a learning experience for middle school students.
So like you can see, you read about the art piece and then you can see in 3D that art piece. So like, if you can't go to the museum, you can see the assets. Like, you can turn it around, you can touch it like in 3D. It's not VR, but I don't think they ever turned that one into a VR experience. But, you know, it's still like using the game technology in a way that's not like an entertainment game.
Sofia Akhmanaeva
That's cool. What, in your opinion, is the perspective for VR market? What is its future? These technologies that are relevant for VR, how could they be used in other spheres in arts or beyond?
Amber Johnson
I think VR might finally be here to stay, like I think I mentioned earlier the Disney World experience, but, I know the Quest 2, these numbers might be slightly different, but 20 million Quest 2 headsets, like the console basically for your head had been sold, and I think a million of the latest Quest 3s. I'm not sure about the other systems because I only really developed for the Quest, but that's a significant number of people who have them in their house now.
There's a lot of really good games out there that are really pushing the immersiveness and, I never, several years ago, I never would've spent maybe $30 on a VR game, but I have now because I really wanted to play it and looked really good using some of the newer technologies. There's a lot. I feel like it's a technology that is going to keep getting better, I think once they kind of solve the issue of the headsets being heavy on your head, because I know personally, I have a hard time playing too long because it pushes on my head and it can give you a headache after a while. I think once they solve that, I think we'll see it a lot, even more widespread. And you know, there's a lot of uses for it. I know they do have it being used in all it is called the “serious games” sphere. It's games that are not meant for entertainment. It's meant for like, like your flight sims and like your training sims. I've even seen and read about VR being used in psychology to help, like combat veterans overcome PTSD and stuff like that. So yeah, there's a lot of interesting use cases for that kind of immersion. But I think, as the technology gets better, we're just going to see it spreading.
Sofia Akhmanaeva
That's very positive perspective on it. I think not all people adhere to this point of view, do you think artistic works will blend with entertainment games in this space?
Amber Johnson
I know specifically in the art space right now, especially in digital art, there's the whole AI art thing going on, and I feel that being able to be more artistic with your art especially in VR, is going to be one of the ways that artists kind of separate themselves from the AI trash that's going to come, because they can really push an artistic experience and not just generic person walking into generic room. Like they can really feel the art. Kind of like how when photography was first invented, they thought no one would ever get their portraits painted again. But there still is portrait painters and there are still artists who paint. It's just, you know, it's more of an artistic expression versus a very realistic representation.
Sofia Akhmanaeva
How high is the entry barrier for artists to work with VR? It seems like it takes a lot of resources,
Amber Johnson
to start developing.
Sofia Akhmanaeva
Yeah. Start switching to this art form.
Amber Johnson
So at the community college I teach at is a two-year degree and I teach my students how to create VR environments by their third semester.
So, you get the intro the first semester. The second semester, they learn 3D modeling. And then by the third semester, they can take an environment art class with me and the final project is to create an interior space emulating a specific architecture style that I tell them. And they do a good job. Like I've gotten some really great work out of them. And I think, if you have the drive, you can do it. There's a lot of free learning resources out there and a lot of the professional technology is free for people just to try.
Sofia Akhmanaeva
That gives hope for a better future of VR, for better artistic experiences.
Amber Johnson
The more people out there who can express themselves artistically, the better.
Sofia Akhmanaeva
Thank you very much, Amber, for being with us today.
Amber Johnson
Thank you for taking time to talk to me.
Sofia Akhmanaeva
Yeah. I hope you have a great experience at the rest of SONA.
Amber Johnson
Thank you.