For our very first episode of the Art&&Code podcast series AMT Lead Researcher, Hannah Brainard, talks with Zoe Sandoval and Matthew Ragan, the creators of SudoMagic, a design and software development studio, which focuses on immersive experiences.
In this podcast, Sandoval and Ragan share how their respective backgrounds in filmmaking and performance have set them up for careers in immersive design. After meeting at a conference for Art & Technology in 2015, the collaborators combined their talents to form the design studio 2020 and work with companies such as Meta, Apple, T-Mobile, and Dell as they reimagined their physical spaces.
Sandoval and Ragan discuss TouchDesigner and other software and hardware used to create these experiences. Diving into the creative process, storytelling and participatory design are recurring elements in their work, including recent projects Calaveras (This is Not a Sugar Skull) , and Lumisphere Experience.
While artificial intelligence and machine learning have been tools in their artistic practice, Sandvol and Ragan offer predictions and ethical concerns around how these models may be trained and implemented in the future. This podcast was made in partnership with the Frank Ratchye Studio for Creative Inquiry.
transcription
Hannah Brainard 00:10
Welcome to another episode of Tech in the Arts, the podcast series of the arts management and technology laboratory. The goal of our podcast series is to exchange ideas, bring awareness and stay on top of the trends. You're listening to our Art&&Code podcast series highlighting artists working at the intersection of performance, real time visuals and live coding. This series is in partnership with the Frank-Ratchye Studio for Creative Inquiry at Carnegie Mellon University. My name is Hannah Brainard, my pronouns are she/her and I'm the Lead Researcher for AMTLab. Today, I'm joined by Zoe Sandoval and Matthew Ragan cofounders of SudoMagic, a design and software development studio, which focuses on immersive experiences. Zoe, and Matthew, thank you so much for joining us today. Just sort of to start things off, could you give a brief overview about some of the immersive technology that you use and how it became a part of your artistic practice professional life and some of the work that you do today?
Zoe Sandavol 01:05
Thank you so much for having me.
Matthew Ragan 01:06
It's wonderful to be here.
Zoe Sandavol 01:07
Yeah, it's super exciting to be here. So SudoMagic. We started SudoMagic a few years ago, actually, during the pandemic, right before right before it happened in February of 2020, which was, you know, at the time, we didn't realize what was going on. But we were actually super fortunate that we had work all throughout the pandemic, folks, were still kind of interested, even though we couldn't really go into some, you know, physical spaces, we still had work in the kind of immersive arts and technology space, which was really great.
Matthew Ragan 01:37
It was incredible. And it was interesting to see that companies use that time to kind of take the space to change their interior spaces, or plan some other ideas and kind of use that empty space to do that kind of infrastructural change, which was great. In some weird, strange turn in the universe.
Zoe Sandavol 01:59
Yeah. And we come from slightly different, but adjacent backgrounds. My background is in filmmaking. I did documentary work at UCLA. And then I got kind of shown some interactive, participatory work. And I was like, Oh, that's really cool. That's really interesting. I love the idea of audiences being able to tell their own stories to be able to be kind of participants in the work. And so I kind of started to veer into the world of theater, because that has had like, more participatory action. And that's actually how I met Matt. Matt has a background in theater.
Matthew Ragan 02:33
Yeah,I was an actor, dancer, a circus performer once upon a time, and it feels like another life. I was working in an educational outreach program in the Northeast in Keene, New Hampshire, and then actually started grad program at ASU. And that was kind of the big transition for me to start thinking about how to leverage some of the Creative Technologies that I was using inside of performance. And then eventually, that morphed into immersive experiences in themed entertainment.
Hannah Brainard 03:03
W ow, there are all sorts of ways to kind of get into this work. That's really nice.
Zoe Sandavol 03:06
Yeah, ours is well, it's like film and then theater. And then we actually met at a conference that was about arts and technology. And I was like, oh, there are other people doing this work. This was back in 2015, which was really fun. And then we kind of kept intersecting paths. And we were like, we should we should start a company. We should do.
Matthew Ragan 03:23
Yeah.
Hannah Brainard 03:25
And then it happened.
Zoe Sandavol 03:26
And then it happened.
Hannah Brainard 03:27
So I imagine I mean, even since 2015, some of the technology has changed a little bit, what's some of the go to software and hardware that you use in your work?
Zoe Sandavol 03:36
So we Funny enough, we actually met because of the software that we still use today. Because at the time, not a lot of people were using it, it's way more popular now. It's called Touch designer, it's a real time rendering environment. It's kind of like a Swiss Army knife of software's you can have connections to other software's really easily. And it's really great for kind of real time effects. And so I was really curious about it, because it was a visual programming environment. It's kind of a node based environment. And my background was in film, my a lot of family that I have are actually engineers, but I decided to kind of go a different route. So I have done like coding before. But that was really great for me because it was visual. And then Matt was actually at this conference and using the software and that's still one of the tools that we use today.
Matthew Ragan 04:23
So when I was in graduate school, I was using processing Maximus P. Quartz Composer once upon a time when that was still a thing that existed Isidora and trying to find kind of a tool toolkit that was going to work well for me, Unity hadn't quite emerged on the market yet and neither had unreal, unreal, was kind of like just on the fringe, and touch designer kind of well fit a lot of the kind of use cases that we needed. It's interesting, I think, one of the pieces that I talked about my grad program a lot and one of the pieces that we continue to kind of engage in dialogue about is the the perspective that a piece of software has so like, you know, any application that you use has a particular ideology, right? Like someone wrote it, someone has made decisions about how you ought to use it. And one of the things that's interesting about touch designer when the reasons we keep using it is that some of those decisions aren't made for you. So you are really put in a position to decide how you want a system to operate, or how you want an installation or experience to kind of unfold.
Zoe Sandavol 05:21
Yeah, and besides that, we do a lot of work using Python. We do some web based work. And that kind of changes, depending on the project. In terms of creative design tools, we've used largely the Adobe Suite. But recently, we've transitioned to using Figma a lot to just because it's super easy to translate into kind of web platforms and web frameworks. And then for our 3d modeling work, we still have been using Blender a lot.
Matthew Ragan 05:50
Yeah, in cloud infrastructure, land to use a lot of AWS services these days, that's still kind of, you know, part and parcel for how we do lots of things on the web. And then, you know, for sensors, it really depends on what project we're working on. We do a lot of prototyping with things like the Kinect or the leap. But when we go to deploy something out of the world that, you know, might be a different piece of hardware that we can actually end up using. We've used ousters, we've used kind of just traditional camera based systems. So it really depends on what project we're working on.
Hannah Brainard 06:19
Kind of jumping off of that. Could you tell us about some of the recent projects that you've worked on that you were excited about?
Matthew Ragan 06:27
Yeah, we've worked on a couple of interesting projects recently. I think one of the ones that's been really kind of top of mind for us lately, was a project called Visions 2030. That was a kind of exciting project that we got to participate in, that was on the campus of Cal Arts. And then we use that as a launching point for one of our own kind of creative projects called Calaveras, which has been really fun to work on.
Zoe Sandavol 06:53
Yeah, it was, I feel like this year was really fun, because we got to do more creative projects, versus previous years, we were kind of grinding really hard. I mean, the first few years of our business, we were, you know, we were like, oh my god, we started a company. So we were saying yes, a lot, which sometimes, you know, it is better to say no, because you don't have enough time to do your own creative work. And so it was great this year, because we could do projects that were more creatively fulfilling. like Matt said, we did a Earth Edition festival, that was by Visions 2030. And that was a festival on kind of sustainability. Thinking about eco conscious futures. That was a really interesting project, because it was one of the first times we used image generation techniques using stable diffusion, which, you know, we in the past have had kind of mixed feelings about. But that was a really lovely experience, because we got to see people's visions of the future, like what they would imagine an ecological sustainable future would look like. So that was really amazing. And then that kind of, like Matt said, brought us into the project that we did most recently, which is called Calaveras (This is not a sugar skull). And that was kind of coming off of a trip from Mexico City. It was also the Day of the Dead right around that time. And we were doing this AI project. So like, it was like all these three, three things kind of came together to create this kind of conversation piece that's in conversation with Magritte's the treachery of images. So you know, this is not a pipe. This is not a sugar skull, they were all AI generated. These weren't actually, you know, Sugar skulls. They were these kind of, like, matte likes to say the machine hallucination, right,
Matthew Ragan 08:32
and machines hallucination of what a sugar skull might look like. And so, you know, I think the we've been, I don't know the right word there, I think that I especially feel like I have a distrust of some of the kind of current trends in the use of machine learning and AI. And so it's was a great opportunity to kind of take a step back and think creatively about how we could use some of those tools, without being quite so cynical about them.
Hannah Brainard 09:01
Oh, that's really interesting. We'll have to tag some links to go view this podcast. Because just looking at some of the work, it's beautiful and impactful. And you can see pieces of you and the work that you do. i It's interesting that you bring up your visions 2030 project, because that is one that I really wanted to dive in a little bit further. So as you were setting up this project, how did you consider how the user would like take part in this experience? What were some of the conversations you had about how to make it maybe more impactful in terms of sustainability?
Zoe Sandavol 09:33
That one was really an interesting project because we were pulled on a little bit later in the process. So there was kind of some already planning and kind of thoughts around around the whole the experience of the festival because it was part of a larger festival and our section was like these three domes. So imagine there were domes that had projection in them. There were domes that had
Matthew Ragan 09:56
a digital campfire idea embraced in this psychedelic journey of
Zoe Sandavol 09:57
digital campfire idea and an AR dome was the Last one. So you'd go through this first dome, and you kind of get a primer about what the intentions are for the experience thinking about and meditating on. If you could imagine a ecological sustainable future, what would that look like? The second dome was this kind of full immersive projection where you really were like, kind of, you know, consciousness and thinking about, you know, sustainable futures. And then you came out into the third dome, which was our experience. And that was kind of a reflection point. So it was kind of like you were, you're being guided through the first two experiences as a primer for the kind of ethos of the festival. And then, in the third experience, we got to invite folks to submit their own visions for for, you know, the gallery
Matthew Ragan 10:45
The future that they wanted to imagine the interesting kind of piece about the kind of pre dome story experiences that users then visitors, users visitors strange paradigm to try and think about, left the dome experience. And then we're kind of guided out to where they could have conversations with other mentors talking about sustainability and how to take some of the ideas that they had formed and the three stage experience and then actualize them in some way.
Zoe Sandavol 11:16
Yeah, so we were kind of like the culminating point for that meditation visioning experience. And what we really had to think about was, how do we have folks submit their visions of the future, and make sure that we are creating the images that we want, especially because we had to think about, they were bringing a lot of high school and middle school kids to this festival as part of like, you know, just off field trips, and things like that. And so we had to think a lot about content curation. Because you know, you give a mouse a cookie, and they'll run wild. And so we had to think of different ways to create either like, we were like thinking, Could this be like a mad libs style? So we'll form the prompt, and they could pick certain words, and then we're like, maybe they'll pick certain words that they shouldn't pick? And then we're like, okay, yeah, what about word bank kind of selection? And so we ended up in conjunction with Visions 2030, and with their partners coming up with a series of themes. And so you would go through the user experience of giving your vision by picking themes surrounding what kind of communities do you imagine? What kind of structures do you imagine in these communities? So what are the buildings? Or what are the kind of the places that we live in look like? What is the environment? Is it? Are we like in an arid environment? Is it a jungle? Are there waterfalls? And then also, what kind of technological solutions? Because they did want to think about, you know, how can we, you know, use technology for good? How can we think about pushing more, you know, kind of clean energy technologies. And then at the end, they were allowed to pick an aesthetic, right? Which was interesting, because that allowed us to give a bit more visual curation,
Matthew Ragan 13:01
yeah, that we use that particular prompt to do an image to image translation. So that ensure that we create a kind of thematic through line through all of the images that were curated and created, the experience itself was that you would kind of come into this, like large dome environment. And there was a set of nine screens on the outside of this kind of like, circular structure, and then another nine screens on the inside. And the images as they regenerated would go kind of like full screen in this like, you know, 80 inch vertical portrait style monitor. So you'd see your vision, kind of like in high def, and then it would wrap back around behind to a gallery on the inside. So there was kind of a two moments where you saw like, one big image, and then you saw the image, in conversation with all the other images that were generated by all the participants. I think one of the really interesting challenges there was that we, you know, the conversations we had, both with the creative studio that we worked with, and with the Visions 2030 was around the kind of guarding against the impulse to just let someone kind of respond wildly front of freeform to a prompt. If you've done any image generation. One of the challenges there is that sometimes more isn't always better. Sometimes it's really about the conciseness of what you're trying to communicate. And so a lot of our efforts really went went into thinking about how do we form and structure that prompts in a way that's going to produce consistent and compelling images, and not necessarily something that's just like, wildly, kind of like both A inconsistent and B, you know, containing material that might not be conducive to an environment that is intended for families?
Hannah Brainard 14:51
Wow, it sounds like you found a really impactful way to help people tell part of their own story, but maintaining kind of a cohesive story through the project. So I wanted to ask you in the past, you worked with UCLA Center for Research and Engineering, media and performance, working, particularly with storytelling initiatives. So could you tell me a little bit more why storytelling is important to you? And how you've used it to tell some of your personal story?
Zoe Sandavol 15:15
Yeah. So, I was exposed to REMAP Well, I was in my undergrad. And then they had gotten a grant from Google to look at the future of storytelling. And at the time, we were given Google Glass. And we're given the opportunity to explore different ways to use glass for narrative and storytelling purposes. And that was really interesting, because we all got to kind of pitch different ideas. And one of the projects that we ended up doing was using Google Glass as a way to do kind of like a live improv performance. And so actors would wear Google Glass, and then the director would using sentiment analysis kind of guide what they should say, based off of the kind of effect. And so it was within the scene. So the director wasn't really intervening verbally, but using kind of like an iPad interface, they would tell them to, you know, be more antagonistic or be more soft and delicate, and then giving them sometimes specific things to say. So I feel like my opportunity working with remap was incredible, because I was exploring different ways to experiment with storytelling. And I've always just initiated the ability to create interactive storytelling environments, because then more folks can tell their stories, while I was at UCLA did a documentary, undergrad focus. So I've always loved kind of doing work that's centered around real people's stories. Even though I love magical realism and fantasy and bright colors. I've always been more drawn to real people's stories. And so while I was there, I did a documentary on my grandparents immigration to the United States from Mexico, which was really nice, because I got to shoot it on 16 millimeter film, we were like the last class that they had do that, and the last class where we got to use the film editing beds. So that was really amazing. And then in my MFA, I also did work exploring my grandfather's archive on my Venezuelan side. So he was a photojournalist in Venezuela. And for a long time, I knew there was an archive of photographs. But I think my dad was hesitant, I don't know who there were all film negatives. So he's very, like, wary of anyone touching them and opening in them. But eventually, I kind of convinced him I was a commodity each time I want to see them. And so we went through a process of scanning 1000s 10s of 1000s of negatives. And I use that as the basis for my thesis, which was an immersive environment, kind of a love letter to Venezuela to Venezuelan diaspora thinking about places that we can no longer return to thinking about kind of memory, love and longing. And so yeah, I've always, you know, been really passionate about sharing this kind of personal stories, sharing intergenerational kind of stories, I think it's super valuable, you know, stories are the ways that we connect, you know, as part of the human experience. So, you know, I just love it.
Hannah Brainard 18:13
That's really beautiful. Matthew, sort of switching gears. So prior to working with Sudomagic, you were part of the software team with Obscura Digital, which was later obtained by Madison Square Garden, and you worked with spear project? That's correct. Yeah. Could you say a little bit more about your experience there? And maybe some of the differences working with a large company like that versus kind of working at your own company?
Matthew Ragan 18:39
Yeah, definitely. You know, it was interesting, I joined obscure digital as an interactive engineer. So I was kind of, you know, a code monkey building the infrastructure that drove a lot of our experiences in installations. And in the transition to Madison Square Garden, part of the kind of arc of that was trying to understand what that venue was going to need, what ideal kind of software tooling, and kind of perspectives that you might kind of embed inside of a studio that was gonna stand up an experience like that. I actually worked at MSG as the director of real time content generation. And so my team was really focused on how to think about using real time tools and environments to generate content that was going to be on the exterior interior, screens for MSG sphere. And I don't know how much of that kind of made its way through the whole pipeline. But what was interesting about that, is that, you know, in architectural environments, and then in any big company, a board, part of the conversation that you always have to engage in is really educational about why is this important? What's the impact of this tooling and perspective and why will it be something that's going to be a critical technology to include in this you know, multi billion dollar venue that's being constructed. And I think the interesting part for me there is that a lot of that same kind of education and conversation finds its way into your own studio. A lot of times we spend working with clients are about trying to tease out what it is that they're trying to create. I mean, sometimes we'll we'll get a brief or requested say, you know, kind of an SFW that's outlined, like, it's XYZ, like, this is the thing that you're going to build, which is great. When that happens, I think, more often than not, we really have entered into and more of a conversation about an idea that someone wants to kind of bring to life. And then part of our job is to try and tease out okay, well, does that mean that you need us to build software? Does that mean that we need to assemble off the shelf pieces? Like, what is it that you're trying to actually achieve? And then also, if you're trying to achieve something new, or really novel or interesting, why is part of the process going to take a long time? Or why do we need to do a phase of research to understand the implications of, you know, this particular decision that we're making. And so I think that kind of education and conversational piece is really similar, you know, being inside of a large organization, and then running your own studio space at the same time. There are some big differences. Also, I think that, you know, we make our own schedules, which is very different than when you work inside of a large organization. And we're also empowered to kind of make choices much more quickly and more nimbly than you might when you're inside of a kind of large organization, which is a blessing and a curse, I suppose at the same time. So it really, I think there's a lot of pieces like gained from working inside of both obscure digital and at Madison Square Garden, that have carried over as we've started our own studio.
Hannah Brainard 21:49
That's great. It's interesting that you talked about education, because diving into your backgrounds, it seems like education is a piece of everything you do both in helping students learn how to take part in this, like you're here at Carnegie Mellon this week, but also constantly learning to be able to help serve your clients well. So as you kind of learned about the current environment of technology, you talked about AI earlier, what are some of the emerging technologies that you're excited about? That you're diving into some that you might be a little more skeptical of?
Zoe Sandavol 22:19
You know, I feel like AI is is one that answers both of those questions, in a way, just because I for a while, I mean, we've been doing work, you know, in our own careers using machine learning and image recognition for a long time, when it was more rudimentary, and then now we're, it's a lot more accessible. And I think, you know, in the past few years, I felt a bit of like, feeling of like an ostrich with its head in the sand, because I'm just like, I can't with all this going on, like, it's just, it's overwhelming. It's the direction that it's going is frightening, it's concerning. But then we were given this opportunity with Visions 2030. And it kind of like gave me a new perspective on it. I'm like, you know, what, if we're the people who have the capacity to drive a more creative vision and thinking about, you know, how can we use? I don't even want to, it's not really AI for good, but how can we just think of different ways to use these applications that aren't militarized that aren't, you know, used in ways to create misinformation. And so it was a really great opportunity, if like, if we're not the people using it, then then only some other people will be using it. And then kind of what will happen, it'll continue to veer in a direction that, you know, I find particularly frightening. And so I think image generation is really interesting. I saw some work that was about, they were images of kind of a liberated future. And like, what would that look like for a group of people? And it was funny, because just last week, I was telling Matthew, I was like, my dad was seeing the Calaveras project that we didn't, he was like, Oh, that's so cool. And he's like, that's amazing. He's like, How'd you do that? And I was like, Well, it's actually not even it's not that hard nowadays. And he was like, Oh, that's so cool. He was like, what if we could imagine? He was like, thinking about my thesis project? He's like, what if we could imagine Venezuela now, but like, as what it used to be like, what if we asked people in diaspora, what would they imagine if they could go back? What would it look like to them? And I was like, oh, that's actually really that's really interesting. You know, it's like, that's really nice. So
Matthew Ragan 24:26
Yeah, I think to answer your question, I think there's there's a unavoidable future at this point with the kind of like momentum and trajectory of technologies, we start to think about what the impacts of machine learning and AI inside of the tools that we use. I think the there's a lot of really interesting enthusiasm, and it's hard not to be enthusiastic about what that's going to unlock for kind of semantic interaction with computing. I think a lot of the pieces that we're starting to see now. Really think really kind of am As a way from screen based interactions, and instead point is toward language based interactions, which is, you know, the future of Star Trek that every, you know, 30 and 40 year old kind of was exposed to on broadcast television. I think the parts that I have the most resistance and skepticism about having worked with technology now and having been a developer are kind of appreciating and recognizing that any of these large systems, any AI system is really at its heart still kind of embedded in some of the surveillance technologies that permeates the web, right. Like we can only have these systems that are predictive, because we'd have extracted so much information from people, either through their email accounts or through their social media accounts. And through all of these kinds of transactions and interactions that weren't necessarily consensual. They were consensual in that I clicked the I Agree box, you're allowed to collect my data. But there is a part of that, that that feels really disingenuous to me, and is also historically tremendously biased towards smaller subsets of the population in terms of who represents who it's good for, and who benefits from it. So, to me, the kind of techno utopian fantasy about how we're going to be liberated from our responsibilities to technology kind of turns a blind eye to some of the trends that are necessary for how we reinforce and train these big models and who's represented and who shows up in the images that we generate. So, you know, I think there's, there's a interesting enthusiasm, but I don't feel like that is necessarily a future that includes everybody that feels like it's in a future that is like, for a subset of the population that isn't necessarily about kind of creating an equal opportunity future. And so I just have a lot of crunchy feelings about that. And don't necessarily feel like it is something that I can enthusiastically say, like, Oh, my God, it's going to be so great. Like, Well, sure. It's going to be great for some people. But it's not going to be great for everybody.
Zoe Sandavol 27:27
Yeah, we just, we just got back from Mexico City. And it was really interesting. When we arrived in the United States. You know, you're in the passport, you're in customs, waiting with your passport to like, give it to the person and you arrive up, you get to the desk and the controller is like, oh, stand in front of this so I can take a picture of you. And you're like, Okay, great. So they take a picture of you. And they go Oh, hi. I hadn't even given them my passport yet. I was still holding it. I think it was Hi, Zoey.
Matthew Ragan 27:52
Welcome home.
Zoe Sandavol 27:52
Welcome home. And I was like, sir, excuse me. And then I was like, what? And then Matt goes through, and they go, Hi, Matthew. And then like, we're leaving. And Matt's like, I was not prepared for them to say my name. And say hello to me, without me giving them any identification. And so you know, don't love that. Don't love that. I feel like that's something that's, you know, it's been prevalent for a long time. Right. You know, I mean, I feel like London is like one of those cities that they say it's like, it's a, you know, there's a lot of surveillance generally. And so, yeah, it's that I feel like it's just as concerning, you know, because our, our freeness our, like, in the world is, like, constantly being surveilled. We know, we know, it's prevalent online, but now it's like, it's everywhere. It's like oh jeez. Okay.
Matthew Ragan 28:44
Yeah, it's, it's interesting, because I think we've, you know, in my graduate school experience, I made a bunch of materials for people to use, other people want to kind of work their way into this industry. And one of the strange kind of realizations, you know, in the past year and a half, since so much kind of like GPT stuff has blown up in the media is kind of reflecting on the fact that I really, I think the audience that I was writing for was other humans, other people. And so much of the content on the web is now just kind of being swallowed by machines. It's like, No, you know, text for machines to be used in other machine processes. And so, I don't know, that part still feels like a little bit weird to me. It feels like it's a less human centric kind of modality. And I, yeah, I'm just wary.
Hannah Brainard 29:37
That's understandable. I mean, it can open doors for new kinds of artistic expression, but at the same time, there's still consideration needed about who has access and what that means really. Well, thank you so much for taking a little time today. It was great to talk with you. Thanks for being at CMU.
Zoe Sandavol 29:55
Yes, it's really nice. Yeah, it's beautiful. It's a beautiful campus. The weather is amazing. It's cold. We are from California and we're like, oh my god, it's cold. This is amazing. So we're super, super happy to be here.
Matthew Ragan 29:57
Great to be here.
Hannah Brainard 30:05
You're the only two people in Pittsburgh saying that.
Zoe Sandavol 30:08
Thanks so much.
Hannah Brainard 30:12
Thank you for listening to the art and code series on tech in the arts. Be on the lookout for new episodes coming to you very soon. If you found this episode, informative, educational or inspirational, be sure to send this to another arts or technology aficionado in your life. You can let us know what you think of this podcast by visiting our website amt-lab.org That's AMT dash La b dot O R G. Or you can email us at ang lab cmu@gmail.com. Follow us on Instagram at Tech in the arts or Facebook and LinkedIn at arts management and technology lab. We'll see you next time.