Cultural organizations consistently question how to connect the public to information in meaningful ways. Frequently, the answer comes down to degrees of personalization and relational connectivity. Providing these can be challenging due to limited staffing, knowledge bases, language differences, access limitations, and a variety of other barriers. One solution could be to combine artificial intelligence and augmented reality in a mobile AR avatar chatbot. An emerging technology, mobile AR avatar chatbots combine these technologies' powers to create interactive and engaging tools for visitors. Examples of mobile AR avatar chatbots in cultural organizations can provide guidance for applying these evolving technologies.

The Rise of AI and AR

Understanding mobile AR avatar chatbots requires understanding their founding technologies: artificial intelligence (AI) and augmented reality (AR).

Credited to Alan Turing over seventy years ago, AI became a proper field of study in 1956. Defining AI is a frequently revisited task, resulting in a growing variety of nuanced conclusions. One of the simplest ways of understanding AI is as a technological “system’s ability to interpret external data correctly, to learn from such data, and to use those learnings to achieve specific goals and tasks through flexible adaptation.”

Essentially, AI can absorb information and apply it intelligently in different circumstances. Since its advent, AI has become a mainstay in all entertainment forms, growing from sci-fi paperbacks to blockbuster movies like “Bladerunner” to musical works like deadmau5’s “The Veldt.” While exact replications of these works are not yet commonplace, what seemed like a far-fetched future has become integrated into everyday reality.

Perhaps part of what makes AI’s incorporation into daily tasks successful is the varying levels of required human interaction. A functioning AI can perform tasks silently in the background of a project or necessitate real-time human interaction. While the imagination may connect more easily to examples like Tesla’s self-driving cars, this technology subtly exhibits itself in such functions as automatic image facial recognition, social media tagging, and digital assistant tools on one’s smartphone. On larger scales, this has led to Google’s search engine dominance, chatbots that function as teaching assistants and tutors, and entire AI-monitored and controlled data centers.

In contrast, AR is a technology that overlays added content or information into the real world. Instead of removing users from their environments, AR adds supplementary attributes to a location, “superimposing a computer-generated image on a user’s view of the real world” and creating a hybrid space that combines physical and virtual elements. Although AR has been around since the late 1960s, the rise of Pokémon Go in 2016 launched the technology into modern mainstream media. It was novel in its combination of location-based technology with augmented reality and its use of nostalgia and readily available smartphones. Importantly, this free gaming application revealed the ways mobile hybrid spaces can provide users with multi-faceted gratifications and benefits, ranging from outdoor physical activity to communal networking.

Figure 1: Pokemon GO Generation 2 overlaying AR object within a physical landscape. Source: Go Hub.

While additive technology for AR engagement exists, such as Microsoft’s HoloLens headset, the ability for engagement through a commonly-owned mobile device elevates the AR’s use and usability across societal entities and disciplines. The app’s demographics demonstrated that this kind of interactive engagement had broad-reaching applicability, including broad cross-sections of ages, socioeconomic statuses, cultures, and comfort levels with technology. It also showed how AR could successfully combine with other types of technology, such as geographic information systems (GIS).

How AI and AR Combine

AI and AR continue to evolve and merge into various aspects of society. However, they are also being combined to work in conjunction with each other. This is primarily being done in two ways.

Some AI models are being used to help create our experiences, replacing traditional building AR methods. Combining these technologies is easy due to their independent strengths and purposes. AR requires a 3D reconstruction of the world to place digital items in physical spaces, albeit virtually. This necessitates building a map of the physical world and tracking movement and space in that map. AI has been trained to build 3D maps of physical spaces, including depth, planes, and the ability to segment images for inclusion (or exclusion) as needed.

However, AI and AR can also be synchronized to make situational decisions and enhance experiences. This second method involves AI altering images, such as through filters that hide or modify images through occlusion or embellishments or through increased interactivity with AR objects. Some triggers for prompting changes are image classifications, object and dimension detection, and audio or text recognition, all using AI algorithms to instigate changes.

One exploratory technology combining these two technologies is the mobile AR avatar chatbot. Chatbots are computer programs that can have interactions with humans through audio or text-based functions. Interactions can include offering solutions to queries, completing tasks, or simply engaging in everyday conversations. Several types of programming models exist for chatbots, but AI-driven chatbots are called “predictive.” AI programming determines actions or responses, wherein the predictive chatbot can dissect language, identify message intent, and respond accordingly.

Figure 2: Screenshot of a video demonstrating a humanoid panda voice chatbot. Source: Karthikeyu Busa on YouTube.

Along with AI programming, most chatbots receive human training to produce more accurate responses. Chatbots serve as interfaces for AI, and the believability of chatbot interactions depends on the AI model. Mobile AR avatar chatbots take interactions a step further, creating avatars that are visible within the users’ physical spaces. Although the visual interface changes, the functionality of the underlying chatbot remains the same. Users can have reference questions answered, engage with contextual or historical information around literary topics, and be given way-finding assistance throughout a building, not just at a help desk. Furthermore, this tool could expand interactions beyond mere informational transactions and into relational encounters with the avatar's presence.

Figure 3: Screenshot of a video showing a humanoid avatar-based chatbot. Source: Rahul Reddy Ravipally on YouTube.

Mobile AR Avatar Chatbot Examples

The research process uncovered three different cultural organizations using mobile AR avatar chatbots. Each represents different types and sizes of cultural organizations. The company Arcade developed two of the examples. ViewAR created the third. Out of the three iterations, only one provided technology impacts.

Figure 4: A visitor using the mobile AR avatar chatbot Miss Perkins at The Ragged School Museum of East London. Source: Arcade.

The Ragged School Museum of East London

At the Ragged School Museum of East London, a historical site introducing visitors to Victorian-era free schools in London, reenactors often staff the museum, lending atmosphere and context to the space. Before the Covid-19 pandemic, the space annually hosted approximately 16,000 students and 25,000 visitors. When at full capacity, the museum runs with eleven staff members, fifteen regular volunteers, and additional interns, youth work experience placements, and special event volunteers. One additional team member is Miss Perkins, a humanoid Mobile AR Avatar Chatbot created by Arcade. As actors and staff are not always available for guided tours, Miss Perkins plays the role with visitors by quizzing them on facts students in the late 1800s would have learned. Visitors can answer her questions but do so at a risk—incorrect answers could be punished with an AR dunce cap on the user’s head.

Vienna Museum of Technology

Open since 1918, the museum covers roughly 24,000 square feet and welcomes about 420,000 visitors per year. The museum sees itself as an interface for technology, making the past and future of technology interactive, educational, and fun. In that spirit, the museum has incorporated technology in visitors’ experiences via a mobile AR avatar chatbot, visualized as a cartoon floating robot with ViewAR. Users interact with the AR avatar chatbot by being led through the museum on tours or to a specific item where the AR Avatar Chatbot serves as a docent. Part of this AR avatar chatbot’s functionality is based on Adlib, a museum collection management system. Users can also ask questions about specific display items and receive information in text, images, videos, and audio.

Figure 5: A user exchange with Roxy the Ranger, the mobile AR avatar chatbot from Sea Life London Aquarium. Source: Arcade.

Sea Life London Aquarium

Sea Life London, the largest aquarium in England and part of a fifty-aquarium consortium located in eighteen countries, garnered media attention during December 2020 for showing penguins Christmas movies. Home to over five hundred species, the institution seeks to provide entertaining engagement through enrichment processes for its inhabitants and visitors alike. One engagement effort for its over one million annual visitors has been through Roxy the Ranger, an Arcade-created mobile AR avatar chatbot. This cartoon humanoid is themed as a curator or aquarist and aims to answer young people’s questions in interactive modes. Roxy can interact with visitors throughout exhibits, show them different AR objects in her hands, and visitors can move her AR vehicles inside aquarium tanks for further exploration. On average, users spent 25% longer in the aquarium and reported higher satisfaction and perceived value of the visit when they engaged with the Roxy chatbot. Additionally, users were observed talking about Roxy with emotion and interest even when Roxy was no longer visually present.

Application Potential and Concerns

Mobile AR avatar chatbots could serve well in cultural organizations promoting knowledge and education, including libraries, museums, historical sites, and natural conservation institutions. The three examples of current inclusion in cultural organizations demonstrate different application potentials for this technology. Mobile AR avatar chatbots could serve as topic experts, provide information in guided tours, interact with additional AR objects, and draw on related media to answer questions. Essentially, this could take the famed Dalí Museum deepfake experiment and make it portable throughout a museum or library, accompanying patrons on their journeys.

This technology could also provide context to spaces, increasing immersive and interactive opportunities for visitors to obtain a unique experience with an empathetic avatar. This personalized, relational connectivity could lead to increased levels of service and satisfaction for patrons. Additionally, this could be a solution for mitigating loneliness or feelings of isolation. Research shows that humans can develop relationships with artificial entities, increasing positive quality of life.

Let us consider just one potential application scenario. Before his death, author F. Scott Fitzgerald shared 22 essential books that everyone should read. Archivists could offer an image of the originally penned list to viewers. Librarians could provide a typed version of the list with call numbers. Curators may even create a display about the event. But consider the power of an F. Scott Fitzgerald AR avatar chatbot in front of the patron. Users could ask the author questions about his life, preferences, books, literary recommendations, and even follow the author to books in a collection. Additional multimedia could be provided as needed, but the engagement would be connected to a virtual representation of the person being learned about.

While this idea has exciting potential, it is not without its concerns. Some issues with this technology involve safety. Significantly, AR has struggled with a reputation involving human error. One study showed a correlation between traffic accidents and the Pokémon Go app, showing a sharp contrast in pre- and post-app launch accidents at designated game stops. Another concern regarding AR is accessing necessary equipment. Within the United States, 81% of the population owns a smartphone. However, only a little over a third of the world’s population makes the same claim. Despite their seeming universality, depending on personally owned smartphones to engage with AR would create additional access barriers. This does not address the limited access to other AR tools, such as smart glasses or headsets. Practically, there also concerns about what this could do to job availability. The argument that technology is replacing people is common. With this device, there could be a decrease in necessary staff. A mobile AR avatar chatbot could answer questions, provide way-finding assistance, and find information for visitors, using a human only as an essential backup. As AI continues to evolve in sophistication, the chatbot would decrease regular human input and job availability.

Cultural Organizations and Mobile AR Avatar Chatbots

Public sentiments have varied during the exploration of these technologies, both by those who comprehend the concepts and by those who do not. Moreover, despite strong concerns by leading industry experts, current and forecasted applications of AI and AR indicate that the technology has the potential to significantly benefit humans across a range of fields. Even though concerns and arguments endure, one crucial fact remains: these technologies are growing. As chatbots and AR continue to emerge, cultural organizations can explore how these tools can elevate user engagement within their spaces.

+ Additional resources

“1. What Are Conversational Bots? -What Are Conversational Bots? [Book].” Accessed April 12, 2021. https://www.oreilly.com/library/view/what-are-conversational/9781492042808/ch01.html.“

Build a Dynamic Avatar-Based Mobile Chatbot Powered with AI and AR Capabilities –IBM Developer.” Accessed April 12, 2021. https://developer.ibm.com/technologies/artificial-intelligence/patterns/build-an-avatar-based-mobile-chatbot-powered-with-ai-and-ar-capabilities/.

“INDOAR Museums | AR Guided Tours & Immersive Experiences.” Accessed April 12, 2021. https://webinar.viewar.com/museum-thankyou.

“Ragged School Museum Trust Annual Account 2018.” Ragged School Museum Trustees, March 31, 2018. https://www.raggedschoolmuseum.org.uk/files/2019/02/Signed-Accounts-2017-18.pdf.

“Technisches Museum Wien -Our Mission.” Accessed April 12, 2021. https://www.technischesmuseum.at/museum/our_mission.

Android Authority. “AI and AR -Is This the Future of Mobile?,” February 1, 2017. https://www.androidauthority.com/ai-and-ar-future-745928/.

Arcade. “Heritage.” Accessed April 12, 2021. https://arcade.ltd/heritage/.

Arcade. “SEA LIFE Junior Ranger.” Accessed April 12, 2021. https://arcade.ltd/junior-ranger/.

Avatar Based Dynamic Chatbot Powered with AI and AR Capabilities. Accessed April 12, 2021. https://www.youtube.com/watch?v=NOX_uWVymco.

Baranowski, Tom. “Pokémon Go, Go, Go, Gone?” Games for Health Journal5, no. 5 (October 2016): 293–94. https://doi.org/10.1089/g4h.2016.01055.tbp.

BotCore. “Using Augmented Reality & Chatbots To Increase Customer Engagement,” February 5, 2021. https://botcore.ai/blog/augmented-reality-chatbots/.

Capgemini Worldwide. “Chatting with the Chatbots? How Intelligence Makes the Conversation Work,” May 23, 2017. https://www.capgemini.com/2017/05/chatting-with-the-chatbots-how-intelligence-makes-the/.

Capgemini Worldwide. “Integrating the Chatbot,” June 26, 2017. https://www.capgemini.com/2017/06/integrating-the-chatbot/.

Capgemini Worldwide. “Why Do We Need Chatbots?,” December 19, 2016. https://www.capgemini.com/2016/12/why-do-we-need-chatbots/.

CitrusBits. “AI+AR: How the Power Couple Fuels Unique Mobile App Experiences,” October 9, 2020. https://www.citrusbits.com/ai-ar-power-couple-fuels-unique-mobile-app-experiences/.

Goyal, Arun. “Evolving Mobile Experiences by Using AR and AI.” Octal IT Solution. Accessed April 12, 2021. ar app development, ai app development, mobile app development.

Hamari, Juho, Aqdas Malik, Johannes Koski, and Aditya Johri. “Uses and Gratifications of Pokémon Go: Why Do People Play Mobile Location-Based Augmented Reality Games?” International Journal of Human-Computer Interaction35, no. 9 (September 2019): 804–19. https://doi.org/10.1080/10447318.2018.1497115.

Harborth, David. “Unfolding Concerns about Augmented Reality Technologies: A Qualitative Analysis of User Perceptions.” Wirtschaftsinformatik 2019 Proceedings, February 28, 2019. https://aisel.aisnet.org/wi2019/track11/papers/1.

Humphries, Monica. “Penguins Watching ‘Elf’ in the London Aquarium Isn’t Just Delightful —It Could Also Keep Them Healthy.” Insider. Accessed April 12, 2021. https://www.insider.com/penguins-watch-elf-holiday-movies-london-aquarium-2020-12.

Kaplan, Andreas, and Michael Haenlein. “Siri, Siri, in My Hand: Who’s the Fairest in the Land? On the Interpretations, Illustrations, and Implications of Artificial Intelligence.” Business Horizons62, no. 1 (January 2019): 15–25. https://doi.org/10.1016/j.bushor.2018.08.004.

Lonely Planet. “Penguins Watch Christmas Movies to Prepare for First Visitors after Lockdown.” Accessed April 12, 2021. https://www.lonelyplanet.com/articles/penguins-watch-christmas-movies.

McLay, Ron. “Managing the Rise of Artificial Intelligence,” May 10, 2018. https://tech.humanrights.gov.au/sites/default/files/inline-files/100%20-%20Ron%20McLay.pdf.

Review42. “39+ Smartphone Statistics You Should Know in 2020,” November 17, 2020. https://review42.com/resources/smartphone-statistics/.

Skjuve, Marita, Asbjørn Følstad, Knut Inge Fostervold, and Petter Bae Brandtzaeg. “My Chatbot Companion -a Study of Human-Chatbot Relationships.” International Journal of Human-Computer Studies149 (May 2021): 102601. https://doi.org/10.1016/j.ijhcs.2021.102601.

Springwise. “An AR Chatbot Serves as a Tour Guide,” July 29, 2019. https://www.springwise.com/innovation/tourism/arcade-ar-chatbot-immersive-tech.

The Ragged School Museum. “School & Museum.” Accessed April 12, 2021. https://www.raggedschoolmuseum.org.uk/school-museum/.

Toole, Jameson. “Combining Artificial Intelligence and Augmented Reality in Mobile Apps.” Medium, February 11, 2020. https://heartbeat.fritz.ai/combining-artificial-intelligence-and-augmented-reality-in-mobile-apps-e0e0ad2cfddc.

ViewAR. “GuideBOT Template for AR Indoor Navigation.” Accessed April 12, 2021. https://www.viewar.com/guidebot-template/.