Immersive Storytelling: Redefining Narratives in the Age of AI
By Clone Wen < cw4295@nyu.edu >
On December 3, Maya Georgieva, Senior Director of the Innovation Centre at XR AI and Quantum Labs at the New School, delivered her talk on Immersive Storytelling in the Age of AI, moderated by CTW (Catch the Wave), a society that brings together artists, creatives, and philanthropists.
The talk dives into the realm of immersive experiences, blending narrative structures, and empowering creators to redefine storytelling in the age of advanced technology. Even itself is using AI-generated pictures for PowerPoint slides. While VR and AR transcend conventional storytelling by breaking the fourth wall, transforming passives into active participants, and inviting audiences to shape the narrative, AIGC breaks down the barriers between creators and gamers and is leading us to an era where audiences are starting to create games and movies.
Maya Georgieva starts by introducing her tenure at NYU. Georgieva witnessed the advent of groundbreaking technologies, from the first iPad era to XR and VR. Her journey then led her to Parsons School of Design’s Innovation Center. Over the past six years at Parsons, Georgieva has been instrumental in shaping the Innovation Centre at Parsons, which has expanded to include XR, AI, and Quantum Labs, each dedicated to exploring the frontiers of technology and its potential for storytelling.
The Rise of Virtual Narratives
With the creation of the Immersive Storytelling Class and Immersive Storytelling Festival, it became a platform for students to exhibit their projects, ranging from narrative game design to environmental advocacy experiences. Students make their own avatars and share their virtual embodiments in Unity and Metaverse.
The talk highlighted the dynamic capabilities of AI-driven storytelling, showcasing various projects that leverage AI tools like Blender, Unity, and Maya to craft immersive experiences. Showcases of student game projects like “Picker’s Planet” and “The Life of Bees” illustrate the depth of storytelling possibilities in immersive mediums. Notably, “The Life of Bees”, as an interactive documentary and game, presents an advocacy experience centred on environmental impact and climate change. This immersive journey takes participants into the plight of bees, demonstrating the power of storytelling to evoke empathy and environmental awareness. The intricate design process involved in creating this experience, including the rigging of characters like bees, underscores the complexity of AI-driven narrative construction. The fruitful class has introduced the students to various positions like digital anthropologist, AR playground engineer, HCI designer, digital idol producer, digital fabrication technician, etc.
AGI: Generative Agents
The talk delved deeper into the prospect of AGI (artificial general intelligence), envisioning a future where thousands of agents coexist in a world, contributing to a form of collective intelligence transcending mere generative AI. The exponential learning and creation potential of AGI pose profound implications, raising questions about the evolving role of human experience, learning, and creation.
The game of generative agents stands at the forefront of AI game designs, as highlighted in the talk as autonomous entities within simulations or digital environments. Unlike traditional AI tools programmed for specific tasks, generative agents possess a degree of autonomy and creative decision-making capabilities within their defined spaces.
Maya’s discussion showcased a scene where generative agents woke up within a simulated environment, engaging in a myriad of activities without explicit directions. These agents represent a departure from traditional AI coding paradigms, relying on generative algorithms that prompt these entities to live, interact, create, and learn within a simulated world. Their behaviour reflects human-like qualities, from performing routine tasks like work and breakfast to engaging in conversations, inviting interactions, and even exercising agency by refusing invitations or social interactions. This design of generative agents introduces a new paradigm where AI entities operate and interact in ways that mimic human behaviour, albeit with their unique digital autonomy. The idea of placing these generative agents within a simulated world, allowing them to operate with minimal direction and instead fostering emergent behaviour, stands as a pioneering approach beyond coding NPC but generative NPC in AI game design.
Unity Muse, Pika AI and Deepfakes
On a more constructive note, the talk emphasised the most recent empowering potential of AIGC tools like Muse AI from Unity and the new Pika AI in reshaping storytelling. These AI-driven platforms enable creators to generate and edit videos using prompts and AI-generated content. This transformational phase encompasses multifaceted processes: world-building, 3D modelling, asset creation, interactive and experience design, and even motion capture, which is already accessible via tools like iPads.
The talk raised pertinent concerns regarding the proliferation of deepfakes, illustrating the ease with which these AI-generated, manipulated videos can be created. The ability to seamlessly superimpose one person’s likeness onto another’s, alter facial expressions, and mimic voices has raised ethical concerns about misinformation and authenticity in storytelling. It allows individuals to create hyper-realistic videos that could potentially deceive viewers. As highlighted in the talk, the simplicity with which these deepfakes can be generated, even by novices, raises concerns about the spread of misinformation and its implications for reality and truth.
Blade Runner 2049
Maya finished her talk with a clip from Villeneuve’s Blade Runner 2049: The video referred to a profound moment where the replicant, on the last stop of finding his identity, is a memory designer, whose role is to create memories and experiences for generations of replicants and non-human entities, granting them a semblance of a childhood or a unique past.
This scene encapsulates the idea of memory designers actively crafting experiences and memories for entities that never had a traditional childhood or formative years. It prompts a reflection on how memories are designed and shaped, serving as a catalyst for contemplating the convergence of human creativity with technological advancements. It foretells a future where technology becomes the conduit for a deeper, more immersive human experience. It envisions a realm beyond the metaverse or existing gaming platforms like Roblox or Fortnite—a realm where our interactions with technology redefine the very essence of what it means to live and experience life.
Challenges and Democratization of AI in filmmaking
The after-talk conversation touches upon how AI and technology could revolutionise filmmaking, allowing individuals to create films from inception to completion. While Hollywood has experienced strikes by the SAG-AFTER and Writer’s Guild’s strikes over AI, the democratisation of the industry could potentially empower every actor and director to create movies without the constraints of traditional studios. It’s a seismic shift that might redefine authorship in film and push the industry towards a new creative paradigm. It could disrupt the industry by altering the nature of celebrity endorsements, image rights, and the creation of movies without direct involvement from the actual actors. This raises ethical questions about the use of these digital likenesses and the control actors have over their identities in the digital realm.
However, this radical transformation still sparks ethical concerns. The talk highlights how technology disrupts professions. Technological advancements can reshape the filmmaking and game design industries, similar to how personal computers revolutionised print shops and magazine offices in the 1980s and 1990s in Chelsea, NY. The fear is that AI might lead to job displacement. This technological shift challenges ethical boundaries and confronts the need for regulation to prevent misuse and protect individuals’ control over their digital likeness.
Expressive AI 