Newsletters > Immersed in The Future

WolfBrown’s new community of practice is designed to support nonprofit arts organizations as they navigate the complex programmatic space of augmented, immersive, and virtual experiences. We hope you find Immersed in the Future rewarding and invite you to join in!

The Future of Augmented Experiences in Theaters and Concert Halls

ISSUE 3 • April 2023

The Future of Augmented Experiences in Theaters and Concert Halls

By Marco E. Bruscoli, Scholar-in-Residence, WolfBrown

Throughout history, technological breakthroughs have opened up arts experiences for new audiences, from the spectral illusions of “Pepper’s ghost” (mid-19th century, the antecedent to contemporary holograms) to theater binoculars/opera glasses, projected supertitles, seatback caption displays, and now smartphone apps. 

At every step, new technologies have been rejected as disruptive and antithetical to the live experience. Despite the best efforts of ushers, house managers, and sometimes artists, it seems increasingly impossible to stop audience members from using their phones during live performances. With digital natives now entering the marketplace for live arts programs, we are on a crash course with the expectations and norms of the next generation of theatergoers. How can nonprofit arts organizations safeguard their cherished norms of distraction-free experiences while embracing a new generation that expects increasingly immersive, interactive, and mediated experiences?

Image of people on stage
Image: Dougados Magali

Today’s audiences, well-acquainted with innovations in cinema (IMAX), gaming, and virtual reality, increasingly embrace inventive approaches to experiencing live performances. One of the latest reinterpretations of Giacomo Puccini’s Turandot illustrates the fusion of theater and technology. This groundbreaking production, a collaboration between director Daniel Kramer and Japanese immersive art collective teamLab, engulfs spectators in a dystopian future replete with LED lighting, virtual luminous planes, and kaleidoscopic environments – a fruitful union between seemingly disparate realms of artistic expression. While incorporating new projection technologies into scenography can create a thrilling visual experience for the audience, this alone does not make it an immersive experience.

The user experience is still passive.

Today’s performing arts producers have access to an ever-widening range of augmented reality (AR) sensory technologies that create new avenues for audience engagement, although much R&D remains to be done. What will replace the age-old infra-red headsets in theaters that offer sound enhancement to those who need it? With millions now enjoying captions on their televisions, how can captioning become a normal part of every theatrical experience? Can we leverage new oculus technologies to bring the stage action closer to audiences sitting in the second balcony?

Image of AR vest

The commercial entertainment sector will continue to ratchet up norms and expectations. Almost all audience members at arena concerts will have augmented experiences within the next few years. The recent collaboration between Live Nation and Snapchat allowed users to select digital visual filters to overlay on the real experience, find friends in the crowd, and see close-ups of the stage. At Coldplay concerts, audiences can request SUBPAC vests. The patented tactile audio device delivers deeply immersive and nuanced bone conduction sound with far greater resolution and range than traditional speakers and headphones. Initially intended for the deaf, the experience sets a new standard for audio – i.e., listening with both ears and bodies. 

With the advent of multisensory stimulation chairs from companies like CJ 4DPLEX in cinemas, news of Apple’s $1 billion investment in the movie sector, and IMAX’s rollout of live events to its network of large format cinemas, a notable transformation towards immersive sensory experiences is on the horizon. For conventional theaters, this shift could give rise to cutting-edge ergonomic recliners featuring integrated adjustable sound amplification, support for state-of-the-art personal audio devices such as headphones, and the introduction of pioneering mixed reality headsets.

Image of people with glasses in a theater
Image: Deutsche Oper am Rhein / Lukas Loss

Deutsche Oper am Rhein and Vodafone are conducting a prototyping study to test AR integration in opera performances. A select group of viewers will be equipped with “digital opera glasses” or technically smart glasses, allowing them to zoom in on the action, see captions of sung words, access background information on the plot, and see animated insertions. 

While public interest in location-based immersive experiences will grow dramatically over the coming years, there is still much work to do to understand how to enhance experiences for audiences in conventional theaters and concert halls. This is a systemwide issue affecting every nonprofit arts presenter in every marketplace. It boggles my mind that funders are not heaping millions of dollars into a coordinated program of R & D to solve this challenge for the benefit of the entire sector and audiences nationwide, both young and old, who need a little help understanding what’s happening on stage.

Graphic border from WolfBrown branded image
Immersed in the Future

WolfBrown Presents: Making Sense of the Landscape of Immersive Experiences

We are hosting Making Sense of the Landscape of Immersive Experiences for a second time on May 22- 23, 2023. This two-day intensive seminar in Detroit will update performing arts administrators on the latest developments in immersive experiences. We provide a unique opportunity to enhance your knowledge in the field, network with other professionals, and begin to develop concepts for new immersive program ideas. Registration is open until May 1.

Projects We're Following

Highlights of immersive experiences that caught our eye

Screenshot of a website about Birmingham Royal Ballet

Birmingham Ballet Embraces Immersive Tech

Birmingham Royal Ballet joins Bloomberg Philanthropies’ Digital Accelerator Program, aiming to explore immersive technology for the international dance sector. By investing in equipment and working with industry partners, the ballet company will create digital and virtual content for diverse audiences, beginning with a Swan Lake VR experience for special needs schools.

Screenshot of a website about Holocaust Museum

Preserving Holocaust Stories: Holograms & VR at Illinois Museum

The Illinois Holocaust Museum & Education Center adopted hologram and virtual reality (VR) technologies to ensure Holocaust survivors’ stories are preserved for future generations. In collaboration with the USC Shoah Foundation, the museum created interactive holograms that respond to visitors’ questions. Additionally, they developed VR experiences where visitors virtually join survivors as they revisit important locations from their past. By leveraging these innovative technologies, the museum seeks to promote empathy, educate the public about the Holocaust, and inspire people to stand against hatred and antisemitism.

Screenshot of a website about Body of Mine at SXSW

BODY OF MINE: Exploring Transgender Experience through Virtual Reality

Cameron Kostopoulos presents an interactive virtual reality experience that places users in the body of a transgender person, allowing them to explore senses and engage with interviews about themes like gender fluidity and LGBTQ+ identities. Inspired by Kostopoulos’ childhood in Texas and the lack of safe spaces for LGBTQ+ people, the project combines intimacy with technology, offering an authentic experience based on real-life testimonials. Using HTC Vive technology for full immersion, BODY OF MINE aims to create a meeting point for audiences, fostering understanding and human connection while discussing a challenging theme.

Screenshot of a website about Roblox

Text-Based AI for Virtual World Building on Roblox

Roblox is exploring the use of generative AI to help developers create custom virtual worlds and objects from text descriptions, making the platform more accessible to novice game developers. The company is testing two AI implementations that generate virtual materials and code based on natural language inputs. The aim is to support creators with varying coding and 3D modeling skill sets and to make Roblox a key player in building the metaverse infrastructure.

Screenshot of a website about Coldplay's Immersive experience

Coldplay's Immersive Concert Film Experience

Coldplay reveals two worldwide screenings of their live film “Music Of The Spheres: Live At River Plate” in April, highlighting the multisensory experience. Directed by Paul Dugdale, the film features guest appearances by HER and BTS’s Jin, along with backstage interviews. The cinema screenings will offer various immersive formats, including ScreenX with a 270° view, 4DX for a multisensory experience, and 4DXScreen, a combination of both, providing fans with a unique and unforgettable concert event.

A deep dive into the latest coverage of immersive art performance

Screenshot of a website about mind-controlled VR games

Mind-Controlled Virtual Reality

Cogitat, a neurotech firm, is developing a system that translates brain activity into actions in virtual reality without physical movement. By wearing a prototype headset, users can perform basic actions by simply thinking about them. This technology could eventually help stroke patients and those with brain injuries remotely control devices.

Screenshot of a website about quantifying immersive experiences

Quantifying Immersion in Theatrical Performances

Researchers use bio-monitors and motion-tracking to study audience immersion in live theater and live-streamed performances. By measuring audience synchrony through physiological responses, they aim to understand the differences between in-person and remote experiences and enhance the immersive quality of live-streamed shows, ultimately improving theater accessibility.

Immersive Tech Spotlight

Innovative developments in immersive technology

Screenshot of a website about Apple's new tech

Apple's Game-Changer: The Mixed-Reality Headset Set to Rival the iPhone

Apple is preparing to unveil a mixed-reality headset later this year, a next-generation product with the potential to rival the iPhone. Developed entirely under Tim Cook’s leadership, the headset will showcase the company’s innovation in virtual and augmented reality technology innovation, marking a new chapter for Apple beyond the iPhone, iPad, and Watch.

Screenshot of a website on Science Daily

Touching the Metaverse: E-Skin Revolution

Researchers at the City University of Hong Kong have developed a soft, wireless electronic skin (e-skin) capable of detecting and transmitting touch, forming a tactile network for multi-user interaction. The e-skin could enhance virtual reality experiences by enabling users to feel objects in the virtual world.