Welcome to the multiverse.
At the end of May, I took part in an interesting experiment: Disney, Marvel, and Industrial Light & Magic teamed up to create a mix of VR and AR. Immersive gaming experience Apple Vision Pro TV series “What…If?”.
And if you have an Apple Vision Pro, be sure to download Marvel’s “What…If?” for free.
Dave Bushor, Director and Executive Producer, Marvel Studios
We consider this an experiment because it’s both a game and a concept prototype. First, it’s exclusive to the $3,500 Apple Vision Pro headset. Second, The game is freeAnd third, the game’s interface is based on the headset’s hand gesture recognition, where participants move their hands to imitate the actions that heroes from the Marvel Universe, especially Doctor Strange, perform in the movies.
As a game, it’s fun but nothing special – the gameplay is fairly formulaic – but the immersion aspects are impressive and thought-provoking, and it offers one look at how games and UI elements may evolve in a spatial computing environment.
But I wanted to know more. What were the design decisions? What were the challenges? What was Disney thinking? Really. What was the team thinking?
So I asked, and Marvel Studios Director and Executive Producer Dave Bushor and ILM Immersive Associate Producer Mai-Linh Le were kind enough to answer:
let’s start.
ZDNET: What inspired you to create an immersive version of your “What…If?” series for Apple Vision Pro?
Dave Bushor: “What if?” is the perfect entry point – ask the audience questions from the beginning and let the characters and crossover points evolve however best suits the story. In this case, developing a story for all-new hardware allowed us to stay true to the series in a new way.
ZDNET: What was the biggest challenge in adapting the “What…If?” series to a mixed reality format?
DB: The biggest challenge was juggling the new conversation we were having with our audience throughout their journey in this new format. We didn’t know where they would be looking or how they would participate, but we knew the story needed to unfold at a pace that would resonate with them.
Mi-Linh Le: I wanted the audience to feel like they were the protagonist in the story. [and it does —DG] And how they impact the various multiverses, for better or worse. It was a challenging balance to find as many “what ifs” as possible while at the same time keeping the audience invested in their roles as heroes.
My-Linh Le, Associate Producer at ILM Immersive
ZDNET: How do you think mixed reality will change the entertainment industry over the next five years?
DB: Adoption. As the boundaries are pushed further technologically and hardware, software and devices become more advanced, it will open the door to bigger and better stories we can tell in mixed reality – and that audiences want.
ML: I believe and hope that mixed reality will bring the worlds we’ve always wanted to visit into our homes, where we can talk to and play with our favorite characters and create our own stories and adventures with them.
ZDNET: My-Linh, that’s a very compelling mission statement. Great! What advancements do you foresee in mixed reality technology over the next decade?
ML: I think that over the next decade we’re going to see better spatial and object recognition, and once we get that, I think it will enable us to make experiences more personalized and more engaging.
DB: Developing photorealistic characters in real time would be an emotional game changer, and while the technical challenges are enormous, when it does happen it will really imprint a story in people’s minds that they have experienced or are currently experiencing.
ZDNET: How do you see mixed reality being integrated into other Marvel and Disney productions?
ML: We would love to see ways to empower fans to take part in the actual story, influence the outcome, become a threat if they so choose, and have their own special experiences with their favorite characters.
DB: As things develop, I think it’s going to be a universe of infinite possibilities, defined only by the limits of how far we can push the ideas that people can participate in. The next best stories to happen in this field are limited to what we can imagine.
ZDNET: Can you talk about the collaborative process between Disney, Marvel and ILM in developing this project?
DB: “Disney and Marvel’s collaboration on the potential of mixed and virtual reality storytelling has been ongoing for many years. ILM Immersive has been a great partner and leader in this space, bringing many talented and passionate people to the project and providing great technical and creative momentum.
ML: With Dave guiding and leading us through his creative vision, we were in constant communication with each other – through story meetings, creative reviews, headset reviews, etc., we felt like one cohesive team even though we were spread across multiple studios.
ZDNET: What are the key elements to a successful mixed reality project in terms of audience engagement?
DB: The biggest factor I think across the board with these types of projects is presence – making the characters feel real, which is something we’ve been doing successfully at ILM Immersive for many years.
Also, how to use Meta Quest in a moving car (as a passenger)
Apple Vision Pro takes that to a whole new level. You have the opportunity to create a sense of presence as a spectator, participant or hero, feeling the characters’ performance and moving realistically through the environment – it feels real on some level and opens the audience’s mind to engage with it on that level.
ML: We also believe that giving people an experience of something new is key: if people can walk away saying, “I’ve never seen anything like this before,” that’s a key factor in whether a mixed reality project will be accepted by the audience.
ZDNET: Yes, that’s definitely true! How did you ensure the immersive experience remained true to the original What…If? series?
ML: We received a lot of mentorship from Dave and the “What If…?” team, but many at ILM Immersive were already big Marvel and “What If…?” fans, so many of them regularly shared their Marvel knowledge with us throughout development.
DB: In developing the story, writers David Dong and Phil McCarty focused not only on the core attributes of the Disney+ series, but also on the core of the “What If…?” comics and the fun that story always unlocks. We hammered out some big ideas we wanted to realize creatively with ILM Immersive, worked them into the main arcs of our characters, and had a lot of fun saying “What if…” to each other in every meeting over the next few months.
MORE: Headset making you sick? Here’s an unexpected cure for VR nausea
ZDNET: How do you balance technological innovation and storytelling in your mixed reality projects?
DB: For me as a director, working with the writers, producers, creative team and the team at Marvel Studios, it was about what elements of the story were most important to create an emotional journey for everyone, for the characters and for you, the protagonist. And then layering in the wish-fulfillment possibilities that this technology is beginning to unlock. We had a lot of ideas that we wanted to try from an innovation standpoint, but I think success will be when we can marry the technology with the overall emotion of the journey that we’re trying to create.
ML: I think we typically approach it from the perspective of how to create great stories and experiences, and then evaluate what is feasible. We always try to push the boundaries of what is possible with technology and have an “exploring everything” mentality with every project, but first and foremost, the story we’re trying to tell needs to make sense.
ZDNET: I want to focus on what you say about the “wish-fulfillment possibilities that this technology is beginning to unlock.” It’s profound and specific to this medium. What lessons have you learned from this project that will inform your future mixed reality efforts?
DB: The biggest lesson I’ve learned is that the future lies with those who continue to combine genres, formats, development methods, game features, and storytelling, and that is the biggest opportunity in this field.
This is something that is both completely new and new for our team and our audience, which creates unique challenges and opportunities. The board is blank, and we hope that future projects and efforts from everyone will expand our innovation and increase audience excitement for more content.
ML: In addition to that, we are doing a lot of testing research for this project and we think it will come to fruition in the future. There is a lot of potential for what can be achieved.
I also returned my Apple Vision Pro and Meta Quest 3 to buy these XR glasses, currently discounted by $140.
ZDNET: How does ILM’s special effects expertise contribute to the development of the immersive experience?
ML: Our art team is incredible. We get to work with some of the best people in the industry, and a lot of people at ILM Immersive come from more traditional VFX jobs. I’m truly impressed every day with the quality and fidelity of the work our team produces, and I think that experience speaks to the level of the work.
ZDNET: What are the biggest opportunities and challenges for mixed reality in the entertainment industry?
DB: The biggest opportunities are in increasing breakthrough creative moments, driving audience engagement through hardware, and social repositioning through an expanding ecosystem.
We need more genre-defining content that people can share with their friends, so stories can transcend the boundaries of niche entertainment and become undeniable, generation-defining experiences for audiences older and younger alike.
ML: Accessibility and reach are also big challenges in this space. Many people have yet to explore mixed reality because it often comes with a big financial barrier. My hope is that over time, mixed reality will become even more accessible, which will lead to a surge in adoption of mixed reality storytelling.
ZDNET: What advice would you give to other companies looking to get into mixed reality entertainment?
ML: The best advice I can give is to push the limits of what technology can do – you never know what’s possible until you know how far you can go.
DB: Focusing on the “why?” question helped us narrow our focus and create an incredible sandbox in which to let our “what if…” stories come to life. This is true from an innovation standpoint as much as it is from a narrative standpoint: Why are we telling this story? Why are these characters on this journey? Why are we using this gesture, this model, this location? If it’s important to you, it’s probably important to someone else too.
Plus: Best VR Headsets: Expert Testing and Reviews
Food for thought
This interview gave me a lot to think about, and I hope it does for you too. I’ve been studying XR and spatial computing for a long time, but having some of the best storytellers in history explain why it excites them gives me a deeper understanding.
I’m excited to see where this team goes next. What do you think? What…If? Have you tried any immersive experiences? Have you immersed yourself in VR or AR? Apple Vision Pro or Metaquest 3 Do you like the headset? Who is your favorite Marvel character? Let us know in the comments below.
You can follow daily project updates on social media, so subscribe! Weekly updated newsletterFollow us on Twitter/X David Gewirtz,On facebook Facebook.com/DavidGewirtzon Instagram Instagram.com/DavidGewirtzYouTube YouTube.com/DavidGewirtzTV.
