HOW RIOT USED TECH FROM THE MANDALORIAN TO BUILD WORLDS’ ASTONISHING MIXED REALITY STAGE

Written By Andrew Webster

October 24, 2020

After a hard-fought win over Korean team Gen.G, all five members of Europe’s G2 Esports stood at the edge of a pool of clear, glistening water to take a bow and celebrate their victory. Two members then picked up their star teammate, Rasmus “Caps” Borregaard Winther, and held him over the water, as if to throw him overboard. It’s a good thing they didn’t — despite how real the water may have looked to viewers, it was nothing but pixels.

The annual League of Legends World Championship is currently underway in Shanghai, and like most major events, it has had to be re-envisioned in order to be possible in our new pandemic-dominated reality. Typically, the early stages of the tournament are something of a traveling road show, with different rounds taking place in different cities. In 2020, things had to change.

With travel restrictions in place, and fans no longer able to attend matches, the team at League developer Riot tried something different. They built out a set made up of massive LED screens in a technology setup similar to what Disney used to create The Mandalorian’s sci-fi landscapes. It has been used to startling effect. Matches have looked like they’ve taken place in a cloudy, cyberpunk Shanghai skyline or amid a flooded landscape. What could have been a drab competition in the absence of fans has turned into perhaps the most impressive Worlds in recent memory.

“There are any number of days where we come to the set and say ‘Wait, I don’t think this has ever been done before.’ You just kind of get used to it after a while,” says Michael Figge, creative director at Possible Productions, which partnered with Riot on the event.

The feat is all the more impressive when you consider the compressed schedule. Typically, producers from Riot and Possible spend well over a year planning for Worlds, but that simply wasn’t possible this year. It wasn’t until May that the decision was made to utilize this tech in a studio without fans.

The setup is a powerhouse, and Riot says that the LED screens — there are more than 900 LED tiles in total — display visuals at 32K resolution and at 60 frames per second. Those visuals were made using a modified version of the Unreal Engine, and in total, the team is made up of 40 artists and technicians. Nick Troop, executive producer for Worlds 2020 at Riot, describes it as “a creative tool that gives us effectively infinite power to manifest whatever our collective imaginations bring to the fore.” And he says one of the most important elements of the whole setup is the way things are shot, powered by four specialized cross-reality cameras.

“Rather than having a single projected camera perspective, we actually have two running simultaneously, effectively all of the time,” he explains. This allows the broadcast team to work in a more traditional way; they can swap between the two simulated perspectives at will, using four cameras to shoot the action on set. “It means that the broadcast team can do what feels to them what feels like a ‘normal television show,’ but in this curated, and beautiful series of environments,” says Troop.

For viewers watching on Twitch or YouTube, the LED soundstage is transformed into a sprawling fantasy world, with AR technology used to make the images expand beyond just the screens. You still see players sitting at desks and playing, but their surroundings are quite elaborate. In a nod to the current state of League of Legends, where four elemental dragons are of pivotal importance in a game, each of the four preliminary rounds of Worlds was styled with a different element.

Initially, there were lots of crumbling rocks and mountains to represent the earth dragon; this was followed by the cloudy Shanghai skyline for the air dragons; later, the set appeared to be flooded with water that stretched on forever. This weekend, during the two semi-finals games, things will shift to fire.

While this technology has been used before, most notably on The Mandalorian, this is the first time it’s been done live. “Pretty much every [cross-reality] expression that has been broadcast to this point has not been live,” explains Possible’s Figge, whose company has worked on everything from Super Bowl halftime shows to Justin Bieber concerts. “It’s been pre-shot, similar to a lot of AR stuff for awards shows in North America. It’s risky to do live. We’re doing up to 10 hours a day of live television on this stage. There’s no second chance at it.”

One of the challenges was balancing the desire to make things look cool without interfering with the players. Everyone onstage — teams, coaches, and support staff — has a somewhat different visual experience than viewers at home, since the AR elements only appear for viewers at home. This turned into something of an advantage for the broadcast team.

“When we do these games, it’s really important for the competitive integrity of the sport for the players not to be able to see the game on the Jumbotron or anything like that. It’s a really difficult design problem,” says Figge. “With this stage, everything that’s above a certain level of height on the stage is completely virtual. It’s augmented reality. So we have the game playing in the background and the players can’t see it.”

That said, while players don’t get the full experience viewers do, it was still important that being onstage felt special. This is the World Championship, after all, something teams from across the globe have been striving for all year long. Without the roar of a crowd to hype up players, the spectacle of a vibrant fantasy backdrop is a solid second option. Those onstage can’t see the AR elements, but they can see the graphics on the screens around them. “It helps ground the player,” says Troop. “They can still have a sense of the [game] world reacting, in a way that I think helps with their Worlds experience. There is a certain mindset that comes from being on stage, and we wanted to preserve that.”

In most years, the technical showcase of Worlds is reserved for the opening ceremonies at the finals. In the past, that’s included an AR K-pop concert and a holographic hip-hop performance. It’s still not clear what this year’s big show will look like (though it will likely involve K-pop again), but you could argue that the early rounds have already stolen the show thanks to this new technology. Each round even opened with its own mini ceremony, featuring choreographed dances set in the fantasy realm; performers jumped across crumbling stone bridges and twirled around with magical spells. Despite the circumstances, Riot turned what could have been a low-key edition of Worlds into a surprisingly memorable one.

“It’s been more educational than frustrating,” says Troop of the experience so far.