Production model relies on vMix, AWS, NDI, Nimble, and other cloud-based tools
Although the traditional sports world has come to a standstill due to the coronavirus pandemic, many major esports properties are soldiering on by hosting competitions with players competing remotely. In the case of Riot Games, which operates multiple League of Legends regional esports leagues across the globe, this meant creating new cloud-based virtualized live-production workflows that would allow both players and the production crew to be safely located at their homes.
Riot Games NAET lead engineer’s home setup for the new cloud-based LoL production workflow. Credit: Carlos Castillo, Riot Games North America
Thankfully for LoL fans, Riot has been embracing cloud-based and remote-production workflows for years. As a result, Riot’s global technology/engineering teams were able to quickly design and build out an at-home-production model that has allowed LoL competition to continue and livestreams to reach fans at home during a period when they could sorely use a distraction.
“We’ve been leveraging cloud-based workflows really heavily for several years, but this unfortunate situation certainly accelerated our plans quickly,” says Scott Adametz, Esports Technology Group Lead, Riot Games. “The big iron equipment that’s in our data center obviously isn’t very helpful when you can’t bring your people to it and they can’t sit and [operate it]. So we knew we had to come up with an alternative remote workflow.”
At the Core: vMix Virtual Switchers Running in Amazon EC2
The newly launched Riot production model is built around a pair of central vMix virtual video switchers running in Amazon Elastic Compute Cloud (EC2). AWS vMix 1 serves as the primary video switcher (for recording, host cams, observer feeds, and graphics), while AWS vMix 2 handles audio, record, and playout (including master audio mix, video packages, and recording).
An overview of Riot-Games’ OPL remote studio architecture
“We needed to incorporate a lot of different sources from all around the world in IP that are traditionally all in the same building and connected via SDI. vMix has been an awesome solution for that,” says Adametz. “We have now templatized the process of spinning up an environment so that it’s a matter of just a couple of clicks and some configuration variables and it’s ready to go.”
Riot Games was able to connect all its casters and production crew remotely for the new LoL cloud-based workflow.
In addition, Riot is running vMix virtual switchers on its casters’ home computers as they are calling the action. Riot’s broadcast team needs to not only incorporate the casters’ video from their streamer rigs and their audio, but they also need to provide the casters with all the observer feeds (in-game cameras) in order to call the game action.
“That’s not something that is very easy to do unless you have something that is templatized and set up to control everything at the center. So that’s one of the unique caveats that vMix solved for us,” says Adametz. “The [vMix virtual switcher] product is unbelievably full-featured for the challenge that we’re working through and it’s performed really admirably.”
Not surprisingly, all Riot casters have a robust gaming setup at their homes, which provided the extensive GPU power necessary for the vMix software.
“We’re not just remoting people into the same equipment they would use if they were sitting there,” says Adametz. “We’ve actually changed the workflow measurably.”
Keys to Success: How Nimble Streamer and NDI Enable Cloud-Based Workflows
Once all the video sources have been connected to the vMix virtual switcher, the outputs are sent to Nimble Streamer media servers running in Amazon EC2, which handle distribution. Riot had previously used Nimble to distribute clean feeds to its regional partners when it produced live events in 18 languages plus English.
Riot Games’ technology team was able to bring NDI into the public cloud for its new remote-production workflow.
Credit: James Wyld – Riot Games Oceania
Nimble sends SRT inputs to the AWS vMix 1 virtual switcher and receiveds a mix of SRT and RTMP (Real-Time Messaging Protocol) outputs back. Nimble is responsible for streaming the RTMP output to viewers on Twitch, YouTube, and Facebook; the program feed to referees; and the program and multiview feeds to the production crew.
In order for this cloud-based workflow to be possible, Riot’s OCE engineering team was able to crack a tough technological challenge: allowing NewTek’s NDI IP video-transport protocol to function within an Amazon EC2 environment.
“That was the big breakthrough that came courtesy of our Oceania team,” says Adametz. “They were able to get NDI working inside EC2 without multicast, so it still requires a little bit of configuration but was still a major breakthrough. That is how we’re able to share these video streams between multiple instances… and recreate the show that we would normally do in an SDI environment completely virtually in EC2.”
Riot’s LoL communications infrastructure is a mix of the gaming-centric Discord VoIP app and proprietary Riedel remote VoIP solutions. The production crew, observers, referees, casters, players, and various other personnel are all connected via Unity cloud comms.
“We are truly replicating the control room — from front bench on — so that crew members are home participating just as they did before,” says Adametz. “We’re not trying to change the show and we understand that this is not the time to force new workflows on [the production team]. We’re just trying to get them back into a comfort zone. We’re heard [crew members] say they never thought that they would be producing LCS from their kitchen or their living room. Our producer has all the normal feeds that he would have in the control room, but producing the show from the safety of his home. That’s what we’re trying to emulate.”
Never thought I’d produce the #LCS from home! Thanks everyone behind the scenes of @LCSOfficial for working tirelessly to get us back on air. #howilcs pic.twitter.com/OjWqvPkrUp
— Arthur Chandra (@ArthurChandra) March 21, 2020
Virtual Desktop Infrastructure (VDI): “Working From Home” Gets a Whole New Meaning
In order to recreate the operator experience from the safety of the production and observer teams’ homes with no loss of functionality or security, Riot is utilizing AWS Workplace virtual desktop infrastructures (VDI) at their respective locations. Riot was able to extend the workspace that the operator would normally have in a control room to their home by setting up zero clients within Amazon Workspaces. As a result, operators can actually control system like the virtual switcher physically with their keyboard and mouse at home.
Zero clients and AWS Workspaces enable cloud-based workflows to be remoted into operator’s homes. Credit: Riot Games – Esports Technology Group
“It’s like renting a desktop in the cloud and then having KVM over the internet directly to your desk,” says Adametz.
Adametz adds that AWS Workspace has been especially useful to casters who are calling the action. They are able to operate the same familiar interface that they are used to, view all game feeds in near real-time, and control their view of all the different observer feeds. At the same time the production team is receiving the casters’ audio inputs, microphone feeds, and any additional elements – such as graphic overlays – and bringing them back to the central vMix switcher in EC2.
“Our plan is to build caster kits that we can deploy to all of our casters around the world,” says Adametz.
Keeping It a Clean Flight: Ensuring Integrity Remains Paramount
In addition to creating a brand-new broadcast workflow that ensures the safety of players and crew from scratch, Riot was also tasked with ensuring the competitive integrity of the sport — which is of paramount importance in esports.
Riot Games must also ensure integrity in its new remote/cloud workflow for League of Legends Esports events.
“First and foremost, we had to make sure that the game retained its authenticity and it was still the game that we all love to play and watch,” says Adametz. “If we started to make restrictions on how it was played, then suddenly it would become a different sport — and that wasn’t an option.”
First, Riot has instituted a five-minute delay using Nimble so that the public live video feed/results aren’t streamed out live until officials have a chance to ensure competitive integrity.
Second, Riot’s anti-cheat team has been equipped with all of the same live/instant analysis and anomaly-logging tools that it has used for years remotely at home.
And third, Riot has established live webcam feeds at each player’s gaming house or home for referees to monitor play. Typically, referees stand on stage and monitor players from behind during live LoL events. However, with players and referees quarantined at home, this webcam video feed of the players allows the referee remotely to watch and monitor for anything that would be considered unacceptable behavior.
While the anti-cheat team has thankfully not identified any abnormal behavior as of yet, these systems provide them with the tools they require despite being located remotely.
“We were able solve these integrity-related challenges with tech and provided support to our league management, which we were really happy about,” says Adametz. “That way, they can ensure that the style and format of league hasn’t changed — we’ve just physically relocated people.”
A Condensed Timeline: Virtualizing a Workflow in a Matter of Days
Just like the entire media and entertainment industry over the past month, Riot was forced to craft this entirely new production workflow on the fly due to the rapid escalation of the coronavirus pandemic. The timeline began with the Teamfight Tactics: Galaxies Showcase on March 12-13, when Riot was forced to pivot almost overnight from a live event to online-only format due to the sudden travel restrictions put in place in China.
Riot Games’ OCE Team practicing for this weekend’s OPL show utilizing full public cloud workflow. Credit: Victor Zanozin, James Wyld, Matthew Boughen – Riot Games Oceania
“We had to find a way to incorporate some of our influencer participants from Shanghai into a show that was already in production planning for months in a way that still felt authentic and gave them a chance to actually feel like a part of the event,” says Adametz. “That was more of a traditional environment rather than the vMix environment, but we were still able to get it done. That got us thinking about what would happen if we were forced into a more restrictive environment where we couldn’t have players or teams together or even a stage and a control room. Over the last two weeks, that has led to our current workflow.”
At first, Riot deployed a small skeleton crew at its LCS studio in Los Angeles, but the campus has now been on full lockdown for several days, thereby requiring Riot to shift to fully virtualized workflow. In the week since, Riot has continued to flesh out its ecosystem and this weeks’ OPL (Oceanic Professional LoL League) Split competition marks the first 100% cloud-based production.
Riot Games Scott Adametz says the company will continue to get more efficient with its remote-production workflow and figure out more ways to enhance the show and the sport
“As we learn more, we’ll continue to get more efficient and figure out more ways to enhance the show and the sport,” says Adametz, “so that we deliver what fans expect and what they love to watch.”
Although Riot Games’ Esports Technology Group, North America team, and Oceania team led the development of this new production model, Adametz says every single LoL region made significant contributions.
“All the credit goes to the technology teams around the world in Riot esports that have come up with ways to really break the mold and come up with new ways to produce content,” he says. “I’ve never seen so many passionate people willing to take leaps of faith and try things that had never been tried before. It’s awe-inspiring what we’ve been able to come up with as a group.”
An Eye Toward the Future: Riot’s Move to the Cloud Just Accelerated
While Adametz and company, of course, wish the impetus for creating this new cloud-based workflow had never occurred, he says this wave of forced innovation will have a seismic impact on the Riot’s technology roadmap for the future.
“It’s no secret that we pride ourselves on being at the forefront of new production workflows and the lessons that we’re learning from this have already changed our trajectory,” says Adametz. “We have a five-year roadmap for how we would like to produce esports in the future and this has certainly accelerated that plan, not derailed it. Obviously, this is a horrible situation and we wish none of this had to happen in the first place, but we will make the best of it and continue producing the best possible content for our fans and players.”
