The technology holds promise even for when venues are filled once again

Fox Sports’ coverage of the 2020 MLB season has featured one of the more interesting solutions for dealing with a stadium without fans: add digital fans. The goal is to both fill the stands of MLB games and lay the groundwork for more augmented reality within a live sports production.

Virtual fans are especially effective on wide shots.

“The idea is to augment the broadcast, not take away from it,” says Dan Pack, managing director/partner, Silver Spoon, the company that gives each of the digital fans life via its Crowded software (with a tech assist from Pixotope and SMT). “We want to show broadcasters that there is a lot of value in using AR.”

Mike Davies, SVP, technical and field operations, Fox Sports, notes that the visual difference in seeing an empty venue vs. seeing a venue full of fans, even if they are virtual, is striking.

“The big thing is that Fox isn’t trying to fool anyone here; everyone knows there are no fans,” he says. “But we are in the business of making a better presentation, and I think that this technology accomplishes this goal.”

Fox, SMT, Pixotope, and Silver Spoon came together in a fairly intricate collaboration, according to SMT COO Kirk Brown. It brought together SMT’s technology allowing virtual graphics to be inserted into the broadcast with geospatial precision; Pixotope’s platform, based on the Unreal Engine 3D graphics real-time processor; and the Silver Spoon software allowing thousands of fans to be dynamically created and controlled instantly.

“For this project,” he explains, “we have four cameras at every venue, taking the pan, tilt, zoom, and focus from each camera and doing image stabilization. That gives rock-solid information to Pixotope, which then takes the animated graphics from Silver Spoon and places them in the precise location in the stadium so we can make the effects look as real as possible.”

The trick is to make sure the virtual fans are placed accurately within their seats. That process begins by taking scans of each stadium from MLB and Fox Sports, showing exactly where the stands, tunnels, and other physical elements are located relative to seats.

“We performed the same thing for ‘The MLS Is Back,’ providing a series of virtual Jumbotrons and virtual advertising on the far side and ends of the field,” says Brown.

Pack notes that the work leading to the Fox Sports implementation actually began for a different sport and a different network. “At first, we were half joking about it, but we kept cracking at it and had a little bit of time to develop and put this together for Fox.”

The Crowded software runs in real time and at 60 fps and is produced onsite at each ballpark. Each character Silver Spoon creates begins with motion capture that records the various states a fan will experience during a game.

Silver Spoon does a tremendous amount of motion capture in its studio, and that is where the process of creating virtual fans begins: by capturing a variety of motions and reactions from fans.

“You start with a neutral position,” Pack explains. “If there is a pop-up to the outfield, the crowd stands up, and they can start cheering or booing. It all has to happen in a way that looks natural and isn’t jarring.”

He describes the process of animating each character as a bit like pulling puppet strings: “It’s fun and built in a way that is scalable and customizable as we preload all the approved logos and build out a series of outfits, shirts, pants, etc. Then we can customize those on the fly with colors and skin tones. And everything with crowd simulation is about making it feel random, but, at the same time, you have control over everything.”

Not every game with virtual fans needs to be a sellout.

Director, the Pixotope system, is based on the Unreal Engine 3D graphics real-time processor. Pack says the graphics GUI allows operators to communicate with all of the Unreal Engine machines at one time. He says the Pixotope Director platform is great because it can process multiple cameras at once, allowing cuts from one of the four cameras to another to be seamless and done efficiently.

Getting the fans into the stands begins with the scan, tagging all of the seats, and then narrowing down the virtual crowd to fit into the individual section.

“If there are any discrepancies,” he adds, “we can paint in or erase part of the crowd by just moving the mouse over it.”

Even when fans return to the ballpark, there will be uses for more AR technologies. Virtual mascots can be branded and placed into real-world environments. And things like “moments of exclusivity,” when a stadium is Coca-Cola red or Pepsi blue, for example, can be taken to new heights. And Super Bowl halftime shows will never be the same.

“We want to make it respectful of the game and also more encompassing,” says Pack.

The next goal, he adds, is to reduce the footprint required onsite, including the ability to do the insertion from a remote location. “It’s all about lowering the barrier to entry. Right now, the big issue is the hardware required to run it. If we can centralize the production, broadcasters can use more machines at a lower cost per game.”

The current offering shows the work that still needs to be done (and, eventually, will be done, given the trajectory of technology and processing power) to bring virtual fans to every corner of a ballpark. For example, placing virtual fans behind home plate needs another leap of capabilities to allow the foreground objects — batter, umpire, pitcher — to be separated from the background object — the seats — with virtual fans placed between foreground and background objects.

“A great team of people have put this together,” notes Brown, “and no one has delivered this quality of live broadcast animated graphics with this level of precision before. It has been exciting to do something new and challenging.”