At the heart of the NFL’s media facility in Los Angeles, as many as 75 video editors and other visual storytellers toil behind arrays of monitors, crafting narratives that enrich football fans’ enjoyment and understanding of the game. Sifting through millions of game statistics and thousands of hours of footage on tight deadlines, they field thousands of media asset inquiries each season. “Imagine finding a literal needle in a haystack—the perfect clip to tell the story. And it’s not just one clip. It’s as many clips as needed for an entire video edit,” says Eric Peters, director, media administration and postproduction at NFL.
Now, thanks to a long-standing partnership with Amazon Web Services (AWS), the NFL has moved quickly to leverage its existing data foundation to build a generative AI-powered system called Playbook Pro to automate its entire media retrievals process. Instead of having to master complex media asset management (MAM) processes that used to take up to a half-hour per query, the NFL’s content creators now make natural language requests through a chatbot and, in as little as 30 seconds, receive the best video clips to illustrate a story.
The new system just came online this summer and is already being deployed to transform the NFL’s video storytelling workflow and creative output this season. “Of course, there’s a lot of buzz around generative AI in general,” says Matt Swensson, senior vice president, product and technology, NFL Media. “What’s been awesome is that we’ve implemented it and are using it actively—and the benefits are already showing in the speed and the accuracy of what we’re trying to do.”
A Challenge for Creatives
Before this football season, the NFL’s content creators, such as video producers and editors, had to master an inefficient manual process for retrieving media assets. They used a cumbersome two-step checkbox and filter-based search method to plumb the rich currents of two separate source systems: Next Gen Stats and the MAM system that organizes and stores the NFL’s video library.
Next Gen Stats is the NFL's system that captures player-tracking data to analyze and contextualize the game—providing clubs with insights on trends and player performance while enhancing fans' experiences in the stadium, online, and during game telecasts. It captures real-time data on action on the field using sensors embedded in players’ uniforms and equipment that measure such factors as locations, speed, distance, and acceleration. It then analyzes this to generate hundreds of advanced statistics that can be applied to anything from safety for players to commentary for fans.
For the NFL’s media post-production team—responsible for delivering polished video segments to its media platforms, teams, and other end users—simply matching data about plays to clips that bring those data stories to life could take 20 minutes or more, even for the most experienced editors. First a content creator would have to navigate search filters and check boxes in Next Gen Stats to identify a play. This would be tagged with a game key and play ID that the editor would then type into the MAM system to retrieve a corresponding clip.
“Too often, users ended up settling for a less-than-desirable clip because it took too much time,” Peters says. “To complicate matters, the high turnover rate among staff made it difficult to train people on how to use these complex systems.”
To improve accessibility, the NFL turned to its long-standing partnership with AWS to develop a solution with generative AI, built on AWS technologies such as Amazon Bedrock, MemoryDB, and WebSocket APIs.
Searching for the Solution
The NFL has collaborated with AWS since 2017 to leverage Amazon’s innovative AI and machine learning services to shape the future of football. Their partnership on data-driven challenges has led to a host of innovations—not just Next Gen Stats but systems such as Digital Athlete and other player health and safety initiatives. “Together, we intensely focus on the fan and player experience, and envision the art of what's possible, and then work backwards into the data foundation and technical capabilities that make it a reality,” says Tanya Coutray, principal data and AI strategy leader at AWS.
This approach ensures that the NFL's data initiatives are tightly aligned with the league’s business objectives, while consistently delivering exceptional experiences for football’s hundreds of millions of fans around the world.
With Next Gen Stats, the NFL has accumulated massive amounts of data, such as XY positional coordinates on the field and corresponding speed and acceleration—for every player, for every play, in every game dating back to 2016. “So let’s say you want to see all of the times a certain quarterback faced the blitz,” says Mike Band, senior manager of NFL Next Gen Stats research and analytics. “There’s metadata for that.”
But in using the old two-part system, an editor who wanted to create a video story on a certain player's blitz history would need to have a thorough grasp of that metadata to formulate the right query to turn up the clips they needed. Harnessing generative AI would allow the NFL post-production team to automate the conversion of natural language queries into system-readable commands, vastly simplifying their search.
Learning the Language of Football
The first stage of search automation—using a chatbot, powered by a large language model (LLM), to replace the old query method—worked fairly well. The LLM would interpret the user's intent, search Next Gen Stats to find the best moment, and then automatically retrieve the corresponding clip from the NFL’s media asset management system. There was just one problem: accuracy.
Once the NFL crafted a lookup table that listed each player’s name, team, and the like, and performed data cleaning to handle inconsistencies, the search results improved dramatically. But there was more to come. “Like many industries, football has its own language, which can be confusing for a model,” Band says. “Initially, the system struggled to interpret football-specific terminology, leading to inaccuracies in retrieving relevant statistics and plays.”
Take the definition of “blitz,” for example. Fans, players, and commentators use the term all the time, but an LLM trained only on broad usage of the term might miss the NFL’s definition of it as five or more pass rushers charging the quarterback. So the league partnered with AWS to customize the underlying LLM with detailed, football-specific definition documentation. This lets the model gain a much better understanding of the data, generating far better results for storytellers.
To further enhance the Playbook Pro chatbot’s performance, a semantic caching layer was added in front of the knowledge base. This cache allows the system to retrieve answers to similar or previously answered questions speedily, without needing to access the full knowledge base every time. “With semantic caching using Amazon MemoryDB, correct answers are saved for faster retrievals of similar questions in the future,” Peters says.
Say a content creator asks the chatbot how many fourth-quarter touchdowns a particular player threw in 2022. This query would be committed to memory in a vector database using “player” instead of a specific player's name. The next time someone asks how many touchdowns a different quarterback has thrown in the fourth quarter, the answer should come up much faster—with video clips attached.
The next challenge was keeping the user in the loop with status. Imagine a video editor with a list of clips to find on a tight deadline. “Although the system responds in 20 to 30 seconds, this can feel like an eternity for users waiting for answers,” Peters says. “We integrated a WebSocket API to provide users with constant updates on the progress of their query, so they could understand where the chatbot was in the retrievals process.”
Redefining the Future of Sports Media
Over the past eight years, the NFL has established an end-to-end data foundation on AWS, with critical functionality like security and privacy built in. This has allowed the league to move quickly, experimenting and innovating with generative AI to build differentiated experiences at scale. “What sets a gen AI innovator such as the NFL apart from the rest is its data-driven culture and strategic investment in a data foundation, making these incremental innovations easier for them,” says AWS’s Tanya Coutray.
As a result of these investments, the NFL was able to create a more efficient, scalable, and user-friendly system powered by generative AI that enhances both internal productivity and fan engagement, setting a new standard for data management and media retrievals in the sports industry. Says NFL’s Eric Peters, “With the use of advanced AI technologies, we transformed how we manage and retrieve media assets and statistics.”
For example, he adds, “For a query that brings back a list of 20 plays, in the past that would have taken about 10 minutes to do all of the necessary searches to get to the same place that you can get to in 30 seconds with Playbook Pro. So we’re saving an exponential amount of time due to the larger number of plays that result from our query.”
This generative AI-powered media retrievals system has been in use for just the first few months of the 2024 NFL season. But the NFL is now exploring ways to build on this technology to radically enhance both player health and safety, and the exciting experience for fans. Will Playback Pro find its way directly to team physicians to visually track patterns of player injuries? Will fans tap these video search capabilities to create their own highlight reels? The possibilities are wide open.
