CDC Announces Winner of the ‘Predict the Influenza Season Challenge’

June 18, 2014 — Today CDC announced the winner of its “Predict the Influenza Season Challenge.” First place recognition and a prize of $75,000 went to Dr. Jeffrey Shaman and his team. Shaman is an assistant professor in the Department of Environmental Health Sciences at the Mailman School of Public Health at Columbia University. The CDC competition was launched in November 2013 to promote innovation in flu activity modeling and prediction.

“It’s a great recognition for my team,” said Shaman. “I hope it brings a little more attention to the idea of forecasting infectious diseases. I think there are a number of government agencies that are beginning to recognize the value of disease forecasting and understand this is a research area in which we should invest.”

Participants in CDC’s contest were asked to forecast the timing, peak and intensity of the 2013-14 flu season using digital data. Eleven teams completed the challenge using a variety of data sources (such as data from social media sites and internet search engines) as well as innovative modeling approaches to forecast national and regional flu activity. Three judges, two from CDC and one external to the agency, reviewed the submissions.

Dr. Shaman’s contest submissions stood out. His forecasting model used data from Google Flu Trends as well as CDC’s influenza-like illness (ILI) data, which CDC publishes online each week during the flu season. Shaman worked to overcome the limitations of both data sources. “CDC ILI data is a moving target and isn’t real-time,” he said, while Google Flu Trends is real-time but has a “known set of biases.” The challenge for Shaman was plugging these data into a mathematical model, and then calibrating that model to produce an accurate and reliable forecast.

“The reality in forecasting is that some of the time you are going to be wrong, and some of the time you are going to be badly wrong,” said Shaman. “You try to make it so that those bad forecasts are less and less frequent … and that reliable, accurate forecasts become more the norm.” Shaman says the name of the game is “optimization,” a term that refers to testing and improving the underlying assumptions and behavior of a model so that it provides more accurate and reliable forecasts.

Shaman’s team tested their model against actual flu activity that had already occurred during the season. By looking at the immediate past, Shaman and his team fine-tuned the model to better predict the future. Each week, Shaman’s team ran between 15-18 forecasts and analyzed the outcomes. By observing whether the forecasts “converged” towards a similar outcome or “diverged” towards varying results, his team was able to determine in real time the reliability of their forecasts.

Shaman’s team presented their forecasts in a similar manner to how a meteorologist provides the chance of rain for each day’s weather forecast. This approach helped communicate flu forecasting in a way that was meaningful to both public health officials and the public.

While acknowledging that much work remains to improve the science of flu forecasting, Shaman views the progress that has been made in weather forecasting as proof of the potential for forecasting infectious diseases. “Forecasts were in existence for weather almost 60 years ago and they weren’t very good,” Shaman said, “but steadily over the past 60 years weather forecasters have improved the skill level, accuracy and reliability of their predictions. If we choose as a nation and as a broader scientific community to invest in infectious disease prediction … it’s going to get better.”

CDC continues to explore the possibilities of flu forecasting. While Shaman’s team won this contest, CDC experts learned from each of the submissions and believe the competition helped improve the science of flu forecasting. CDC hopes to keep working with contest participants to discuss forecasting approaches, data sources and lessons learned.

CDC currently monitors flu activity each year using routine flu surveillance systems that do not utilize social media or predict flu activity. Routine surveillance systems combined with flu forecasting have the potential to predict flu activity in the United States. This may prove useful to public health officials for vaccination campaigns, communicating to the public, allocating resources, and implementing strategies to combat the spread of flu disease. The “Predict the Influenza Season Challenge” was the first step towards this goal, and CDC plans to build from it and continue to test and evaluate the most promising flu forecasting methods over multiple future flu seasons.

For more information about flu, visit CDC Seasonal Influenza (Flu). For more information about the “Predict the Influenza Season Challenge,” see the official contest announcement at https://federalregister.gov/a/2013-28198.