At midnight every day in Bologna, Italy, rows of supercomputers inside a former tobacco factory churn through millions of measurements to predict how the Earth’s weather will change.
Six hours later, energy traders across Europe refresh their browsers for the latest outlook. These mainframe-generated forecasts are often the biggest factor in determining where and when to move energy across the power grid. But a new artificial intelligence model threatens to make them obsolete.
Unlike standard weather simulations, which only process information from satellites and sensors, the AI model from Europe’s intergovernmental forecasting center also integrates historical data. Before its release late last month, the center tested the new method against its conventional model in Bologna and found the AI more accurately predicted temperature, precipitation, wind and tropical cyclones, all while consuming less computing energy.
Convulsed by extreme weather
The model is poised to help traders in Europe and beyond move quickly in power and natural gas markets, which are affected by extreme weather, geopolitics and fluctuations in renewable sources. It could help minimize energy gluts and shortages in Europe, the world’s fastest-warming continent, and provide key information for determining where to build wind and solar farms.
“We can update our information set more often than we are used to,” said Daniel Borup, CEO of Danish trading firm InCommodities A/S. “That obviously leads to improvements in our predictions. It allows us to improve our job and distribute energy better.”
Like its traditional model, the European Centre for Medium-Range Weather Forecasts’ new system – the first AI model released by a major prediction center – estimates temperatures, wind speeds and solar power two weeks in advance. But its improved accuracy means companies and policymakers can move faster on critical weather-related decisions, from canceling rail service to rerouting ships around storms or dispatching trucks to spread sand on icy roads, according to the center.
That level of forecasting could be essential for managing market volatility. Earlier this month, robust solar generation in Germany pushed power prices in several countries below zero. That reversed an earlier trend when a stretch of cloudy, windless weather, known as a Dunkelflaute, reduced renewable output and sent German electricity prices soaring.
The upgrade marks a radical shift from the standard approach of using supercomputers to process millions of measurements to recreate a snapshot of atmospheric physics, then fast-forwarding the model to predict future weather patterns.
Machine learning techniques
Climate and weather datasets were already structured for AI and could benefit from machine learning techniques developed for other scientific research, said Florian Pappenberger, the European center’s deputy director-general and lead forecaster. The forecast center has been experimenting with machine learning techniques since 2018.
“Weather and climate are Big Data problems,” he said. “We have huge amounts of data – humongous amounts – so it’s a perfect match” for the center’s new model.
Once processed, the AI model generates a raw forecast in three minutes, compared to the 30 minutes needed by the center’s supercomputers to produce a conventional outlook, which typically takes six hours to finalize. While the AI model was created by the European intergovernmental group and closely watched by traders across the continent, the forecast itself is global and used by industries and meteorologists worldwide, including in the U.S.
Twenty minutes might not seem like much, but it helps companies, trading firms and government officials respond more quickly to shifts in weather. For example, grid operators can call for more electricity ahead of a cold snap. The two-week forecast period is critical for traders betting on how energy demand will impact prices, said Dan Harding, a meteorologist leading research and development at the European weather analytics firm MetDesk.
“It’s what the markets move on most,” he said.
Machine meteorology
The European center’s AI forecast was refined through collaborations with university scientists and research on experimental weather models developed by tech companies such as Nvidia Corp., Huawei Technologies Co., Microsoft Corp. and Alphabet Inc.’s Google. These results convinced Christian Bach, InCommodities’ head of quant and weather intelligence, that AI models – including the center’s – were outpacing conventional forecasting methods.
“It was really the first indication that machine learning is going to be a big thing,” he said.
Another way to illustrate AI’s rapid rise in meteorology is the European forecasting center’s plan to improve its outlooks over the next decade. AI played a small role in 2020, but the center’s new 10-year road map predicts AI will enhance nearly every aspect of forecasting. The rapid rise of AI and machine learning in meteorology has been “faster than expected,” according to the plan. Data-driven models are “already at a maturity where we can confidently expect them to play an important part in operational prediction.”
AI’s ability to create forecasts quickly with fewer computing resources makes it a good fit for energy traders eager for more frequent weather updates, said Rob Hutchinson, a meteorologist leading the energy and utilities team at the Swiss weather analytics firm Meteomatics AG.
More accurate
Testing from Meteomatics shows the European center’s AI forecasts appear more accurate than conventional models in estimating temperatures five days ahead, he said.
“Speed is one thing, but there are certain parameters and time horizons where there does appear to be some additional accuracy as well,” he said.
However, Hutchinson and other meteorologists don’t expect AI models to replace conventional forecasts anytime soon. The European center is releasing its AI models alongside traditional forecasts and envisions a hybrid system that integrates the most accurate elements of both approaches.
“It’s a lot of marketing hype, sticking AI in front of it and pretending it’s better,” Hutchinson said. “But it’s much more nuanced than that. We have to let the numbers speak for themselves.”
Despite rapid improvements, AI models remain less accurate than conventional forecasts for cloud cover, dust and some weather extremes, Pappenberger said. The current AI model is also only used for a type of forecast that generates one prediction at a time. The next version will apply to ensemble forecasts, which generate 50 predictions per run.
Connecting more directly
The next step, Pappenberger said, is connecting AI models more directly with data from satellites and weather stations. In the future, AI could also tap new streams of weather information from non-standard sources, including cars, appliances, phones and other devices.
“AI weather models have the potential to increase the frequency of forecast updates and improve performance,” said Edoardo Simioni, head of trading and flexibility at Copenhagen-based electricity supplier Reel ApS. Advances in technology, he added, are “ultimately good for the market.”
