Forecasts and uncertainty

If you look back to the Towns & Cities forecast tabs or the Rural forecasts, you will notice that at the bottom of the 10 day forecasts is this phrase:

“Forecasts and temperatures for days 1-5 are produced by MetService meteorologists. Forecasts and temperatures for days 6-10 are automatically generated by MetService’s computer weather modelling system.”

What does this mean?

For the first five days in the forecast, the information in the forecast has been written by one of the forecasting team. MetService employs almost 60 meteorologists, who work in shifts to cover the weather 24 hours a day, seven days a week, every day of the year.

Observations, radar images and information from satellites are used by the meteorologist on duty to update the forecasts for the short-term (say the next few hours to a day). By looking at what is happening now, the forecasters can have a fair idea of how weather systems are moving and where they are heading.

You can see this for yourself by having a look at the radar images on metservice.com and watching how showers develop and move, or by watching a band of rain march up the country.

Observations are great for watching weather that has already happened, but much of the time we are interested in what is going to occur further ahead – sometimes even when the system bringing the weather hasn’t yet formed. To do this, computers use complex mathematical equations to model the physical atmospheric processes. These equations are solved by using incredibly powerful computers capable of computing millions of equations every second. The information from these can then be used as guidance by the forecasters.

The output produced by several different global weather models from a number of institutes around the world are used on the forecasting bench to create the forecasts for the first five days, with a trained meteorologist translating the data into forecasts. After day five the forecaster steps away and the computer takes over, and the longer range forecasts are produced without the benefit of human intervention.

Many different elements are combined to make a forecast. This image shows there is more information avaliable for the shorter range forecasts compared to days further into the future.

Many different elements are combined to make a forecast. This image shows there is more information avaliable for the shorter range forecasts compared to days further into the future.

How accurate are the forecasts?

Mother Nature is a tempestuous animal, and even with the best forecasting team, the most powerful computer models and the latest weather observations, weather forecasts can never be perfect.

Looking back over the last year, the accuracy of the short-term forecasts for temperature is around 81% and for rainfall 85%.

So do these stats count for the longer range forecast as well? Forecasting the weather into the future becomes harder the longer out we go. As we head into the future, the uncertainties in the forecast grow, leading to less confidence. Because of the increased level of uncertainty involved, the accuracy of the longer range forecast reduces.

For example, this chart shows the rainfall accuracy percentage for one computer model over the last few months.

For this graph “Accuracy”  is defined in the following manner: The forecast is correct if  a) no rain/shower was forecast for the day and no precipitation was recorded at the observing site, or b) rain or showers were forecast and there was at least a trace of precipitation recorded at the observing site. If either of these conditions were satisfied the forecast was considered accurate.

This graph is based on one computer model as the base forecast source. “Accuracy” is defined in the following manner: The forecast is correct if a) no rain/shower was forecast for the day and no precipitation was recorded at the observing site, or b) rain or showers were forecast and there was at least a trace of precipitation recorded at the observing site. If either of these conditions were satisfied the forecast was considered accurate.

As you can see, the computer forecast for day zero (today) is about 20 percent better than the computer forecast for day nine.

So what causes the drop in accuracy?

Every calculation that is made introduces an element of uncertainty into the forecast and so the more calculations you need to make, the more the uncertainty increases.  The diagram below might help explain a little better.

Imagine dropping a ball down a pin board, as it meets a pin it can go to the left or right. After two levels of pins the ball would be in one of three spots and the route it takes one of four possible ones. If we increase the levels of pins to four the number of outcomes increases to five and the number of possible routes the ball can take also increases.   It is a very simplistic model but if we imagine that each level of pins relates to a time step in the model small changes at any time step can have a difference on the final outcome.

Imagine dropping a ball down a pin board, as it meets a pin it can go to the left or right. After two levels of pins the ball would be in one of three spots and the route it takes one of four possible ones. If we increase the levels of pins to four, the number of outcomes increases to five and the number of possible routes the ball can take also increases.
It is a very simplistic model but if we imagine that each level of pins relates to a time step in the model, small changes at any time step can make a difference to the final outcome.

As mentioned earlier, the amount of computing power needed to run a model or numerical weather predictions (NWP) is huge and a computational balance has to be made between the area covered and the length of time that it goes into the future.

For the first 84 hours of a forecast, the NWP have a spatial resolution of 8km over an area slightly larger than New Zealand; this is the model you can find here

The five day rainfall models on metservice.com are from NWP that cover the globe, but instead of having a point every 8km, the resolution of these models is around 50km. These models are good for picking developing systems and broadscale features, but because of the large resolution they struggle to pinpoint smaller features like isolated showers and sea breezes.

Timing

As with comedy, the secret of success in forecasting is timing, and this is another factor that can be very variable the further into the future we head. For example, take these images which show the same forecast period but were produced by three consecutive model runs 12 hours apart:

These forecasts, showing pressure, rainfall and upper level winds, are for the same time but from different computer model runs. Although the story is similar the timings are very different, meaning the difference between

These forecasts, showing pressure, rainfall and upper level winds, are for the same time but from different computer model runs. Although the story is similar, the timings are very different, meaning the difference between a wet day in Southland and a dry one. As we get closer to the forecast day, the differences between computer models and model runs tend to be smaller and we can be more confident about the forecast.

As you can see, the story for all three forecasts is the same: an area of low pressure over the Tasman Sea moves eastwards across New Zealand bringing rain and showers. The difference comes in the timing: the earliest model run on the left shows the rain (the blue shaded areas) mainly out the west; the next run pushes the rain onto the West Coast; and the latest run pushes the rain further still onto the country.

Getting closer to the forecast day, these changes in timing and the variation in model runs tend to be smaller. With a forecaster sitting between the raw model output and the final forecast, these differences can be minimised.

So how good is the long range forecast?

The longer range forecast is really a guide based on the best model run we have to date, but it is worth remembering that it is still  just computer output – without the benefit of having been reviewed by the MetService forecasting team – and being prepared for the forecast to change as we get closer to the day of forecast.

 

It’s a fine day. Isn’t it?

What do we mean when we say the weather is “fine”?

The word fine is often used to convey the positive attributes of something. It is synonymous with good, well, enjoyable.

How are you? I’m fine!
How was the movie? It was fine.
This is a fine bottle of wine.

When we write weather forecasts we define the term fine to mean that the sun casts sharp shadows. If cloud is thick enough to stop the sun from casting sharp shadows then, even if it doesn’t rain, we don’t think that’s a fine day.

However, New Zealand isn’t known as the “Land of the long white cloud” for nothing, and only infrequently is the sky completely cloud free for a whole day. Cloud often comes and goes. So, when writing forecasts, there are a number of questions to be answered when describing the state of the sky:

  • How much of the sky will be covered by cloud?
  • How thick will the cloud be?
  • How will the amount of cloud vary throughout the day?
  • Is there going to be more or less cloud than the previous/coming days?

Our perception of fine weather also varies with the seasons. In the summer months the sun is stronger and even if there is a lot of thin or high cloud it can still manage to cast sharp shadows. Also, if cloud does block the sun for short periods of time we are less likely to notice because the air is warmer; in fact, it might feel like a relief for a short time!  In the winter when the sun is weaker it may struggle to cast sharp shadows, and the day will feel cooler. If the sun is blocked by cloud, even for a short time, then it can affect the temperature more significantly and make you feel colder.

So, when it’s not a clear-cut blue sky day we consider all these things, as well as how the weather will make people feel. Will they feel it was a fine day? Or a cloudy day?

Auckland on a Fine Day. Photo by Joerg Mueller.

The science behind the new Auckland forecasts

We’ve added a lot more detail to the forecasts for Auckland on metservice.com.

In addition to the existing Auckland forecast on the Towns & Cities page, we’ve divided the greater Auckland region into five sub-regions – each with its own characteristic  rainfall, temperature and wind patterns:

To provide hourly predictions of air temperature, wind speed and rainfall for these sub-regions, we’re using

  • Data from our own localised-for-New-Zealand weather modelling and statistical processing systems (for more about this, see the blog on MetService’s Investment in Forecasting)
  • Observations of temperature and wind speed from representative weather stations within each of the sub-regions (see next point), which we’re blending with  the modelled data for the first few hours of the forecast.
  • The weather stations we’re using are Whangaparaoa (for North Shore), Whenuapai (for Waitakere), Auckland Airport (for Manukau), and Ardmore (for Hunua). For Auckland City we’ve created a “virtual weather station” near the Newton Interchange; this will do the job for now, but we want to replace it with a real station within Auckland City soon.

 Auckland Central forecast page screenshot

The index map at the bottom left of the map area links back to the Auckland Towns & Cities page you’re already familiar with, containing the overall Auckland urban forecast and max/min temperatures for the next 10 days.

This initiative was partly motivated by the great feedback we received about the ‘dust graphs’ of wind speed and rainfall added to the Christchurch pages in February, to alert people to the potential dust nuisance in the areas affected by liquefaction and, more recently, demolition of large buildings.

As always, we’re looking forward to hearing what you think! Tweet @MetService or drop us an email at enquiries@metservice.com

Verification of Maximum and Minimum Temperature in City Forecasts

This blog post is the second in a three-part series discussing verification of MetService forecasts. Here, we present the method used for verifying maximum and minimum temperature in city forecasts, along with some recent examples.

Verification Scheme

Four times daily (around 11:15am, 4:15pm, 10:50pm and 3:45am) MetService issues city forecasts for more than 40 locations. Currently, for 32 of these (and soon for most of the remainder), the forecasts of tomorrow’s maximum temperature and minimum temperature from today’s late morning issue are verified against observations from a nearby automatic weather station.
Over a given period (typically, a month), for each location, the forecasts of tomorrow’s maximum and minimum temperatures from the late morning issue of the city forecasts are compared with observed maximum and minimum temperatures. As with the verification of precipitation in city forecasts, “tomorrow” is the 24-hour period between midnight tonight and midnight tomorrow and the scheme operates automatically – that is, there is no input by MetService staff.

Some Results

Results dating from the implementation of this particular scheme in March 2009 through to 28 October 2010 are below.







The graphs clearly demonstrate what every experienced forecaster knows:

  • Temperature forecasting is generally most difficult over the east of the South Island
  • It is much harder to forecast the minimum temperature than it is to forecast the maximum temperature.

Performance Target

MetService’s performance target for the forecast of maximum and minimum temperature in city forecasts is maximum temperature within 2° C and minimum temperature within 4° C 77% of the time for the 2010/11 financial year and 80% of the time for the 2011/12 financial year.

Limitations

Even across small distances, there can be significant variations in temperature during the course of a day – just ask any horticulturalist. Thus, the observation point used to verify the forecast for a given place may or may not fairly represent the temperature there – or may be a good indication in some weather situations but not in others. MetService observation sites are commonly at airports, which in general are at least some distance from the city or town they serve.