Wednesday 19 October 2016

The Ministry of Business, Innovation and Employment (MBIE) has commissioned a report to determine if there is a level of government-held weather data which should be opened up for public access (over and above data already available) to better stimulate innovation and promote economic growth. MetService is actively engaging with this review and with the Open Data Programme, and we thought you'd like to know more about how we use this kind of information.

What is the "weather data" being considered by the review?

“Weather data” for the purposes of the review is defined to mean observations of the state of the atmosphere from either in-situ or remote sensing systems (e.g. ground-based weather stations, weather balloons, weather surveillance radar, satellites etc.). Within the environmental services sector, observations make up the most basic data from which value-add services, such as weather forecasts or climatological products, are derived.

MetService operates a comprehensive observing network over and around New Zealand which complies with recognised international standards as prescribed by the World Meteorological Organization (WMO). This includes data collected from:

  • Surface observations over New Zealand (automatic weather stations)
  • Upper air observations using a variety of means (ground-based equipment, weather balloons, aircraft, satellites)
  • Weather surveillance radars ("rain radars")
  • Ships
  • A network of drifting buoys in the Tasman Sea

In addition to data from the observing network which MetService is required to have and maintain to WMO standards under its public weather safety contract with the Ministry of Transport (MoT), MetService also manages observing equipment, and collects and produces highly-specialised data and forecasts, for a range of commercial customers in New Zealand and internationally. This revenue provides the majority of MetService’s profit (as an SOE, MetService operates as a commercial company that must fund itself through commercial activity).

The diagram below represents all the kinds of observations that go into a typical observing network: 

A few notes on the observing system:

  • The NMS you see to referred to in the bottom-right corner of the diagram (which receives all of this information) is MetService - we are New Zealand's official National Meteorological Service with WMO. You can find more information about the national weather services we provide to the country and the region here.
  • You might be wondering how we get satellite data - MetService certainly doesn't own any satellites! We receive data from satellites (owned by other countries) which provide coverage of our area of the planet. The two key satellites we use data from are the Himawari-8 satellite (owned by the Japan Meteorological Agency) and the GOES satellite (owned by NOAA in the United States). We show imagery from both satellites on our website (but rather than slow things down for our website users by showing very high-resolution imagery, we encourage those who are interested in this to go direct to the source for it).
  • MetService does, however, own nine weather radars (and planning for number 10 is underway). Like satellite data, the data from these radars comes in huge file sizes. Our aim is to show imagery on the website that gives you a clear idea of where rain is coming from and how fast, at a regional and city level. The most important use of radar data is in the forecast room, for our Severe Thunderstorm Warning service. 

How MetService uses observation data

Observations are one of the three core components of weather forecasting (the other two components being computer models and skilled meteorologists).They are so important because to know where you're going (i.e. to produce a reliable forecast) you need to know where you are right now (receive reliable observations). MetService uses observations in three key ways:

  • as input into global and local computer models; 
  • as guidance for skilled meteorologists in the forecast room; and
  • for public interest on our website and apps.

To look into the future, MetService starts with a number of global weather models which are produced by large weather modelling centres in the USA, UK and elsewhere. These models require massive computing power, using supercomputers. Thanks to the high quality of these global models, MetService doesn't need to own and maintain its own supercomputer. Instead, we use the best models from around the world (which have our observations fed back into them to help improve their usefulness for our part of the world), and then add a more localised layer of information to them using our own models.

These local models are produced by our own research team and fine-tuned by our own forecasters to maximise their accuracy for our little patch of the globe. We use cloud-based computing to get the "data crunching" done - and to give you an idea of how much storage that takes, as at September this year the modelling team has moved 54 Terabytes of data into the cloud that previously would have to have been stored in our data centre. This is the equivalent of all the information in the largest library in the world (the USA’s Library of Congress with more than 162 million items), stored more than 5 times over.

On top of the skill added by our own weather modelling team are the skills of our meteorologists who interpret what the observations, weather models and other ‘guidance’ are telling us about what the weather will do over the coming days. The forecasts are discussed and agreed by the team of meteorologists on duty, not just one person – but each person is focused on a specific area or aspect of weather so that everything is considered in detail. 

MetService's website and apps are extremely popular with New Zealanders. In addition to essential forecasts and severe weather information, they enable us to make a whole range of data and information available to the public and increase public engagement with our communication channels. This helps generate wider awareness of public weather safety information (our #1 aim), as well as a significant return (through advertising revenue) on MetService's investment in the technology required to run these digital platforms.

How MetService makes its observations accessible

  • Observational data from the Regional Basic Synoptic Network (a core set of observations considered "adequate to meet the requirements of Members and of the World Weather Watch") is freely distributed internationally through WMO with no restrictions on use. Among other things, these observations are used to help improve the usefulness of the global models for this part of the world.
  • The MoT's contract with MetService specifies the meteorological data collected with public funds that is to be provided as ‘open access data’, and also stipulates the methods and conditions under which that data is to be made accessible - including to competitors.
    Here's more information about this data and how to access it:
  • In addition, much of the data that is collected for the MoT contract is made publically available on our website and apps in near or real time - this includes 7.5 minute (real time) rain radar imagery; 1 minute temperature observations; 10 minute wind direction, wind speed and wind gust observations; and hourly rainfall, humidity and pressure observations. This is the same data as our forecasters see in the forecast room!
  • MetService also produces data and forecasts outside of its contract with the MoT, and we make many of these available to the public on our websites, apps and social media in addition to the data and forecasts produced for the MoT contract. 
  • Importantly, all observations we collect (whether for commercial customers or public weather services) are available to forecasters for use in public safety forecasting - including Severe Weather Warnings and Watches.
  • All observational data gathered in direct support of the public forecast services provided under the MoT contract is also passed to NIWA for archiving in the National Climate Database, CliFlo

We hope this information helps you understand the kinds of observational data available in New Zealand, at least from MetService anyway, as well as how it's used and how and where you can access it.

How to make a submission to the Review

MBIE has contracted PWC to provide an independent report on open access to weather data. The report will look at data availability, cost, use and potential, and how this works in relation to New Zealand’s open government data principles. They are seeking public submissions to help inform the report. In particular they want to learn about your thoughts on:

  • current arrangements for access to weather data
  • the future state of weather data availability in New Zealand
  • the potential for innovations from access to weather data that is not currently available.

Information from submissions will be analysed and included in the report.

If you are interested in submitting your views, please use the MBIE online submission form.

Note that you do not need to answer every question, and that partially-completed submissions will still be included in the analysis.

Submissions close 5pm, Tuesday 8 November 2016.