Monthly Archives: February 2012

Getting ECMWF data

So, you’ve heard ‘ECWMF is far better than  NCEP’, and you want a piece of some ECMWF data action. Where do you get get it? How do you find it?.  Here’s a list to help you (me) out.

There are multiple routes to ECMWF data heaven. The most flexible and powerful, but most complicated is MARS. To use MARS fully you need the client, a piece of open-source software covered by a GPL licences, but costs £100. It is used to communicate with the data servers at ECMWF, and can subset data on the server side to reduce the volume of data needed to be transferred.  Quicker way to MARS is via  Web MARS.

Some links on MARS
www.ecmwf.int/services/computing/training/material/mars.pdf
www.ecmwf.int/publications/manuals/mars/practice/
www.ecmwf.int/publications/manuals/mars/practice/
www.ecmwf.int/services/archive/

The other route is the ECMWF data server. This gives you straight access, no software needed, no (not many) questions asked. It is actually just a linux server sitting outside the ECMWF firewall running MARS, but only allowing access to the disk archive, and not the tape archive.

To grab a dataset requires a request. Some fields used in the request are:

class ECMWF classification (od, rd, e4, …)
stream originating forecasting system(oper, wave, enfo, seas, …)
expver version of the experiment (01 operational, 11, aaaa)
domain area covered by the data (Global, Mediterranean, …)
system seasonal forecast operational system (1, 2, 3)
type type of field (an, fc, …)
levtype type of level (pl, ml, sfc, pt, pv)
levelist levels for the specified levtype (off if levtype=sfc)
param meteorological parameter (t, temperature, 130, 30.128)
number ensemble member (1, 2, …)
channel brightness temperature frequency band
diagnostic iteration sensitivity forecast products
frequency direction 2-d wave spectra products
product section,
latitude longitude
ocean products

An example of a MARS request is:


retrieve,
class = ei,
stream = oper,
expver = 1,
date = 20071201/to/20071231,
time = 00/06/12/18,
type = an,
levtype = sfc,
param = sd,
target = “era-int.200712.sd”

The request is used to build a MARS tree, which describes the data hierachy to work through.

Truncation is done before interpolation to save resources, and is done depending on the final grid specified. The mapping between the final grid and the resolution truncated to is:
Grid increment Truncation
2.5  ≤ Δ T63
1.5  ≤ Δ < 2.5 T106
0.6  ≤ Δ < 1.5 T213
0.4  ≤ Δ < 0.6 T319
0.3  ≤ Δ < 0.4 T511
0.15 ≤ Δ < 0.3 T799
0.09 ≤ Δ < 0.15 T1279
0.0   ≤ Δ < 0.09 T2047

To subset an area use the boundaries North/West/South/East.

Archived data is available to all registered users. Real-time data only to the big spenders!

ECMWF model data

The global gridded datasets available from central meteorological agencies are amazing. They are the distillation of thousands of global observations from all kinds of sources, pulled together into physically consistent picture of the atmosphere using immensely advanced meteorological models. They are a triumph of the scientific method and the power of cooperation.

They are also confusing as hell to work with. The main problems are:

  • multiple similar datasets, often different versions derived from others 
  • different vertical coordinate systems
  • multiple access routes to the same/similar data 
  • no up-to-date description of the current system
  • information has to pieced together from multiple, contradictory sources

People assume you’re familiar with the models before you start: “Oh, you need the forecast from the T159L60 model”. This post is, by definition*, an idiot’s guide to ECWMF data.  I’ll post this in two parts: the first describing the general system, and the second describing the practicalities actually getting and working with data.

* because it was written by me


ECMWF run the Integrated Forecast System. This is a spectral model: in 2006 the deterministic operational model ran at a resolution of a resolution of T799L91. A summary of this is


Numerical scheme TL799L91 (triangular truncation, resolving up to wavenumber 799 in spectral space, linear reduced Gaussian grid. 91 levels between the earth’s surface and 80 km), Semi-Lagrangian two-time-level semi-implicit formulation.
Time-step 12 minutes
Smallest wavelength resolved 50 km
Number of grid points in model 76,757,590 in upper air and 3,373,960 in surface and sub-surface layers
Grid for computation of physical processes is the Gaussian grid, on which single level parameters are available. The average distance between grid points is close to 25km.
Variables at each grid point
wind, temperature, humidity, cloud fraction and water/ice content (also (recalculated at each time-step) pressure at surface grid-points), ozone
The spectral resolution refers to the highest wavenumber retained in the solution. A the relation between spectral and spatial resolution is given below.
    Spectral Gaussian Lat/lon
    T63 N48 1.875
    TL95 N48 1.875
    T106 N80 1.125
    TL159 N80 1.125
    T213 N160 0.5625
    TL255 N128 0.7
    TL319 N160 0.5625
    TL399 N200 0.450
    TL511 N256 0.351
    TL639 N320 0.28125
    TL799 N400 0.225
    TL1023 N512 0.176
    TL1279 N640 0.141
    TL2047 N1024 0.088
As far as I understand it, the output of the spectral model is converted to reduced gaussian grid, using some kind of magic, very likely involving Fourier transforms.  A gaussian grid plays well with spectral models on a sphere, as the grid points are evenly distributed over the sphere.  This is then converted into a regular latitude-longitude grid (aka cylindrical equidistant aka Plate Carre aka unprojected) as this is what most sane people like to work with.

So remember: the end resolution oft quoted e.g. 1.0 degrees, 0.5 degrees, 0.25 degrees  is the resolution of the end product. It is possible to derive low-resolution datasets from a high resolution model and vice-versa, although obviously going to from a low resolution model to high resolution output will not magically add information.

The resolution above is used for the deterministic forecast.  This also is used to give the initial conditions of the control run of the ensemble forecast. 

The control run (unperturbed) of the ensemble is truncated to a lower resolution from the operational analysis.  In addition, 50 perturbed members make up the full 51-member ensemble.  The ensemble model is run in two legs, with a lower resolution after 10 days.
According to this news article, the resolution of the deterministic and ensemble system were upgraded on 26th January 2010 to the following scheme, which is around 16km resolution!

Deterministic Ensemble Prediction System (EPS)
Leg A Leg B / C
Current
Upgrade
Current
Upgrade
Current
Upgrade
Spectral
T799
T1279
T399
T639
T255
T319
Reduced Gaussian grid
N400
N640
N200
N320
N128
N160
Horizontal grid resolution
~25 km
~16 km
~50 km
~30 km
~80 km
~60 km
Dissemination (LL)
0.25
0.125
0.5
0.25
0.5
0.5
Model Level
Vertical resolution
91
91
62
62
62
62
 

In addition, there are multiple reanalysis datasets, most notably ERA-Interim.  This uses cycle 31r2, which is the system introduced in 2006. This is T255 spectral resolution, 60 vertical levels, and a reduced gaussian grid at approximately 79km spacing.

The ERA-interim reanalysis contains analyses at 00z, 06z, 12z and 18z, and 10 day forecasts from 00z and 12z.  Pressure level and surface level data are archived every 3-hours out to 240 hours. Model level data is archived out to 12 hours. Forecast fields are not available on isentropic or potential vorticity levels.

Another product is the multi-model ensemble, where analyses from different centres: ECMWF, UK Met Office, NCEP and DWD are used to drive an ensemble.