Monthly Archives: July 2013

Compiling WRF

Dependencies

See the post on dependencies. Once all the dependencies are installed, compiling WRF should be plain sailing.

WRF

./compile 2>&1 | tee compile.log

WPS

At first I got this error:

f951: error: unrecognized command line option "-f90=gfortran"

Due to this line in configure.wps, which was incompatible with my version of GCC

DM_FC               = mpif90 -f90=gfortran

Just take out the -f90 option.

If your jasper and png libraries are in a non-standard place, then you may have to modify configure.wps file i.e.:

COMPRESSION_LIBS        =  -L/usr/lib -ljasper -L/usr/lib -lpng
COMPRESSION_INC         =  -I/usr/include

WRF dependencies

Introduction

There are a lot of good instructions for compiling WRF out there, but one thing which is often glossed over are the dependencies.  The instructions might say “WRF needs NetCDF”. So you go hunting down NetCDF, and, because you don’t have root access, you try and compile it from source. But hang on, do you need it with HDF5 support? That will require HDF5, which in turn will require gzip, possibly szip. OpenDAP support? zlib and curl.  Do they need to be compiled statically? Shared? What’s the difference anyway? Why can’t I just download a single installation file and go and have a cup of tea? Why am I wasting my life trying to find how to set LD_LIBRARY_PATH to point to the right thing when I don’t even know what a linker is.

Dependencies in installation order:

zlib

This is most likely installed. Use locate zlib.h to find out where it is on your system

szip

Get szip source code. If you have root access compile it into somewhere central like /usr. If you don’t have root access, compile it into $HOME/usr

tar -xvzf szip-2.1.tar.gz
cd szip-2.1
./configure --prefix=/where/to/install
make
make check
make install

Jasper

Get the source code. If you have root access compile into /usr if not, compile into $HOME/usr. Should compile fairly easily:

./configure --prefix=/usr
make
install

HDF5

I had no success with the pre-built binaries, so get the HDF5 source code. If you’re installing on a cluster and have root access I would strongly reccomend installing it as a module. Create a directory for the module and install into there. If you don’t have root access you can install into your $HOME directory. I usually install into $HOME/usr.

mkdir /cm/shared/apps/hdf5/1.8.11
tar -xvzf hdf5-1.8.11.tar.gz 
cd hdf5-1.8.11 
./configure --prefix=/cm/shared/apps/hdf5/1.8.11 --enable-fortran --enable-cxx 
make check 
make install

Then make a module file which sets the HDF5 variable to point to the base of the installation i.e.

/cm/shared/modulefiles/hdf5/1.8.11

#%Module -*- tcl -*-
##
## modulefile
##
proc ModulesHelp { } {

puts stderr "\tAdds HDF5 to your environment variables,"
}

module-whatis "adds HDF5 to your environment variables"

set              version              1.8.11
set              root                 /cm/shared/apps/hdf5/$version
append-path      PATH                 $root/bin
setenv           HDF5DIR              $root
setenv           HDF5INCLUDE          $root/include
setenv           HDF5LIB              $root/lib

NetCDF

Relax. Take a few deep breaths. Get the source code. Make sure you get the C interface, the Fortran interface and the C++ interface. This will give you full functionality with NCO. Build with NetCDF4 support, you will be glad of it later when you can compress your wrfout files with a single command. I made use of some hints

tar -xvzf netcdf-4.3.0.tar.gz#
cd netcdf-4.3.0
export CPPFLAGS=-I${HDF5}/include
export LDFLAGS=-L${HDF5}/lib
export LD_LIBRARY_PATH=${HDF5}/lib

./configure --prefix=/cm/shared/apps/netcdf/gcc/64/4.3.0/ --enable-netcdf4
make
make check
make install
export NETCDF=/cm/shared/apps/netcdf/gcc/64/4.3.0

Now compile the Fortran and C++ netcdf interfaces:

export CPPFLAGS="-I${HDF5}/include -I${NETCDF}/include" 
export LDFLAGS="-L${HDF5}/lib -L${NETCDF/lib}"
export LD_LIBRARY_PATH="${HDF5}/lib:${NETCDF}/lib" 
./configure --prefix=$NETCDF 
make 
make check 
make install

Create a modulefile similar to the HDF5 example, which sets the NETCDF environment vars.

Udunits

Not a dependency, but useful for NCO. Fairly straightforward to build. Code here. You can also make this a module, I’ll assume from now on the UDUNITS variable is set to the base of the installation.

NCO

Not a depenency for WRF, but a very useful tool set. NCO has its own nice little dependency tree! [NCO Source Code](http://nco.sourceforge.net/mwg-internal/de5fs23hu73ds/progress?id=iH6LwY1X+F , but there is help here

  • ANTLR 2.7.x (not version 3.x!) (required for ncap2 – you can get away without this)
  • GSL (desirable for ncap2)
  • netCDF (absolutely required)
  • OPeNDAP (enables network transparency)
  • UDUnits (allows dimensional unit conversion)

Then build with this:

export NETCDF4_ROOT=$NETCDF 
export CPPFLAGS="-DHAVE_NETCDF4_H -I${NETCDF}/include -I${UDUNITS}/include" 
export LDFLAGS="-L${NETCDF}/lib -L${UDUNITS}/lib" 
./configure --prefix=/cm/shared/apps/nco/gcc/4.3.2 --enable-netcdf4

Or (single command):

NETCDF4_ROOT=$NETCDF \
CPPFLAGS="-DHAVE_NETCDF4_H -I${NETCDF}/include -I${UDUNITS}/include -I${GSL}/include" \
LDFLAGS="-L${NETCDF}/lib -L${UDUNITS}/lib -L${GSL}/lib" \
./configure --prefix=/cm/shared/apps/nco/gcc/4.3.2 --enable-netcdf4

NCL

Not a dependency, but essential if you want nice pretty pictures. Thankfully, the pre-built binaries work very well. You can still install them as a module, just create a directory in the module structure and untar the binaries into that. The make a module file, and make sure it sets the NCARG_ROOT variable, as well as appending to PATH.

Queuing and Scheduling

Resources

Good intro

Translation between PBS and SGE

Using environment vars/arguments

Explanation of queue error state E

Useful HPC commands

Check load status of cluster:

qhost
qstat -f

Find out what queues are available:

qconf -sql

Information about a named queue:

qconf -sq serial.q

List parallel environments:

qconf -spl

Number of slots usually equals number of cores

Login to named queue (i.e. post-processing):

$> qlogin -q <qname>
$> ./my_post_processing_stuff
$> exit

Login to an iterative queue, requesting a number of slots:

$> qlogin -pe <pe_name> <nslots>
$> mpirun -n <nslots> <executable>
$> exit

Example job submission script:

#!/bin/bash

#
# Active comments for SGE
#
#$ -S /bin/bash
#$ -N <jobname>
#$ -v MPI_HOME
#$ -cwd
#$ -q <qname>
#$ -pe openmpi <nprocs>
#$ -j yes
#$ -o <logfile>

. /etc/profile.d/modules.sh

# Module stuff
module add gcc
# Load one of these
module add openmpi/gcc
# module add openmpi/psc
# module add openmpi/pgi
# module add openmpi/intel



# Does not require the -machinefile flag

CMD="$MPI_HOME/bin/mpirun -np $NSLOTS <executable>"

echo $CMD

$CMD

Using python to poll the status of a job:

import subprocess
cmd = 'qstat -f | grep %s' % job_id
proc = subprocess.Popen([cmd], stdout=subprocess.PIPE, shell=True)
output = proc.stdout.read()
if output==None:
    return output
else:
    status = output.split()[4].strip()
    return status