Installing TIE-GCM on Ubuntu using gfortran

This is a very specific post, most people will want to skip over it entirely. It is about how to install the ionosphere/thermosphere model TIE-GCM on Ubuntu computers using gfortran.

I get a couple of emails a week asking me for help with installing NCARs TIE-GCM v2.0 with GFortran (rather than the standard IFORT). There are a couple of steps that are needed, but the method below is tested with versions of Ubuntu from 14.04-18.04. You can install the model on Linux systems other than Ubuntu but you will need to download and build some packages which are included in the Ubuntu repository.

First we make sure we have the necessary packages installed:

sudo apt install csh libnetcdf* libopenmpi* curl libcurl3-dev gfortran

And that we can use gmake

sudo ln -s /usr/bin/make /usr/bin/gmake

Before we can build the model we need the Earth System Modeling Framework (ESMF) library. This step can cause a lot of trouble because using the wrong version of ESMF seems to cause lots of problems. I have found that TIE-GCM on Ubuntu will only work with ESMF version 7.0.2 (this is not the latest release). It can be downloaded from the ESMF website (this link should be to the right version).

Once it is downloaded extract it to a location of choice, in this example I’ll just drop it in my home directory:

mv ~/Downloads/esmf* ~/
tar -xf ~/esmf_*.tar

This will create a folder called ‘esmf’ containing everything we need (so there is a folder in the location ‘/home/sean/esmf’ in my case). We then need to update some environment variables.

You need to add to the bottom of your bashrc file the ESMF directory (ESMF_DIR) and the type of mpi being used (ESMF_COMM). You can do this with the following commands:

echo 'export ESMF_DIR=~/esmf' >> ~/.bashrc
echo 'export ESMF_COMM=openmpi' >> ~/.bashrc
source ~/.bashrc

The final line will make sure the bashrc file is reloaded. You can test this has worked by typing into a terminal:

echo $ESMF_COMM

and ‘openmpi’ should be returned. We’re now read to build ESMF. First navigate to the ESMF directory and you can check what settings are going to be used:

cd ~/esmf
gmake info

You should see at the top our non-default options for ESMF_DIR and ESMF_COMM. You can then build the library, and check that it built correctly:

gmake
gmake check

Every test should pass. If not something has gone wrong and do not proceed until you have worked out why it failed (for me this has only happened when using the ‘wrong’ version of ESMF – remember we want 7.0.2).

Now, if you haven’t already, download the TIE-GCM model and data from the NCAR website. You’ll want TIE-GCM version 2.0 (30 MB) and the 5.0 degree (1.8 GB) and 2.5 degree (3.8 GB) resolution data files. Untar them and place them in any directory structure you want (but remember where you keep put them). I usually make a folder called ~/data and combine both 5.0 and 2.5 degree files in there and another folder simply called ~/tiegcm. To create the same folder structure as I do (for the TIE-GCM model) simply:

mkdir ~/tiegcm
mv ~/Downloads/tiegcm2.0.tar ~/tiegcm/
cd ~/tiegcm
tar -xf tiegcm2.0.tar

Inside ~/tiegcm you should then have a folder called tiegcm2.0 (which will contain the source code and scripts) as well as a series of ‘job’ and ‘inp’ files and a ‘readme’. Next we need another environment variable this time with the details of where the TIE-GCM data directory is, assuming it is ~/data then:

echo 'export TGCMDATA=~/data' >> ~/.bashrc
source ~/.bashrc

Again we can check this worked by from a terminal running ‘echo $TGCMDATA’. Next we need to ensure our makefiles are correct. First make a copy of one of the default Make info files for us to edit

cp ~/tiegcm/tiegcm2.0/scripts/Make.gfort_hao64 ~/tiegcm/tiegcm2.0/scripts/Make.gfort_ubuntu

Now we need to edit the file to make sure the paths are correct. Specifically you need to make sure you’ve updated the paths for F90, MPIF90, MPIRUN, LIB_NETCDF, INC_NETCDF and LIB_ESMF. If you’re on Ubuntu, and you installed the packages from the repos then these will simply be:

F90 = gfortran
MPIF90 = mpif90
MPIRUN = mpirun
LIB_NETCDF = /usr/lib -lnetcdff -lnetcdf
INC_NETCDF = /usr/include
LIB_ESMF = ~/esmf/lib/libO/Linux.gfortran.64.openmpi.default

Make sure you keep all of the other parts of the file, and only edit the above. You can download my Make info file here, which should work for you if you installed ESMF in ~/esmf. If not, you’ll need to update the paths.

Finally we need need to update the Makefile for compiling TIE-GCM. I really struggle understanding Makefiles but the one that is included fails because an if clause is outside of any rule scope. So I’ve found that 3 lines have to be commented out for the model to build and work. The file, ‘~/tiegcm/tiegcm2.0/scripts/Makefile’ as downloaded from NCAR looks like this (lines 51-54):

ifeq ($(F90),/usr/bin/gfortran)
  ESMF_F90COMPILER=$(MPIF90)
  @echo "set ESMF_F90COMPILER=$(ESMF_F90COMPILER)"
endif

We need to comment out all of but the ESMF_F90COMPILER lines. Specifically making the files look like this:

#ifeq ($(F90),/usr/bin/gfortran)
ESMF_F90COMPILER=$(MPIF90)
#  @echo "set ESMF_F90COMPILER=$(ESMF_F90COMPILER)"
#endif

Alternatively you can download the updated file here, and replace the ‘Makefile’ in your ‘~/tiegcm/tiegcm2.0/scripts’ directory.

Now everything is set up and good to go. We just need to make sure the paths in our ‘job’ and ‘inp’ file are correct and the model can be run. In ~/tiegcm/tiegcm-linux.job update the path locations. If you’ve installed everything where I have then the following should work (obviously change things like the number of processors (nproc) if you want to):

set modeldir = ~/tiegcm/tiegcm2.0
set execdir  = ~/tiegcm/tiegcm.exec
#set tgcmdata = tiegcm_res5.0_data
set input    = ~/tiegcm/tiegcm_res5.0.inp
set output   = ~/tiegcm/tiegcm_res5.0.out
set modelres = 5.0
set make     = Make.gfort_ubuntu
set mpi      = TRUE
set nproc    = 4
set debug    = FALSE
set exec     = TRUE
set utildir  = $modeldir/scripts

That should be everything and now you can run TIE-GCM by:

cd ~/tiegcm
./tiegcm-linux.job

As I said, these instructions are designed for Ubuntu (where everything you need, like NETCDF) are provided in the repos. If you want to install this on another Linux distribution you may have to compile these yourself. If anyone needs help with this, drop me a message in the comments below and I’ll make a guide for that as well.

This entry was posted in Linux, Science and tagged , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.