Compiling METROMS on Olivia#

Author: Thorbjørn Østenby Moe

Note

This file explains the steps taken to compile METROMS on Olivia. I am by no means an expert on this, so please verify the steps. Some of the troubles I had are likely due to my lack of experience. However, the final configuration I reached seems to work.

This file is mainly for my own reference, but feel free to use it if you find it useful.

Note 1: This can also be interpreted as differences between other clusters and Olivia, but my reference is to Fram.
Note 2: Something with the Open MPI packages seems to be more sensitive on Olivia than previously on Fram, especially when mixing loaded packages (perhaps NRIS/GPU, NRIS/CPU, & NRIS/Login), but there may be other incompatibilities instead/in addition.

Compilation#

  1. Make sure that cluster/home/$USER/metroms_ideal/static_libs/roms-3.8.tar.gz exists and that you have a directory in cluster/home/$USER/metroms_ideal/apps/ with a header file.

  2. Navigate to cluster/home/$USER/metroms_ideal/apps.

  3. Type:

    source myenv.bash olivia

  4. Type:

    source modules.sh

  5. Type:

    ./sbuild_roms_NoCice_roms38.sh Path/to/header_file

  6. Debug or find your new compilation (preferably in $work).

1. myenv file (myenv.bash)#

  1. Make sure that the host option is set to Olivia:

    if [ "$METROMS_MYHOST" == "olivia" ]; then

  2. Change work path to reflect new structure on Olivia:

    export METROMS_TMPDIR=/cluster/work/projects/nn9608k/metroms_ideal_run

    (This is a recurring theme.)

2. modules file (modules.sh)#

  1. Make sure that the host option is set to Olivia:

    if [ "$METROMS_MYHOST" == "olivia" ]; then

  2. Load specific modules and compatible versions:

    module load NRIS/CPU
module load intel/2023b
module load intel-compilers/2023.2.1
module load netCDF-Fortran/4.6.1-iimpi-2023b

    Notes:

    • The NRIS/CPU module was essential to get the right netCDF.mod file.

    • Older Intel packages ensure compatibility with the ROMS compilation.

3. build file (sbuild_roms_NoCice_roms38.sh)#

  1. The MPI and MPI_F90 options should be on.

  2. Make sure the host option is set to Olivia:

    if [ "$METROMS_MYHOST" == "olivia" ]; then

  3. FORT settings:

    export FORT=ifort
export I_MPI_F90=ifort

  4. Compiler flags added on advice from NIRD/Sigma2:

    export MY_CPP_FLAGS="-march=core-avx2 -mtune=core-avx2"
export MY_CFLAGS="-march=core-avx2 -mtune=core-avx2"
export MY_FFLAGS="-march=core-avx2 -mtune=core-avx2"

    (Necessary for Intel compilers on AMD architecture.)

4. The compilation#

Common issues encountered:

  1. Header file problems: Compiler searched for Path/to/header_file/.h instead of Path/to/header_file.h. Fixed by creating the .h directory.

  2. Module problems: netcdf.mod not found. Fixed by loading NRIS/CPU.

  3. Compiler problems: mod_kinds.f90 incompatible with newer Intel. Older Intel packages resolved this.

  4. Flag problems: Olivia uses AMD CPUs; special flags required (-march=core-avx2 -mtune=core-avx2).

  5. Extra files problems: Some files failed due to undefined variables. Adding empty dummy files in common/modified_src/ helped temporarily.

  6. Magical fix: Copying and renaming the .h directory finally allowed compilation to succeed.

5. Post compilation#

  1. Missing romsM file: Turn off debug in build file, turn on USE_large.

  2. Paths in ocean.in outdated: Update all paths for input files.

  3. Model runs on test case but not full model: Must specify nodes and tasks per node on Olivia.

6. run script (runon_olivia.sh)#

  • Standard memory per CPU: 2GB

    #SBATCH --mem-per-cpu=2G

  • Specify nodes and tasks:

    #SBATCH --ntasks=256
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64

  • Work directory by project:

    workdir=/cluster/work/projects/nn9608k/metroms_ideal_run/$runname/$runid

  • Source Olivia environment:

    source ~/metroms_ideal/apps/myenv.bash olivia

  • Additional previously loaded packages are likely unnecessary.