Version 13 (modified by clevy, 3 years ago) (diff)

Install the modelling framework (NEMO and XIOS)

Last edition: 10/15/18 16:28:03 by nemo

1. Extract the NEMO code

Using your my_login /pw registered on this web site:

svn --username "mylogin" co http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM

2. Extract and install XIOS

1 - Install the NetCDF4 library: XIOS needs NetCDF4 (not NetCDF3). In addition, if you want to use the "one_file" mode (see bellow), you will need the hdf/netcdf flibraries properly compiled to allow parallel IO.

Note that NetCDF files created by XIOS are not compatible with NetCDF3. The softwares you use to see or analyze NEMO outputs needs to be linked with NetCDF4 and not NetCDF3.

2 - Download XIOS The up to date version of XIOS is XIOS2 (previoulsy XIOS1). XIOS uses xml input files. Those are slightly different from XIOS1 to XIOS2. For the moment, interface for both XIOS1 and XIO2 are available in NEMO, so as the xml files for both. From nemo_v3_6_STABLE and further, use of XIOS2 is now strongly suggested.

Download the XIOS2 software (this will create your xios-2.0 directory): cd $HOME svn co -r 819 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk xios-2.0 ln -s xios-2.0 XIOS

Install the XIOS library and executable: Follow the documentation available here: http://forge.ipsl.jussieu.fr/ioserver/wiki/documentation

Compile NEMO (see NEMO quickstart guide) using appropriate directives :

  • NEMO must be compiled with with following cpp keys (see your NEMOCGM/CONFIG/"MY_CONFIG"/cpp*.fcm file:

key_iomput and key_mpp_mpi

and also (if using XIOS2)

key_xios2

  • you need to set appropriate PATHS for XIOS libraries in your ARCH/arch-xxxx.fcm file,

see for example in NEMOGCM/ARCH, the arch files: arch-macport_osx.fcm, arch-PW6_VARGAS.fcm, arch-X64_CURIE.fcm or NOC Input files for XIOS: You need the following files:

domain_def.xml field_def.xml iodef.xml Templates can be found here:

NEMOGCM/CONFIG/SHARED for field_def.xml and domain_def.xml see NEMOGCM/CONFIG/ORCA2_LIM for examples of iodef*.xml

(Templates for XIOS2 version are different from XIOS1 and can be found here:

NEMOGCM/CONFIG/NEMOGCM/CONFIG/GYRE_XIOS for field_def.xml, file_def.xml, domain_def.xml, and example of iodef.xml)

Run XIOS in attached mode: XIOS used as a library in NEMO, NEMO is the unique executable This mode has bad performances but is convenient as you have only one executable. It is suitable if you use a relatively low number of cores.

Define the attached mode in iodef.xml:

<variable id="using_server" type="boolean">false</variable>

Choose the number of files created:

You can create as many files as NEMO cores or produce directly 1 file for all NEMO cores, by choosing the attribute type of the tag "file_definition" in iodef.xml

  • As many files as NEMO cores :

<file_definition type="multiple_file" sync_freq="1d" min_digits="4">

Note that in this case, the files you will get are not located on the subdomains used in the mpi domain decomposition of NEMO. XIOS (even when it is used as a library) creates its own mpi domain decomposition in order to have subdomains containing complete lines (better for IO performances).

  • A unique file for all NEMO cores:

<file_definition type="one_file" sync_freq="1d" min_digits="4">

Note that this mode is not available if you compiled XIOS with the option "—netcdf_lib netcdf4_seq" or if the hdf/netcdf flibraries are not properly compiled to allow parallel IO.

Check and eventually change the buffer_size

The buffer size is defined in iodef.xml according to your domain size, the number of cores you use and the memory available on your machine. Note that you will get a clear error message if this buffer is too small:

<variable id="buffer_size" type="integer">10000000</variable>

Buffer_size must be bigger than jpi*jpj*jpk*8*3 (with jpi(not jpiglo) and jpj(not jpjglo) NEMO subdomain size)

Run NEMO:

mpirun -np x ./opa Run XIOS in detached (or server) mode: XIOS used as a library in NEMO and as an external executable. This mode has the best performances but is less convenient as you have to deal with two executables (NEMO and XIOS).

Define the mode in iodef.xml:

<variable id="using_server" type="boolean">true</variable>

Choose the number of files created:

If you want to create 1 file per XIOS core or produce directly 1 file for all XIOS cores by choosing the attribute type of the tag "file_definition" in iodef.xml

  • One file per XIOS core :

<file_definition type="multiple_file" sync_freq="1d" min_digits="4">

  • A unique file for all XIOS cores:

<file_definition type="one_file" sync_freq="1d" min_digits="4">

Note that this mode is not available if you compiled XIOS with the option "—netcdf_lib netcdf4_seq" or if the hdf/netcdf flibraries are not properly compiled to allow parallel IO.

Check and eventually change the buffer_size

The buffer size is defined in iodef.xml according to your domain size, the number of cores you use and the memory available on your machine. Note that you will get a clear error message if this buffer is too small:

<variable id="buffer_size" type="integer">10000000</variable>

Buffer_size must be bigger than jpi*jpj*jpk*8*3 (with jpi(not jpiglo) and jpj(not jpjglo) the subdomain size)

Copy xios executable in the directory to run NEMO:

cp XIOS/bin/xios_server.exe .

Create once for all app.txt file:

cat >app.txt -np x ./opa # nemo will run on x cores -np y ./xios_server.exe # xios will run on y cores Run NEMO:

mpirun —app app.txt Advanced options:

From version 482 of XIOS onwards, several advanced features have been added to the server's capabilities. These include: the ability to perform computations with combinations of output fields; the ability to change the precision of output variables and the ability to add user-defined attributes. The following PDF document contains details and some worked examples:

3. Compile and create NEMO executable

The main script to compile and create executable is called makenemo and located in the CONFIG directory. To identify the source code you need, to build the makefile and run it.
As an example, compile GYRE, with ifort on linux to create a MY_GYRE configuration:

cd NEMOGCM/CONFIG; ./makenemo –m ifort_linux –r GYRE -n MY_GYRE

More options

  • –n [CONFIG]: for configuration name, existant or new config to install
  • –m [COMPILER]: compiler choice (use files names in NEMOGCM/ARCH, see examples below)
  • –h: help
  • –r [REFCONFIG]: reference configuration as first guess when installing a new one
  • –j[num]: number of processes to compile (-j 0:no compilation)
  • –d [DIR]: NEMO subdirectory (not needed)
  • –t [DIR]: temporary directory to store libraries
  • -clean [CONFIG_NAME] : remove the Makefile and all the files created afterwards

At the first use, you need the -m option to specify compiler, its options and libraries, then for next compilation, it is assumed you will be using the same compiler.
If –n option is not specified, ORCA2_LIM is the default configuration used. -d option has been implemented for use in non-interactive shell. The WORK directory is created under each configuration directory.

Tools used during the process

  • functions.sh : bash functions used by makenemo, for instance to create the WORK directory
  • cfg.txt : text list of configurations and source directories
  • bld.cfg : FCM rules to compile

Description of directories

  • ARCH : compilation option files, with format arch_compiler.fcm, the compiler name has to be provided with –m option
  • CONFIG : all configurations and a cpp.fcm file containing the list of CPP keys to each configuration
  • EXTERNAL : package to implement an embbeded model (AGRIF)
  • NEMO : source codes
  • SETTE : package to make tests to ensure the reproducibility and restartability of the code after user changes
  • TOOLS : useful softwares to different utilities. For example a tool under MPP_PREP computes the number of water processors for all possible decompositions (up to a maximum number of processors).

Examples

Compiling GYRE, with ifort on linux to create a MY_GYRE configuration

makenemo –m ifort_linux –r GYRE -n MY_GYRE

Recompile it

makenemo

Now, create and compile ORCA_LIM3

makenemo –n ORCA_LIM3 (and answer)

Now ORCA_LIM3_v2 based on the previous one

makenemo –n ORCA_LIM3_v2 -r ORCA_LIM3

Now, create and compile ORCA2_LIM_2_2, add (add_key) and delete (del_key) keys, based on ORCA2_LIM (just un example)

makenemo –n ORCA2_LIM_2_2 -r ORCA2_LIM del_key "key_iomput" (and answer)

To remove a bad configuration

makenemo –n ORCA2_LIM_2_2 clean_config (and answer)

4. Viewing and changing list of active cpp keys

For a given configuration (here called "MY_CONFIG", the list of active cpp keys can be found in

NEMOGCM/CONFIG/"MYCONFIG"/cpp_"MY_CONFIG".fcm

This text file can be edited to change the list of active cpp keys. Once changed, one needs to recompile NEMO using the makenemo command in order ofr this change to be taken in account.

Attachments (2)

Download all attachments as: .zip