Parallelization in ORCHIDEE

Author: D. Solyga, A. Cozic
Last revision: J. Ghattas 2020/03/19

ORCHIDEE is parallelized using the librairy MPI and OpenMP in hybrid mode which means using both methods at the same time. Running the offline ORCHIDEE driver it is only possible to activate the MPI parallelization. Using ORCHIDEE coupled to LMDZ, the default mode is now hybrid mode but using only MPI or sequential mode is still possible. For OpenMP in LMDZ, use LMDZ5/trunk rev 1729 with some modifications (see below) or rev 1907 or later and for ORCHIDEE use revision 1615 on the trunk or later.

Read also this documentation in French version parallelisation_orchidee.2.pdf​.

MPI

The parallelization is transparent as far as there are no interactions between the pixels.
Except if you develop the routing module or if you make changes related to reading and writing files or changing the grid information, you don't need to worry about the parallelization. However, you have some rules to respect :

• Use the subroutines called restget_p, restput_p and getin_p instead of restget, restput and getin. Note that all processes should call these subroutines _p and not only the master.
• It is now possible to use restget_p also for scalar values. But you can still find examples in the code of what was done before for scalars, with a workaround where only the master MPI did the call (to restget and not restget_p), as in the following example:

      IF (is_root_prc) THEN
         var_name = 'day_counter'
         CALL restget (rest_id_stomate, var_name, 1   , 1     , 1, itime, &
              &                 .TRUE., xtmp)
         day_counter = xtmp(1)
         IF (day_counter == val_exp) day_counter = un
      ENDIF
      CALL bcast(day_counter)
  

OpenMP

Since December 2013, rev 1615 in trunk/ORCHIDEE, OpenMP has been included into ORCHIDEE used in coupled mode with LMDZ (since configuration LMDZOR_v5.2).

How to launch a simulation with MPI-OMP parallelization

The default mode using the libIGCM coupled configurations are set up using hybrid parallelization. The number of cores MPI and OMP are defined in config.card, section Executable. For example, the LMDZOR_v6.4 configuration, by default, is set for the atmospheric executable 71 cores MPI and 8 cores OMP and for XIOS 1 MPI (leading to use (71+1)x8=576 procs):

[Executable]
ATM= (gcm_${ResolAtm}_${OptMode}.e, lmdz.x, 71MPI, 8OMP)
SRF= ("", "")
SBG= ("", "")
IOS= (xios_server_${OptMode}.exe, xios.x, 1MPI)

If you want to change to use only MPI parallelization, change to set 1OMP as follow:

[Executable]
ATM= (gcm_${ResolAtm}_${OptMode}.e, lmdz.x, 71MPI, 1OMP)
SRF= ("", "")
SBG= ("", "")
IOS= (xios_server_${OptMode}.exe, xios.x, 1MPI)

The default mode using the libIGCM offline configurations are set up using MPI parallelization. In the following example, 31 cores MPI is set for the orchidee driver and 1 core MPI for XIOS:

[Executable]
SRF= ("", "")
SBG= ("", "")
OOL= (orchidee_ol_${OptMode}, orchidee_ol, 31MPI)
IOS= (xios_server_${OptMode}.exe, xios.x, 1MPI)

If you want to run in sequential mode, set 1 core MPI for the ORCHIDEE executable and no executable for XIOS. This is useful if you run on a small regional domain, on one pixel or if you want to debug easier. See following example where lauching ORCHIDEE driver in sequential mode and XIOS in attached mode(no execuable for XIOS):

[Executable]
SRF= ("", "")
SBG= ("", "")
OOL= (orchidee_ol_${OptMode}, orchidee_ol, 1MPI)
IOS= ("","")

Coding help

Forbidden

It's forbidden to use !$ at the beginning of a line, except for OMP directives. Note that you can use !!$ if you want for comments.

Basics Rules

• All variables declared in a module are variables with SAVE attribute.
• All SAVE variables need to be declared as OMP_THREADPRIVATE (except for variables in src_parallel/). For example:

  INTEGER(i_std), ALLOCATABLE, SAVE, DIMENSION (:) :: indexveg
  !$OMP THREADPRIVATE(indexveg)
  INTEGER(i_std), ALLOCATABLE, SAVE, DIMENSION (:) :: indexlai
  !$OMP THREADPRIVATE(indexlai)
  

• All call to IOIPSL routines need to be suffix by _p except if only working with master process. These routines are declared in src_parallel/ioipsl_para.f90. All processes and threads do the same call. For example:

  CALL histwrite_p(hist_id, 'beta', kjit, vbeta, kjpindex, index)
  CALL getin_p('FORCE_CO2_VEG',fatmco2)
  

• If you need to add an OMP barrier, use the routine barrier2_omp:

  CALL barrier2_omp()
  

Useful variables for parallelization

• is_root_prc declared in src_parallel/mod_orchidee_para.F90 ---> mpi and omp process master (=omp master thread on mpi process master)
• is_omp_root declared in src_parallel/mod_orchidee_para_var.F90 --> omp thread master, one per mpi process
• is_mpi_root declared in src_parallel/mod_orchidee_para_var.F90 --> mpi process master, all omp threads on the master process mpi are true