Parallelization in ORCHIDEE

ORCHIDEE is parallelized in hybrid mode, using both MPI and OpenMP. Running the offline ORCHIDEE driver it is only possible to activate the pur MPI parallelization. Using ORCHIDEE coupled to LMDZ, it is possible to run in hybrid mode using reccent versions of both models. For OpenMP in LMDZ, use LMDZ5/trunk rev 1729 with some modifications (see below) or rev 1907 or later and for ORCHIDEE use revision 1615 on the trunk or later.

Read also this documentation in frensh version parallelisation_orchidee.2.pdf​.

---

MPI

The parallelization is transparent as far as there are no interactions between the pixels.
Except if you develop the routing module, you don't need to worry about the parallelization. However, you have some rules to respect :

• 1. Use the subroutines called restget_p, restput_p and getin_p instead of restget, restput and getin. Note that all processes should call these subroutines _p and not only the master.
• 2. Exception : if you restart a scalar, you have to use restget followed by a bcast command :

      IF (is_root_prc) THEN
         var_name = 'day_counter'
         CALL restget (rest_id_stomate, var_name, 1   , 1     , 1, itime, &
              &                 .TRUE., xtmp)
         day_counter = xtmp(1)
         IF (day_counter == val_exp) day_counter = un
      ENDIF
      CALL bcast(day_counter)
  

  for restput, follow the example :

      IF (is_root_prc) THEN
         var_name = 'day_counter'
         xtmp(1) = day_counter
         CALL restput (rest_id_stomate, var_name, 1, 1, 1, itime, xtmp)
      ENDIF
  

  Note that only the master does the call to restget and restput.

---

OpenMP

Since december 2013, rev 1615 in trunk/ORCHIDEE, OpenMP has been included into ORCHIDEE used in coupled mode with LMDZ (configuration LMDZOR_v5.2)

How to launch a simulation on curie (or ada) with MPI-OMP parallelisation ?

Update LMDZ

You can work with LMDZ5/trunk rev 1729 but you need to update 2 files, as below, or use LMDZ5/trunk rev 1907 or later. Revisions older than 1729 has not been tested.

If you use rev 1729, update as follow:

cd modeles/LMDZ/libf/phylmd
rm mod_synchro_omp.F90
rm mod_phys_lmdz_omp_transfert.F90
svn update -r 1905 mod_synchro_omp.F90
svn update -r 1905 mod_phys_lmdz_omp_transfert.F90

Compilation

ORCHIDEE
To compile Orchidee you need to modify modipsl/util/AA_make.gdef

• On Curie

  #-Q- curie F_C = mpif90 -c -cpp -openmp
  (...)
  #-Q- curie F_O = -DCPP_PARA -O3 -DCPP_OMP -openmp-threadprivate compat $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
  

• On Ada

  #########################################" To be complete 
  

LMDZ
It's mandatory to have the same compilation for LMDZ, ORCHIDEE and IOIPSL. So you need to check the file modeles/LMDZ/arch/arch*.fcm.

• On Curie

  %PROD_FFLAGS -O3 -fp-model precise -openmp-threadprivate compat
  

• On Ada

  ####################################### To be complete
  

Configuration Makefile
Recreate the Makefile with the command

cd modipsl/util
./ins_make

Then modify it by activating hybrid mpi-omp compilation for LMDZ. In config/LMDZOR_v5.2/Makefile, change into -parallel mpi_omp :

(cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -parallel mpi_omp -arch $(FCM_ARCH) gcm ; cp bin/gcm_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/gcm.e ; )

Make sure the preprocessing key ORCHIDEE_NOOPENMP is not set as argument to makelmdz_fcm. This key is used with older versions of LMDZ without OpenMP.

Setup your first simulation

Follow ​the platform documentation to know how setup your simulation using libIGCM. Some changes are needed to indicate to libIGCM that you work in hybrid MPI-OpenMP mode.

• Change "Executable" section in config.card by indicating the number of MPI processes and the nubmer of omp threads (2, 4 or 8). For exemple:

  ATM= (gcm.e, lmdz.x, 24MPI, 2OMP)
  

• Modify the main job header
  • On Curie

    #MSUB -n 24 # reservation des processeurs pour le job
    #MSUB -c 2  # nombre de threads OMP
    #MSUB -N 3  # nombre de nœuds de calcul = nbMPI x nbOMP /16 = 24 x 2 / 16
    (...)
    BATCH_NUM_PROC_TOT=48 # nombre de cœurs : nbMPI x nbOMP = 24 x 2
    

  • On Ada

    ######################################################## To be complete
    

How to launch a simple job in MPI_OMP

See here how to launch a simple test job ​https://forge.ipsl.jussieu.fr/orchidee/wiki/HowTo/TestCaseBatch. Use also curie.info documentation (write curie.info in a Curie terminal) for more information.

---

Coding Rules

Forbidden

It's forbidden to use !$ at the beginning of a line, except for OMP directives. Note that you can use !!$ if you want for comments.

Basics Rules

• All variables declared in a module are variables with SAVE attribute.
• All SAVE variables need to be declared in THREADPRIVATE (except for variables in src_parallel/)

  INTEGER(i_std), ALLOCATABLE, SAVE, DIMENSION (:) :: indexveg
  !$OMP THREADPRIVATE(indexveg)
  INTEGER(i_std), ALLOCATABLE, SAVE, DIMENSION (:) :: indexlai
  !$OMP THREADPRIVATE(indexlai)
  

• All call to IOIPSL routines need to be suffix by _p except if working with master process. These routines are declared in src_parallel/ioipsl_para.f90. All processes and threads do the call. For exemple:

  CALL histwrite_p(hist_id, 'beta', kjit, vbeta, kjpindex, index)
  CALL ioconf_setatt_p('UNITS', '-')
  CALL getin_p('FORCE_CO2_VEG',fatmco2)
  

• If you need to add an OMP barrier, use the routine barrier2_omp

  CALL barrier2_omp()
  

Useful variables for parallelization

• is_root_prc declared in src_parallel/mod_orchidee_para.F90---> mpi and omp process master (=omp master thread on mpi process master)
• is_omp_root declared in src_parallel/mod_orchidee_omp_data.F90 --> omp thread master
• is_mpi_root declared in src_parallel/mod_orchidee_mpi_data.F90 --> mpi process master