wiki:DevelopmentActivities/Xios

NOTE this page was written during the impelementation of XIOS in ORCHIDEE. It is not update since then. Use the how to page to learn about how to use XIOS today.

Xios in ORCHIDEE

The interface for XIOS is implemented in ORCHIDEE/trunk rev rev 1788 for usage of ORCHIDEE offline.
This page is about the on going work with XIOS in ORCHIDEE. The work with XIOS is done directly in the trunk of ORCHIDEE. Here are some notes during implementation and testing phase. Note ! All tests for now have only been done with ORCHIDEE offline mode. The implementation in ORCHIDEE will need to change when adapting to run with LMDZ.

Validation

The implementation of XIOS interface in ORCHIDEE is uder validation work. See details here : DevelopmentActivities/Xios/Validation

How to install ORCHIDEE with XIOS : at obelix

Test done with ORCHIEE rev 2469, LMDZ5 rev 2168 and XIOS branchs/xios-1.0 rev 519.

  • Install configuration ORCHIDEE_trunk or LMDZOR_v6.
  • Adapt compile options:

1) Change to have modeles/ORCHIDEE/arch/arch-ifort_LSCE.path as follow:

NETCDF_LIBDIR="/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl -lstdc++"
NETCDF_INCDIR=/usr/local/install/netcdf-4.3.2p/include
IOIPSL_INCDIR=${ORCHDIR}/../../lib
IOIPSL_LIBDIR=${ORCHDIR}/../../lib
XIOS_INCDIR=${ORCHDIR}/../XIOS/inc
XIOS_LIBDIR="${ORCHDIR}/../XIOS/lib -lxios"

2) Change in util/AA_make.gdef, the version of netcdf to the following:

#-Q- lxiv8    NCDF_INC = /usr/local/install/netcdf-4.3.2p/include 
#-Q- lxiv8    NCDF_LIB = -L/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf

3) If coupled mode, change also to have modeles/LMDZ/arch/arch-ifort_LSCE.path as follow:

NETCDF_LIBDIR="-L/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl"
NETCDF_INCDIR=-I/usr/local/install/netcdf-4.3.2p/include
IOIPSL_INCDIR=$LMDGCM/../../lib
IOIPSL_LIBDIR=$LMDGCM/../../lib
ORCH_INCDIR=$LMDGCM/../../lib
ORCH_LIBDIR=$LMDGCM/../../lib
OASIS_INCDIR=
OASIS_LIBDIR=
INCA_LIBDIR=$LMDGCM/../INCA/config/lib
INCA_INCDIR=$LMDGCM/../INCA/config/inc
XIOS_INCDIR=$LMDGCM/../XIOS/inc
XIOS_LIBDIR=$LMDGCM/../XIOS/lib
  • Recreate makefiles with ./ins_make.
  • Compile for offline configration in config/ORCHIDEE_OL with gmake with_xios or for coupled configurations as defalut in config/LMDZOR_v6 with gmake.
  • Launch : Tests done for offline with xios in attatched and server mode.
    • Using libIGCM, add in the main job, before execution :
        . /usr/share/Modules/init/ksh
        module load netcdf/4p
      

How to install ORCHIDEE with XIOS for offline use : at CURIE

  • Extract
    > svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
    > cd modipsl/util
    
    Add extraction of XIOS by adding following line in mod.def : (XIOS revision 446, 455, 471 and 480 have been tested successfully with ORCHIDEE, revision 452 and 472 do not work) :
    #-C- ORCHIDEE_trunk  XIOS/trunk                               480     12 XIOS               modeles
    
    > ./model ORCHIDEE_trunk
    
  • Adapt compiling with netcdf parallel. Since revision [1875] there is no need to modify makefiles or arch files. Only loading parallel netcdf and hdf5 is needed. At curie, following has to be done(this is the default case using the proposed IPSL environment):
    module unload netcdf hdf5
    module load netcdf/4.3.3.1_hdf5_parallel
    module load hdf5/1.8.9_parallel
    
  • Compile using fcm
    > cd ../config/ORCHIDEE_OL
    > gmake with_xios
    
  • Lance
    • The executable can now be used with XIOS or with IOIPSL or with both at the same time.
      • Activate XIOS by adding XIOS_ORCHIDEE_OK=y
      • Deactivate IOIPSL by setting WRITE_STEP=0 and STOMATE_HIST_DT=0 in run.def
    • This executable must be used with netcdf and hdf5 parallel, add in the job just before execution following lines(done in libIGCM since tag v2.2) :
      module unload netcdf hdf5
      module load netcdf/4.3.3.1_hdf5_parallel
      module load hdf5/1.8.9_parallel
      
    • The executable can be launched in
      • attached mode : this means that the server XIOS is not used. You only launches the orchidee_ol executable.
      • using the server xios : in this case, one or several cores are dedicated to the server and the reste to orchiee_ol.
  • Lance using libIGCM
    • xios files can not have suffix _xios if IOIPSL output is activated as standard. Instead change to use .xios
    • if using multiple_file, add min_digits="4" on the same line in iodef.xml. This can be kept even for one_file :
      <file_definition type="one_file" par_access="collective" enabled=".TRUE." min_digits="4">
      
  • Rebuild
    • If using file_definition type="multiple_file" then rebuild must be done. Use this rebuild : /ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild. For exemple :
      /ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -o sechiba_history_xios.nc sechiba_history_xios_*
      
    • If using file_definition type="one_file" then no rebuild is needed.
  • Visulize netcdf output
    • ferret : compatible to all netcdf versions, IOIPSL and XIOS output
    • nco : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel.
    • cdo : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel and also a newer version of cdo.
    • Load following modules to use interactivly these tools :
      module unload netcdf hdf5 cdo
      module load netcdf/4.3.3.1_hdf5_parallel
      module load hdf5/1.8.9_parallel
      module load cdo/1.6.0
      

How to install LMDZ-ORCHIDEE with XIOS : at CURIE

  • Extract
    > svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
    > cd modipsl/util
    
  • Extract configuration LMDZOR_v6
    > ./model LMDZOR_v6
    
  • Load as for orchidee offline :
    module unload netcdf hdf5
    module load netcdf/4.2_hdf5_parallel
    module load hdf5/1.8.9_parallel
    
  • Compile (by default with XIOS and for hybrid mpi_omp mode)
    cd modipsl/config/LMDZOR_v6
    gmake
    
  • Lance

Use all xml files found in modipsl/modeles/LMDZ/DefLists and modeles/ORCHIDEE/src_xml except iodef.xml.

The configuration LMDZOR_v6 is under construction. The experiment LMDZOR/clim is half-adapted.... To be finished.

For old versions of LMDZ : LMDZOR with OpenMP

Do as above but :

  • take revision 1729 of LMDZ and update some files :
cd modipsl/modeles/LMDZ
svn update -r 1905 libf/phylmd/mod_synchro_omp.F90
svn update -r 1905 libf/phylmd/mod_phys_lmdz_omp_transfert.F90
svn update -r 1986 bld.cfg 
svn update -r 1986 makelmdz_fcm
svn update -r 1986 arch/arch-X64_CURIE.path
  • Add initialization of XIOS in LMDZ/libf/dyn3dpar/mod_const_mpi.F90, change the subroutine Init_MPI to the following :
      SUBROUTINE Init_mpi
    
    #ifdef CPP_IOIPSL
        USE IOIPSL
    #else
    ! if not using IOIPSL, we still need to use (a local version of) getin
        USE ioipsl_getincom
    #endif
    
    #ifdef CPP_XIOS
        USE XIOS
    #endif
      IMPLICIT NONE
    #ifdef CPP_MPI
         INCLUDE 'mpif.h'
    #endif
        INTEGER             :: ierr
        INTEGER             :: thread_required
        INTEGER             :: thread_provided
        LOGICAL             :: xios_orchidee_ok
        CHARACTER(len=*),PARAMETER      :: id="client"           !! Id for initialization of ORCHIDEE in XIOS
    
    
        xios_orchidee_ok=.FALSE.
        CALL getin('XIOS_ORCHIDEE_OK ',xios_orchidee_ok)
        WRITE(*,*)'In LMDZ: xios_orchidee_ok=',xios_orchidee_ok
    
    #ifdef CPP_MPI
    !$OMP MASTER
          thread_required=MPI_THREAD_SERIALIZED
    
          CALL MPI_INIT_THREAD(thread_required,thread_provided,ierr)
          IF (thread_provided < thread_required) THEN
            PRINT *,'Warning : The multithreaded level of MPI librairy do not provide the requiered level',  &
                    ' in mod_const_mpi::Init_const_mpi'
          ENDIF
    
          IF ( xios_orchidee_ok) THEN
    #ifdef CPP_XIOS
             PRINT*,'Before calling xios_initialize in LMDZ'
             CALL xios_initialize(id,return_comm=COMM_LMDZ)
    #endif
          ELSE
             COMM_LMDZ=MPI_COMM_WORLD
          END IF
    
          MPI_REAL_LMDZ=MPI_REAL8
    !$OMP END MASTER
    #endif
    
       END SUBROUTINE Init_mpi
    
  • before compiling, add in LMDZ/arch/arch-X64_CURIE.fcm so that
    %PROD_FFLAGS         -O3 -fp-model precise
    %OMP_FFLAGS          -openmp -openmp-threadprivate compat
    %OMP_LD              -openmp -openmp-threadprivate compat
    
  • in ORCHIDEE, add openmp compiling directives in ORCHIDEE/arch/arch-X64_CURIE.fcm
    %OMP_FFLAGS          -openmp -openmp-threadprivate compat
    %OMP_LD              -openmp -openmp-threadprivate compat
    

Implementation

In the definition phase using IOIPSL, in the call to histdef, an operator is declared on the field. Using XIOS, some of these operators will be set in the xml file (which contains the definition phase) and some will be applied to the variables in the call to xios_orchidee_send_field.

operation in histdef terme in xios_orchidee_send_field operation in xml
flux_op one_day/dt(*) -
flux_scinsec 1/dt (**) -
ave - average
once - once
tmaxcels - maximum
tmincels - minimum
sumscatter - accumulate

(*) one_day/dt must be replaced by 48.0 to obtain excactly the same results in double precision between XIOS and IOIPSL output files, for instantaneous output frequency.
(**) 1/dt must be replaced by 0.0005555556 to obtain de same results between XIOS and IOIPSL output files, even in float precision, for instantaneous output frequency. This value is temporary commited in revision [1788] until revision [1877].

Last modified 5 years ago Last modified on 06/07/16 10:51:54