wiki:DevelopmentActivities/Xios

Context Navigation

Version 22 (modified by jgipsl, 9 years ago) (diff)
--

Xios in ORCHIDEE

The interface for XIOS is implemented in ORCHIDEE/trunk rev rev 1788 for usage of ORCHIDEE offline.
This page is about the on going work with XIOS in ORCHIDEE. The work with XIOS is done directly in the trunk of ORCHIDEE. Here are some notes during implementation and testing phase. Note ! All tests for now have only been done with ORCHIDEE offline mode. The implementation in ORCHIDEE will need to change when adapting to run with LMDZ.

Validation

The implementation of XIOS interface in ORCHIDEE is uder validation work. See details here : Branches/Xios/Validation

How to install ORCHIDEE with XIOS : at obelix

Test done with ORCHIEE rev 2469, LMDZ5 rev 2168 and XIOS branchs/xios-1.0 rev 519.

Install configuration ORCHIDEE_trunk or LMDZOR_v6.

Change to have modeles/ORCHIDEE/arch/arch-ifort_LSCE.path as follow:

NETCDF_LIBDIR="/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl -lstdc++"
NETCDF_INCDIR=/usr/local/install/netcdf-4.3.2p/include
IOIPSL_INCDIR=${ORCHDIR}/../../lib
IOIPSL_LIBDIR=${ORCHDIR}/../../lib
XIOS_INCDIR=${ORCHDIR}/../XIOS/inc
XIOS_LIBDIR="${ORCHDIR}/../XIOS/lib -lxios"

If coupled mode, change also to have modeles/LMDZ/arch/arch-ifort_LSCE.path as follow:

NETCDF_LIBDIR="-L/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl"
NETCDF_INCDIR=-I/usr/local/install/netcdf-4.3.2p/include
IOIPSL_INCDIR=$LMDGCM/../../lib
IOIPSL_LIBDIR=$LMDGCM/../../lib
ORCH_INCDIR=$LMDGCM/../../lib
ORCH_LIBDIR=$LMDGCM/../../lib
OASIS_INCDIR=
OASIS_LIBDIR=
INCA_LIBDIR=$LMDGCM/../INCA/config/lib
INCA_INCDIR=$LMDGCM/../INCA/config/inc
XIOS_INCDIR=$LMDGCM/../XIOS/inc
XIOS_LIBDIR=$LMDGCM/../XIOS/lib

No change if AA_make.gdef is done.

Launch : module load netcdf/4p in the job... not yet done...

How to install ORCHIDEE with XIOS for offline use : at CURIE

Extract

> svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
> cd modipsl/util

Add extraction of XIOS by adding following line in mod.def : (XIOS revision 446, 455, 471 and 480 have been tested successfully with ORCHIDEE, revision 452 and 472 do not work) :
#-C- ORCHIDEE_trunk  XIOS/trunk                               480     12 XIOS               modeles

> ./model ORCHIDEE_trunk

Adapt compiling with netcdf parallel. Since revision [1875] there is no need to modify makefiles or arch files. Only loading parallel netcdf and hdf5 is needed. At curie, following has to be done(this is the default case using the proposed IPSL environment):
```
module unload netcdf hdf5
module load netcdf/4.2_hdf5_parallel
module load hdf5/1.8.9_parallel
```

Compile using fcm

> cd ../config/ORCHIDEE_OL
> gmake with_xios

Lance
- The executable can now be used with XIOS or with IOIPSL or with both at the same time.
  - Activate XIOS by adding XIOS_ORCHIDEE_OK=y
  - Deactivate IOIPSL by setting WRITE_STEP=0 and STOMATE_HIST_DT=0 in run.def
- This executable must be used with netcdf and hdf5 parallel, add in the job just before execution following lines(done in libIGCM since tag v2.2) :
```
module unload netcdf hdf5
module load netcdf/4.2_hdf5_parallel
module load hdf5/1.8.9_parallel
```
- The executable can be launched in
  - attached mode : this means that the server XIOS is not used. You only launches the orchidee_ol executable.
  - using the server xios : in this case, one or several cores are dedicated to the server and the reste to orchiee_ol.

Lance using libIGCM
- xios files can not have suffix _xios if IOIPSL output is activated as standard. Instead change to use .xios
- if using multiple_file, add min_digits="4" on the same line in iodef.xml. This can be kept even for one_file :
```
<file_definition type="one_file" par_access="collective" enabled=".TRUE." min_digits="4">
```

Rebuild
- If using file_definition type="multiple_file" then rebuild must be done. Use this rebuild : /ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild. For exemple :
```
/ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -o sechiba_history_xios.nc sechiba_history_xios_*
```
- If using file_definition type="one_file" then no rebuild is needed.

Visulize netcdf output
- ferret : compatible to all netcdf versions, IOIPSL and XIOS output
- nco : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel.
- cdo : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel and also a newer version of cdo.
- Load following modules to use interactivly these tools :
```
module unload netcdf hdf5 cdo
module load netcdf/4.2_hdf5_parallel
module load hdf5/1.8.9_parallel
module load cdo/1.6.0
```

How to install LMDZ-ORCHIDEE with XIOS : at CURIE

Extract

> svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
> cd modipsl/util

Extract configuration LMDZOR_v6
```
> ./model LMDZOR_v6
```

Load as for orchidee offline :

module unload netcdf hdf5
module load netcdf/4.2_hdf5_parallel
module load hdf5/1.8.9_parallel

Compile (by default with XIOS and for hybrid mpi_omp mode)
```
cd modipsl/config/LMDZOR_v6
gmake
```

Lance

Use all xml files found in modipsl/modeles/LMDZ/DefLists and modeles/ORCHIDEE/src_xml except iodef.xml.

The configuration LMDZOR_v6 is under construction. The experiment LMDZOR/clim is half-adapted.... To be finished.

For old versions of LMDZ : LMDZOR with OpenMP

Do as above but :

take revision 1729 of LMDZ and update some files :

cd modipsl/modeles/LMDZ
svn update -r 1905 libf/phylmd/mod_synchro_omp.F90
svn update -r 1905 libf/phylmd/mod_phys_lmdz_omp_transfert.F90
svn update -r 1986 bld.cfg 
svn update -r 1986 makelmdz_fcm
svn update -r 1986 arch/arch-X64_CURIE.path

Add initialization of XIOS in LMDZ/libf/dyn3dpar/mod_const_mpi.F90, change the subroutine Init_MPI to the following :

  SUBROUTINE Init_mpi

#ifdef CPP_IOIPSL
    USE IOIPSL
#else
! if not using IOIPSL, we still need to use (a local version of) getin
    USE ioipsl_getincom
#endif

#ifdef CPP_XIOS
    USE XIOS
#endif
  IMPLICIT NONE
#ifdef CPP_MPI
     INCLUDE 'mpif.h'
#endif
    INTEGER             :: ierr
    INTEGER             :: thread_required
    INTEGER             :: thread_provided
    LOGICAL             :: xios_orchidee_ok
    CHARACTER(len=*),PARAMETER      :: id="client"           !! Id for initialization of ORCHIDEE in XIOS


    xios_orchidee_ok=.FALSE.
    CALL getin('XIOS_ORCHIDEE_OK ',xios_orchidee_ok)
    WRITE(*,*)'In LMDZ: xios_orchidee_ok=',xios_orchidee_ok

#ifdef CPP_MPI
!$OMP MASTER
      thread_required=MPI_THREAD_SERIALIZED

      CALL MPI_INIT_THREAD(thread_required,thread_provided,ierr)
      IF (thread_provided < thread_required) THEN
        PRINT *,'Warning : The multithreaded level of MPI librairy do not provide the requiered level',  &
                ' in mod_const_mpi::Init_const_mpi'
      ENDIF

      IF ( xios_orchidee_ok) THEN
#ifdef CPP_XIOS
         PRINT*,'Before calling xios_initialize in LMDZ'
         CALL xios_initialize(id,return_comm=COMM_LMDZ)
#endif
      ELSE
         COMM_LMDZ=MPI_COMM_WORLD
      END IF

      MPI_REAL_LMDZ=MPI_REAL8
!$OMP END MASTER
#endif

   END SUBROUTINE Init_mpi

before compiling, add in LMDZ/arch/arch-X64_CURIE.fcm so that

%PROD_FFLAGS         -O3 -fp-model precise
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%OMP_LD              -openmp -openmp-threadprivate compat

in ORCHIDEE, add openmp compiling directives in ORCHIDEE/arch/arch-X64_CURIE.fcm

%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%OMP_LD              -openmp -openmp-threadprivate compat

Implementation

In the definition phase using IOIPSL, in the call to histdef, an operator is declared on the field. Using XIOS, some of these operators will be set in the xml file (which contains the definition phase) and some will be applied to the variables in the call to xios_orchidee_send_field.

operation in histdef	terme in xios_orchidee_send_field	operation in xml
flux_op	one_day/dt(*)	-
flux_scinsec	1/dt (**)	-
ave	-	average
once	-	once
tmaxcels	-	maximum
tmincels	-	minimum
sumscatter	-	accumulate

(*) one_day/dt must be replaced by 48.0 to obtain excactly the same results in double precision between XIOS and IOIPSL output files, for instantaneous output frequency.
(**) 1/dt must be replaced by 0.0005555556 to obtain de same results between XIOS and IOIPSL output files, even in float precision, for instantaneous output frequency. This value is temporary commited in revision [1788] until revision [1877].

Download in other formats:

Plain Text