How the code for the Schwarz method implementation in LMDZ is organized

Implementation of the Schwarz algorithm in LMDZ

Time stepping and the Schwarz loop (similar to NEMO)

The schwarz loop is implemented in the dyn3dmem/leapfrog_loc.F procedure that is responsible for the time stepping.

As with NEMO there are three loops around the original time-stepping scheme of dyn3dmem/leapfrog_loc.F:

• the outer loop increments counter iswloop with values : 1 to nsloops.
• the middle loop increments counter kswr with values : 1 to mswr.
• the inside loop increments counter itau with values : (iswloop - 1) * ntsinswr to iswloop * ntsinswr

As for NEMO if you run a 5 day simulation with a coupling window of 1 day and a Schwarz iteration number of 6. You get nsloops = 5 and mswr = 6. The value of ntsinswr depends on your time step. It is the number of dynamical time steps during a coupling window.

The loop is implemented around the time stepping scheme.

The beginning of the loops is before the 1 CONTINUE command:

c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

      iswloop = 1

      swzloop: DO WHILE ( iswloop <= nsloops )

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'leapfrog_loc'
         WRITE(lunout,*) '*** Schwarz loops ***'
         WRITE(lunout,*) 'iswloop = ',iswloop
      ENDIF
c$OMP END MASTER

      kswr = 1

      swzit: DO WHILE ( kswr <= mswr ) 

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'kswr = ',kswr
      ENDIF
c$OMP END MASTER

      IF ( kswr == 1 ) THEN
        CALL store_current_time_swz
        CALL dynredem1_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
        swz_store = .TRUE.
      ELSE
        CALL restore_current_time_swz
        CALL dynetat0_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
        swz_restore = .TRUE.
      ENDIF

      itau = (iswloop - 1) * ntsinswr   

c-----------------------------------------------------------------------
c   Debut de l'integration temporelle:
c   ----------------------------------
c et du parallelisme !!

   1  CONTINUE ! Matsuno Forward step begins here

The end of the loops is after the test for MATSUNO/LEAPFROG vs PURE_MATSUNO method:

      END IF ! of IF(.not.purmats)

c------------------------------
cend of schwarz loop
c------------------------------

      END DO swzit! kswr

      iswloop = iswloop + 1

      END DO swzloop! iswloop

      call fin_swz_dyn ! deallocate schwarz dynamics pointers
      call fin_swz_phy ! deallocate schwarz physics arrays

a condition to cycle the loop on kswr is added inside the MATSUNO/LEAPFROG condition:

c-----------------------------------------
c Test de la fin de la fenêtre de Schwarz
c-----------------------------------------

            IF ( itau == iswloop * ntsinswr .AND.
     &         .NOT.(itau == itaufin .AND. kswr == mswr ) ) THEN
              kswr = kswr + 1
              CYCLE swzit
            ENDIF

c-----------------------------------------------------------------------
c   gestion de l'integration temporelle:
c   ------------------------------------

            IF( MOD(itau,iperiod).EQ.0 )    THEN
                    GO TO 1
            ELSE IF ( MOD(itau-1,iperiod). EQ. 0 ) THEN

It works only if the number of steps in the coupling windows is commensurate with the Matsuno steps frequency.

The loop needs to initialise the Schwarz parameters before starting:

c-------------------------------------
c Initialization of schwarz parameters
c-------------------------------------

      CALL init_swz_dyn ! allocate schwarz dynamics pointers
      CALL init_swz_phy ! allocate schwarz physics arrays

!      ncplfrq  = 86400
      ntsinswr = ncplfrq / dtvr
      nsloops  = itaufin / ntsinswr

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'leapfrog_loc, ncplfrq  : ',ncplfrq
         WRITE(lunout,*) '              ntsinswr : ',ntsinswr
         WRITE(lunout,*) '              nsloops  : ',nsloops
         WRITE(lunout,*) '              mswr     : ',mswr
      ENDIF
c$OMP END MASTER

c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

      iswloop = 1

Some modifications are done to the variables that indicate the last iteration to the routine phylmd/physiq_mod.F90.

c-----------------------------------------------------------------------
c   calcul des tendances physiques:
c   -------------------------------
c    ########   P.Le Van ( Modif le  6/02/95 )   ###########
c
       IF( purmats )  THEN
          IF( ( itau.EQ.itaufin.AND..NOT.forward )
     &       .AND. (kswr == mswr) ) lafin = .TRUE.
       ELSE
          IF( ( itau+1. EQ. itaufin )
     &       .AND. (kswr == mswr) ) lafin = .TRUE.
       ENDIF

storing and restoring

As for the NEMO implementation the routines that store the state of the code and restore it are grouped in the Schwarz module: dyn3dmem/schwarz.F90.

There are two sets of routines. Some for the dynamical state variables and some for the physical state variables.

The storing and restoring of the dynamical state is done at dyn3dmem/leapfrog_loc.F level.

The physical state storing and restoring is done in the routine physiq from module phylmd/physiq_mod.F90. It uses the logical variables swz_restore and swz_store from dyn3dmem/schwarz.F90 which are set in the Schwarz loop of dyn3dmem/leapfrog_loc.F. Look at the if section at the beginning of the Schwarz loop in the previous section on time stepping implementation:

      IF ( kswr == 1 ) THEN

The storing and restoring of the physical state is partly implemented in these files:

phylmd/
fonte_neige_mod.F90
pbl_surface_mod.F90
traclmdz_mod.F90

coding details

The Schwarz parameters set by the user in config.card are read in :

dyn3dmem/conf_gcm.F90

The Schwarz parameters that are needed by the different subroutines, including the loops limits, are defined in:

dyn3d_common/control_mod.F90

The dynamic variables needed for storing and restoring are declared in:

dyn3dmem/schwarz.F90

The physics variables are declared in:

dyn3dmem/schwarz.F90
phylmd/phys_state_var_mod.F90

As for the NEMO implementation the outputs are modified to only happen at kswr=ksout in:

phylmd/
iophy.F90
phys_output_write_mod.F90

Note that the "xios calendar" is only updated when kswr=ksout in phylmd/phys_output_write_mod.F90:

    DO iinit=1, iinitend
!      print *,'IFF iinit=', iinit, iinitend 
#ifdef CPP_XIOS
       !$OMP MASTER
       IF (vars_defined) THEN
          IF (prt_level >= 10) then
             write(lunout,*)"phys_output_write: call xios_update_calendar, itau_w=",itau_w
          ENDIF
!          CALL xios_update_calendar(itau_w)
          IF (kswr == ksout) THEN
             CALL xios_update_calendar(itap)
          ENDIF
       ENDIF
       !$OMP END MASTER
       !$OMP BARRIER
#endif

coupling with OASIS

The coupling with OASIS is done by adding an "absolute physical time stepping counter" itap_swz that is used for sending and receiving fields via OASIS in the following files:

change_srf_frac_mod.F90
surf_ocean_mod.F90
surf_seaice_mod.F90

validation procedure

The LMDZ implementation of Schwarz looping has been validated on a forced version where the repeating of the Schwarz window was found to be identical to a reference simulation.