How the code for the Schwarz method implementation in LMDZ is organized

Implementation of the Schwarz algorithm in LMDZ

Time stepping and the Schwarz loop (similar to NEMO)

The schwarz loop is implemented in the dyn3dmem/leapfrog_loc.F procedure that is responsible for the time stepping.

As with NEMO there are three loops around the original time-stepping scheme of dyn3dmem/leapfrog_loc.F:

• the outer loop increments counter iswloop with values : 1 to nsloops.
• the middle loop increments counter kswr with values : 1 to mswr.
• the inside loop increments counter itau with values : (iswloop - 1) * ntsinswr to iswloop * ntsinswr

As for NEMO if you run a 5 day simulation with a coupling window of 1 day and a Schwarz iteration number of 6. You get nsloops = 5 and mswr = 6. The value of ntsinswr depends on your time step. It is the number of dynamical time steps during a coupling window.

The loop is implemented around the time stepping scheme.

The beginning of the loops is before the 1 CONTINUE command:

c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

      iswloop = 1

      swzloop: DO WHILE ( iswloop <= nsloops )

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'leapfrog_loc'
         WRITE(lunout,*) '*** Schwarz loops ***'
         WRITE(lunout,*) 'iswloop = ',iswloop
      ENDIF
c$OMP END MASTER

      kswr = 1

      swzit: DO WHILE ( kswr <= mswr ) 

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'kswr = ',kswr
      ENDIF
c$OMP END MASTER

      IF ( kswr == 1 ) THEN
        CALL store_current_time_swz
        CALL dynredem1_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
        swz_store = .TRUE.
      ELSE
        CALL restore_current_time_swz
        CALL dynetat0_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
        swz_restore = .TRUE.
      ENDIF

      itau = (iswloop - 1) * ntsinswr   

c-----------------------------------------------------------------------
c   Debut de l'integration temporelle:
c   ----------------------------------
c et du parallelisme !!

   1  CONTINUE ! Matsuno Forward step begins here

The end of the loops is after the test for MATSUNO/LEAPFROG vs PURE_MATSUNO method:

      END IF ! of IF(.not.purmats)

c------------------------------
cend of schwarz loop
c------------------------------

      END DO swzit! kswr

      iswloop = iswloop + 1

      END DO swzloop! iswloop

      call fin_swz_dyn ! deallocate schwarz dynamics pointers
      call fin_swz_phy ! deallocate schwarz physics arrays

a condition to cycle the loop on kswr is added inside the MATSUNO/LEAPFROG condition:

c-----------------------------------------
c Test de la fin de la fenêtre de Schwarz
c-----------------------------------------

            IF ( itau == iswloop * ntsinswr .AND.
     &         .NOT.(itau == itaufin .AND. kswr == mswr ) ) THEN
              kswr = kswr + 1
              CYCLE swzit
            ENDIF

c-----------------------------------------------------------------------
c   gestion de l'integration temporelle:
c   ------------------------------------

            IF( MOD(itau,iperiod).EQ.0 )    THEN
                    GO TO 1
            ELSE IF ( MOD(itau-1,iperiod). EQ. 0 ) THEN

It works only if the number of steps in the coupling windows is commensurate with the Matsuno steps frequency.

The loop needs to initialise the Schwarz parameters before starting:

c-------------------------------------
c Initialization of schwarz parameters
c-------------------------------------

      CALL init_swz_dyn ! allocate schwarz dynamics pointers
      CALL init_swz_phy ! allocate schwarz physics arrays

!      ncplfrq  = 86400
      ntsinswr = ncplfrq / dtvr
      nsloops  = itaufin / ntsinswr

c$OMP MASTER
      IF (mpi_rank==0) THEN
         WRITE(lunout,*) 'leapfrog_loc, ncplfrq  : ',ncplfrq
         WRITE(lunout,*) '              ntsinswr : ',ntsinswr
         WRITE(lunout,*) '              nsloops  : ',nsloops
         WRITE(lunout,*) '              mswr     : ',mswr
      ENDIF
c$OMP END MASTER

c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

      iswloop = 1

Some modifications are done to the variables that indicate the last iteration to the routine phylmd/physiq_mod.F90.

c-----------------------------------------------------------------------
c   calcul des tendances physiques:
c   -------------------------------
c    ########   P.Le Van ( Modif le  6/02/95 )   ###########
c
       IF( purmats )  THEN
          IF( ( itau.EQ.itaufin.AND..NOT.forward )
     &       .AND. (kswr == mswr) ) lafin = .TRUE.
       ELSE
          IF( ( itau+1. EQ. itaufin )
     &       .AND. (kswr == mswr) ) lafin = .TRUE.
       ENDIF

storing and restoring

As for the NEMO implementation the routines that store the state of the code and restore it are grouped in the Schwarz module: dyn3dmem/schwarz.F90.

There are two sets of routines. Some for the dynamical state variables and some for the physical state variables.

The storing and restoring of the dynamical state is done at dyn3dmem/leapfrog_loc.F level.

The dynamical state storing and restoring is done in the routine physiq from module phylmd/physiq_mod.F90. It uses the logical variables swz_restore and swz_store from dyn3dmem/schwarz.F90 which are set in the Schwarz loop of dyn3dmem/leapfrog_loc.F. Look at the if section at the beginning of the Schwarz loop in the previous section on Schwarz loop implementation:

      IF ( kswr == 1 ) THEN

T

coding details

The Schwarz parameters set by the user in config.card are read in dyn3dmem/conf_gcm.F90.

The Schwarz parameters that are needed by the different subroutines, including the loops limits, are defined in dyn3d_common/control_mod.F90.

coupling with OASIS

validation procedure