Version 10 (modified by snguyen, 4 years ago) (diff)

--

# How the code for the Schwarz method implementation in LMDZ is organized

## Implementation of the Schwarz algorithm in LMDZ

### Time stepping and the Schwarz loop (similar to NEMO)

The schwarz loop is implemented in the dyn3dmem/leapfrog_loc.F procedure that is responsible for the time stepping.

As with NEMO there are three loops around the original time-stepping scheme of dyn3dmem/leapfrog_loc.F:

• the outer loop increments counter iswloop with values : 1 to nsloops.
• the middle loop increments counter kswr with values : 1 to mswr.
• the inside loop increments counter itau with values : (iswloop - 1) * ntsinswr to iswloop * ntsinswr

As for NEMO if you run a 5 day simulation with a coupling window of 1 day and a Schwarz iteration number of 6. You get nsloops = 5 and mswr = 6. The value of ntsinswr depends on your time step. It is the number of dynamical time steps during a coupling window.

The loop is implemented around the time stepping scheme.

The beginning of the loops is before the 1 CONTINUE command:

```c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

iswloop = 1

swzloop: DO WHILE ( iswloop <= nsloops )

c\$OMP MASTER
IF (mpi_rank==0) THEN
WRITE(lunout,*) 'leapfrog_loc'
WRITE(lunout,*) '*** Schwarz loops ***'
WRITE(lunout,*) 'iswloop = ',iswloop
ENDIF
c\$OMP END MASTER

kswr = 1

swzit: DO WHILE ( kswr <= mswr )

c\$OMP MASTER
IF (mpi_rank==0) THEN
WRITE(lunout,*) 'kswr = ',kswr
ENDIF
c\$OMP END MASTER

IF ( kswr == 1 ) THEN
CALL store_current_time_swz
CALL dynredem1_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
swz_store = .TRUE.
ELSE
CALL restore_current_time_swz
CALL dynetat0_swz(vcov,ucov,teta,q,masse,ps,phis,p,pks,pk,pkf)
swz_restore = .TRUE.
ENDIF

itau = (iswloop - 1) * ntsinswr

c-----------------------------------------------------------------------
c   Debut de l'integration temporelle:
c   ----------------------------------
c et du parallelisme !!

1  CONTINUE ! Matsuno Forward step begins here
```

The end of the loops is after the test for MATSUNO/LEAPFROG vs PURE_MATSUNO method:

```      END IF ! of IF(.not.purmats)

c------------------------------
cend of schwarz loop
c------------------------------

END DO swzit! kswr

iswloop = iswloop + 1

END DO swzloop! iswloop

call fin_swz_dyn ! deallocate schwarz dynamics pointers
call fin_swz_phy ! deallocate schwarz physics arrays
```

a condition to cycle the loop on kswr is added inside the MATSUNO/LEAPFROG condition:

```c-----------------------------------------
c Test de la fin de la fenêtre de Schwarz
c-----------------------------------------

IF ( itau == iswloop * ntsinswr .AND.
&         .NOT.(itau == itaufin .AND. kswr == mswr ) ) THEN
kswr = kswr + 1
CYCLE swzit
ENDIF

c-----------------------------------------------------------------------
c   gestion de l'integration temporelle:
c   ------------------------------------

IF( MOD(itau,iperiod).EQ.0 )    THEN
GO TO 1
ELSE IF ( MOD(itau-1,iperiod). EQ. 0 ) THEN
```

It works only if the number of steps in the coupling windows is commensurate with the Matsuno steps frequency.

The loop needs to initialise the Schwarz parameters before starting:

```c-------------------------------------
c Initialization of schwarz parameters
c-------------------------------------

CALL init_swz_dyn ! allocate schwarz dynamics pointers
CALL init_swz_phy ! allocate schwarz physics arrays

!      ncplfrq  = 86400
ntsinswr = ncplfrq / dtvr
nsloops  = itaufin / ntsinswr

c\$OMP MASTER
IF (mpi_rank==0) THEN
WRITE(lunout,*) 'leapfrog_loc, ncplfrq  : ',ncplfrq
WRITE(lunout,*) '              ntsinswr : ',ntsinswr
WRITE(lunout,*) '              nsloops  : ',nsloops
WRITE(lunout,*) '              mswr     : ',mswr
ENDIF
c\$OMP END MASTER

c-----------------------------------
c  Beginning of schwarz looping
c-----------------------------------

iswloop = 1
```

Some modifications are done to the variables that indicate the last iteration to the routine phylmd/physiq_mod.F90.

```c-----------------------------------------------------------------------
c   calcul des tendances physiques:
c   -------------------------------
c    ########   P.Le Van ( Modif le  6/02/95 )   ###########
c
IF( purmats )  THEN
IF( ( itau.EQ.itaufin.AND..NOT.forward )
&       .AND. (kswr == mswr) ) lafin = .TRUE.
ELSE
IF( ( itau+1. EQ. itaufin )
&       .AND. (kswr == mswr) ) lafin = .TRUE.
ENDIF

```

### storing and restoring

As for the NEMO implementation the routines that store the state of the code and restore it are grouped in the Schwarz module: dyn3dmem/schwarz.F90.

There are two sets of routines. Some for the dynamical state variables and some for the physical state variables.

The storing and restoring of the dynamical state is done at dyn3dmem/leapfrog_loc.F level.

The physical state storing and restoring is done in the routine physiq from module phylmd/physiq_mod.F90. It uses the logical variables swz_restore and swz_store from dyn3dmem/schwarz.F90 which are set in the Schwarz loop of dyn3dmem/leapfrog_loc.F. Look at the if section at the beginning of the Schwarz loop in the previous section on time stepping implementation:

```      IF ( kswr == 1 ) THEN
```

The storing and restoring of the physical state is partly implemented in these files:

```phylmd/
fonte_neige_mod.F90
pbl_surface_mod.F90
traclmdz_mod.F90
```

Note that the Schwarz variables used for storing and restoring have the same name as their original counterpart with an s appended at the end.

## coding details

The Schwarz parameters set by the user in config.card are read in :

```dyn3dmem/conf_gcm.F90
```

The Schwarz parameters that are needed by the different subroutines, including the loops limits, are defined in:

```dyn3d_common/control_mod.F90
```

The dynamic variables needed for storing and restoring are declared in:

```dyn3dmem/schwarz.F90
```

The physics variables are declared in:

```dyn3dmem/schwarz.F90
phylmd/phys_state_var_mod.F90
```

As for the NEMO implementation the outputs are modified to only happen at kswr=ksout in:

```phylmd/
iophy.F90
phys_output_write_mod.F90
```

Note that the "xios calendar" is only updated when kswr=ksout in phylmd/phys_output_write_mod.F90:

```    DO iinit=1, iinitend
!      print *,'IFF iinit=', iinit, iinitend
#ifdef CPP_XIOS
!\$OMP MASTER
IF (vars_defined) THEN
IF (prt_level >= 10) then
write(lunout,*)"phys_output_write: call xios_update_calendar, itau_w=",itau_w
ENDIF
!          CALL xios_update_calendar(itau_w)
IF (kswr == ksout) THEN
CALL xios_update_calendar(itap)
ENDIF
ENDIF
!\$OMP END MASTER
!\$OMP BARRIER
#endif
```

## coupling with OASIS

The coupling with OASIS is done by adding an "absolute physical time stepping counter" itap_swz that is used for sending and receiving fields via OASIS in the following files:

```change_srf_frac_mod.F90
surf_ocean_mod.F90
surf_seaice_mod.F90
```

## validation procedure

The LMDZ implementation of Schwarz looping has been validated on a forced version where the repeating of the Schwarz window was found to be identical to a reference simulation.