Version 1 (modified by snguyen, 4 years ago) (diff)


How the code for the Schwarz method implementation in NEMO is organized

Implementation of the Schwarz method

The time stepping loop

The essential part to understand is how the original time stepping loop of NEMO in nemogcm.F90 is modified to allow rewinding of the state of NEMO after a coupling window has passed.

Initially the code has one loop over the command CALL stp( istp ) for each time step.

With the Schwarz algorithm, the total amount of time steps is divided in coupling windows (called Schwarz loops in the code), each coupling window is repeated the number of schwarz iterations given as a parameter (mswr), and for each schwarz iteration a number of time steps (the length of a coupling window) is done.

For a computation of 5 days with 15 time steps per day, a coupling window of 1 day and 5 schwarz iterations (mswr=5) there will be 3 loops.

  • One global loop over the coupling windows : 5 to iterate. iswloop
  • One inner loop over the number of schwarz iteration for each coupling window : 5 (mswr=5). kswr
  • One most inner loop over the time steps done in one Schwarz coupling window : 15 time steps. istp

The counter given to stp is rewind when the Schwarz coupling window is repeated.

This is coded like this in nemogcm.F90:

         iswloop = 1
         IF (lwp) WRITE(numout,*) 'init iswloop =',iswloop

         DO WHILE ( iswloop <= nsloops .AND. nstop == 0 ) ! iterate on n schwarz loops

         kswr = 1
         IF (lwp) THEN 
            WRITE(numout,*) '*** schwarz loops ***'
            WRITE(numout,*) 'iswloop =',iswloop
            WRITE(numout,*) 'kswr = ',kswr
         CALL swz_store  ! store Ocean state at first schwarz iteration before first time step
         CALL swz_store_lim3 ! store lim3 variables state
         CALL swz_store_limdiahsb ! store limdiahsb variables
         IF(lwp) WRITE(numout,*) 'store schwarz initial state '

         DO WHILE ( kswr <= mswr .AND. nstop == 0 )

         IF (lwp) WRITE(numout,*) '  kswr = ',kswr
         istp = nit000 + (iswloop-1)*ntsinswr ! set time step to first value for iswloop (current schwarz loop)

         IF ( kswr > 1 ) CALL swz_reinit(istp) ! reset Ocean state to first time step for new schwarz iteration 

         IF (lwp) WRITE(numout,*) 'maxistp  = ',nit000+iswloop*ntsinswr-1
         DO WHILE ( istp <= nit000+iswloop*ntsinswr-1 .AND. nstop == 0 )

            IF (lwp) WRITE(numout,*) '    istp = ',istp,';    istpswz = ',istpswz 
#if defined key_agrif
            CALL stp                         ! AGRIF: time stepping
            CALL stp( istp )                 ! standard time stepping
            istp = istp + 1
            istpswz = istpswz + 1
            IF( lk_mpp )   CALL mpp_max( nstop )
         END DO ! istp
         kswr = kswr + 1

         END DO ! kswr

         iswloop = iswloop +1

         END DO ! iswloop

The variable iswloop is the current Schwarz loop and nsloops is the number of Schwarz loops for the given time of simulation and Schwarz window size. For 5 days with a window of 1 day nsloops=5.

Inside the Schwarz repeating loop DO WHILE ( kswr <= mswr .AND. nstop == 0 ), in the time stepping loop, istp starts at the value nit000 + (iswloop-1)*ntsinswr and finishes at nit000+iswloop*ntsinswr-1.

This allows istp to take the value it would have for the current Schwarz window if no sub-iteration was done: setting mswr=1 gives you the original time stepping.

The Schwarz parameters are initialized (or computed) before entering the time loops, before and after the call to nemo_init depending on what is needed:

      ntsinswr = 1 !! set to allow fldread in nemo_init before computation of schwarz indices
      kswr = 0     !! set here to allow day_init to set nitrst properly

      WRITE(numout,*) "ntsinswr = ",ntsinswr

      !                            !-----------------------!
      CALL nemo_init               !==  Initialisations  ==!
      !                            !-----------------------!

      !! set integer constants for computation
!      ncplfrq  = 86400 !cpl_freq( 'O_QnsMix' ) read in namrun (nemoinit)
      ntsinswr = ncplfrq / INT(rdt)
      nsloops  = (nitend-nit000+1)/ntsinswr

      !! outputs to check values
      IF(lwp) THEN                  ! control print
         WRITE(numout,*) ' computation of loop indices for schwarz computation '
         WRITE(numout,*) '   nit000   = ',nit000
         WRITE(numout,*) '   nitend   = ',nitend
         WRITE(numout,*) '   rdt      = ',rdt
         WRITE(numout,*) '   ncplfrq  = ',ncplfrq
         WRITE(numout,*) '   ntsinswr = ',ntsinswr
         WRITE(numout,*) '   nsloops  = ',nsloops 
         WRITE(numout,*) '   mswr     = ',mswr
         WRITE(numout,*) '   nitrst   = ',nitrst

There is also an "absolute" time step counter : istpswz which is incremented at each time step including those repeated for the Schwarz algorithm. It is needed for the coupling with OASIS.

The Schwarz algorithm: storing and restoring

The Schwarz algorithm is implemented by storing and restoring the state of the ocean and ice at the proper times.

The routines used to store and restore a state are all in schwarz.F90 which is a new subroutine.

The storing is done just before the schwarz looping is done: inside iswloop loop, before kswr loop. Like this we are sure to store the initial condition of a schwarz loop.