New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
ticket/0664_TRA_TRP – NEMO
wiki:ticket/0664_TRA_TRP

Version 21 (modified by cetlod, 14 years ago) (diff)

--

Last edited Timestamp?

Author : Christian Ethé

ticket : #664

Branch : DEV_r2006_merge_TRA_TRC


Description

Motivations

System simplification : Merge of active and passive tracer advection/diffusion modules to avoid duplication of almost identical modules

Strategy

(1) Creation of generic advection/diffusion modules

A generic routine will be able to work with either active or passive tracers. It will therefore use 4D tracer arrays that will be given in the argument of the routine. The 3 fields, before, now and after are systematically provided in argument even if some all are not necessary used in a given scheme.

in diffusion modules (traldf..., trazdf...) : 4D tracer are given through input argument, where the 4th dimension is the tracer type (=2 for temp. and salinity in the active tracer case, =jptrc in the passive tracer one)

in advection module (traadv...) : both the 4D tracer and transport components (e2u*e3u*u,...) are given through input argument.

for example, for TVD advection the routine start as follows:

   SUBROUTINE tra_adv_tvd ( kt, cdtype, p2dt, pun, pvn, pwn,   &
      &                                       ptb, ptn, pta, kjpt   )
      !!----------------------------------------------------------------------
      !!                  ***  ROUTINE tra_adv_cen2  ***
      !!
      !! ** Purpose :
      !!
      !! ** Method  : 
      !!----------------------------------------------------------------------
      INTEGER         , INTENT(in   )                               ::   kt              ! ocean time-step index
      CHARACTER(len=3), INTENT(in   )                               ::   cdtype          ! =TRA or TRC (tracer indicator)
      INTEGER         , INTENT(in   )                               ::   kjpt            ! number of tracers
      REAL(wp)        , INTENT(in   ), DIMENSION(        jpk     )  ::   p2dt            ! vertical profile of tracer time-step
      REAL(wp)        , INTENT(in   ), DIMENSION(jpi,jpj,jpk     )  ::   pun, pvn, pwn   ! 3 ocean transport components
      REAL(wp)        , INTENT(in   ), DIMENSION(jpi,jpj,jpk,kjpt)  ::   ptrab, ptran    ! before and now tracer fields
      REAL(wp)        , INTENT(inout), DIMENSION(jpi,jpj,jpk,kjpt)  ::   ptraa           ! tracer trend


(2) Test for T & S 3D arrays

In order to test the new tra... modules without changing the temperature and salinity field every where in the code, we replace T and S 3D arrays by 4D array just within the TRA modules in the call of the new routine. For example, in the step routine

   SUBROUTINE stp( kstp )
      INTEGER, INTENT(in) ::   kstp   ! ocean time-step index

      !+++ Initialisation phase 

      CALL tra_swap 

      !++++

       bla bla bla bla bla bla
       bla bla bla bla bla bla

      !>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      ! Active tracers                              (ua, va used as workspace)
      !<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

                             tsa(:,:,:,:) = 0.e0               ! set tracer trends to zero

                             CALL tra_sbc    ( kstp )       ! surface boundary condition
      IF( ln_traqsr      )   CALL tra_qsr    ( kstp )       ! penetrative solar radiation qsr
      IF( lk_trabbc      )   CALL tra_bbc    ( kstp )       ! bottom heat flux
      IF( lk_trabbl      )   CALL tra_bbl    ( kstp )       ! advective (and/or diffusive) bottom boundary layer scheme
      IF( lk_tradmp      )   CALL tra_dmp    ( kstp )       ! internal damping trends
                             CALL tra_adv    ( kstp )       ! horizontal & vertical advection
      IF( n_cla == 1     )   CALL tra_cla    ( kstp )       ! Cross Land Advection (Update Hor. advection)
      IF( lk_zdfkpp )        CALL tra_kpp    ( kstp )       ! KPP non-local tracer fluxes
                             CALL tra_ldf    ( kstp )       ! lateral mixing
#if defined key_agrif
                             CALL tra_unswap
      IF(.NOT. Agrif_Root()) CALL Agrif_Sponge_tra          ! tracers sponge
                             CALL tra_swap
#endif
                             CALL tra_zdf    ( kstp )       ! vertical mixing and after tracer fields

      IF( ln_dynhpg_imp  ) THEN                             ! semi-implicit hpg (time stepping then eos)
         IF( ln_zdfnpc   )   CALL tra_npc    ( kstp )            ! update after fields by non-penetrative convection
                             CALL tra_nxt    ( kstp )            ! tracer fields at next time step
                             CALL eos( tsa, rhd, rhop )       ! Time-filtered in situ density for hpg computation
         IF( ln_zps      )   CALL zps_hde( kstp, jpts, tsa, gtsu, gtsv,  &    ! zps: time filtered hor. derivative
            &                                          rhd, gru , grv  )      ! of t, s, rd at the last ocean level

      ELSE                                                  ! centered hpg  (eos then time stepping)
                             CALL eos( tsn, rhd, rhop )       ! now in situ density for hpg computation
         IF( ln_zps      )   CALL zps_hde( kstp, jpts, tsn, gtsu, gtsv,  &    ! zps: now hor. derivative
            &                                          rhd, gru , grv  )      ! of t, s, rd at the last ocean level
         IF( ln_zdfnpc   )   CALL tra_npc    ( kstp )       ! update after fields by non-penetrative convection
                             CALL tra_nxt    ( kstp )       ! tracer fields at next time step
      ENDIF
                             CALL tra_unswap


       bla bla bla bla bla bla
       bla bla bla bla bla bla

The module traswap.F90 is created and contains 2 routines which (un)swap the 3D T & S arrays onto 4D arrays

MODULE traswp
   !!==============================================================================
   !!                       ***  MODULE  traswp  ***
   !! Ocean active tracers: swapping array 
   !!==============================================================================
   USE par_oce
   USE oce             ! ocean dynamics and active tracers

   IMPLICIT NONE
   PRIVATE

   PUBLIC   tra_swap    ! routine called by step.F90
   PUBLIC   tra_unswap    ! routine called by step.F90


CONTAINS

   SUBROUTINE tra_swap
      !!----------------------------------------------------------------------
      !!                  ***  ROUTINE tra_swp  ***
      !!                   
      !! ** Purpose : Store temperature and salinity aaray into a 4D array 
      !!
      !!----------------------------------------------------------------------

      tsn(:,:,:,jp_tem) = tn(:,:,:)      ;      tsn(:,:,:,jp_sal) = sn(:,:,:)
      tsb(:,:,:,jp_tem) = tb(:,:,:)      ;      tsb(:,:,:,jp_sal) = sb(:,:,:)
      tsa(:,:,:,jp_tem) = ta(:,:,:)      ;      tsa(:,:,:,jp_sal) = sa(:,:,:)

   END SUBROUTINE tra_swap

   SUBROUTINE tra_unswap
      !!----------------------------------------------------------------------
      !!                  ***  ROUTINE tra_unswap  ***
      !!                   
      !! ** Purpose : Store temperature and salinity aaray into a 4D array 
      !!
      !!----------------------------------------------------------------------

      tn(:,:,:) = tsn(:,:,:,jp_tem)      ;      sn(:,:,:) = tsn(:,:,:,jp_sal)
      tb(:,:,:) = tsb(:,:,:,jp_tem)      ;      sb(:,:,:) = tsb(:,:,:,jp_sal)
      ta(:,:,:) = tsa(:,:,:,jp_tem)      ;      sa(:,:,:) = tsa(:,:,:,jp_sal)

   END SUBROUTINE tra_unswap

   !!======================================================================
END MODULE traswp

For example the call in tra_adv becomes :

   SUBROUTINE tra_adv( kt )
      !!----------------------------------------------------------------------
      !!                  ***  ROUTINE tra_adv  ***
      !!
      !! ** Purpose :  
      !!
      !! ** Method  : 
      !!----------------------------------------------------------------------
      INTEGER, INTENT( in ) ::   kt   ! ocean time-step index
      !!
      INTEGER ::   jk   ! dummy loop index
      REAL(wp), DIMENSION(jpi,jpj,jpk) ::   zun, zvn, zwn   ! effective transports
      !!----------------------------------------------------------------------

      DO jk = 1, jpkm1
         !                                                ! eulerian transport only
         zun(:,:,jk) = e2u(:,:) * fse3u(:,:,jk) * un(:,:,jk)
         zvn(:,:,jk) = e1v(:,:) * fse3v(:,:,jk) * vn(:,:,jk)
         zwn(:,:,jk) = e1t(:,:) * e2t(:,:)      * wn(:,:,jk)
         !
      END DO
      zwn(:,:,jpk) = 0.e0                                 ! no transport trough the bottom

      !                                                   ! add the eiv transport (if necessary)
      IF( lk_traldf_eiv )   CALL tra_adv_eiv( kt, zun, zvn, zwn, 'TRA' )
      CALL tra_adv_tvd( kt, 'TRA', r2dt, zun, zvn, zwn, tsb, tsn, tsa, jpts )   !  TVD 

N.B. the transport given in argument is now the effective transport, i.e. including the one associated with eddy induced velocity, or bbl.


N.B : We've first planned to use the fortran RESHAPE function for tests on T & S tracers. This function allowed to create a 4D array by merging two 3D arrays : RESHAPE( tb, (/jpi,jpj,jpk,2/), sb ). But, the performances of vectorisation when using the RESHAPE function are very poor - tests have been done on CCRT NEC-SX8 computer


(3) Apply to the passive tracers

with the 4D tracer arguments, the same module are use for active and passive tracers. The trcadv module becomes:

   SUBROUTINE trc_adv( kt )
      !!----------------------------------------------------------------------
      !!                  ***  ROUTINE trc_adv  ***
      !!
      !! ** Purpose :  
      !!
      !! ** Method  : 
      !!----------------------------------------------------------------------
      INTEGER, INTENT( in ) ::   kt   ! ocean time-step index
      !!
      INTEGER ::   jk   ! dummy loop index
      REAL(wp), DIMENSION(jpi,jpj,jpk) ::   zun, zvn, zwn   ! effective transports
      !!----------------------------------------------------------------------

      DO jk = 1, jpkm1
         !                                                ! eulerian transport only
         zun(:,:,jk) = e2u(:,:) * fse3u(:,:,jk) * un(:,:,jk)
         zvn(:,:,jk) = e1v(:,:) * fse3v(:,:,jk) * vn(:,:,jk)
         zwn(:,:,jk) = e1t(:,:) * e2t(:,:)      * wn(:,:,jk)
         !
      END DO
      zwn(:,:,jpk) = 0.e0                                 ! no transport trough the bottom

      !                                                   ! add the eiv transport (if necessary)
      IF( lk_traldf_eiv )   CALL tra_adv_eiv( kt, zun, zvn, zwn, 'TRC' )

      CALL tra_adv_tvd( kt, 'TRC', r2dt, zun, zvn, zwn, trb, trn, tra, jptra )    ! 2nd order centered


(4) Final step : replace T and S 3D arrays by 4D arrays TS throughout the code

  • This action is only depends now on the replacement of these 3D by 4D arrays in OBC, BDY and AGRIF

Additional Tasks

  • TRENDS : As a starting point, both active & passive tracers trends modules ( trdmld.F90 & trdmld_trc.F90 ) will be encapsulate in one module
  • Re-organisation of the initialisation phase : The initialisation phase of each physical processes which was done in the associated module is now moved in opa_init
  • Re-organisation of OFFLINE component : The Offline part (OFF_SRC) of the code is now used the same routines of the online (OPA_SRC) + some additional
  • Open boundaries conditions ( OBC and BDY ) are not merged and still not available for passive tracers

Testing

Testing could consider (where appropriate) other configurations in addition to NVTK].

NVTK Tested'''YES/NO'''
Other model configurations'''YES/NO'''
Processor configurations tested[ Enter processor configs tested here ]
If adding new functionality please confirm that the
New code doesn't change results when it is switched off
and ''works'' when switched on
'''YES/NO/NA'''

(Answering UNSURE is likely to generate further questions from reviewers.)

'Please add further summary details here'

Processor configurations tested : ORCA2_LIM, ORCA2_LIM_PISCES and GYRE_LOBSTER on 1 single processor and 1x4 processors

  • Restartability is OK for all the 3 configurations
  • Reproductiblity is OK for the 3 configurations

NB : To test reproductibility, add the CPP key key_mpp_rep1 when compiling both in mono and multi processors runs

  • etc----

Bit Comparability

Does this change preserve answers in your tested standard configurations (to the last bit) ?'''YES/NO '''
Does this change bit compare across various processor configurations. (1xM, Nx1 and MxN are recommended)'''YES/NO'''
Is this change expected to preserve answers in all possible model configurations?'''YES/NO'''
Is this change expected to preserve all diagnostics?
,,''Preserving answers in model runs does not necessarily imply preserved diagnostics. ''
'''YES/NO'''

If you answered '''NO''' to any of the above, please provide further details:

  • Which routine(s) are causing the difference?
  • Why the changes are not protected by a logical switch or new section-version
  • What is needed to achieve regression with the previous model release (e.g. a regression branch, hand-edits etc). If this is not possible, explain why not.
  • What do you expect to see occur in the test harness jobs?
  • Which diagnostics have you altered and why have they changed?Please add details here........

System Changes

Does your change alter namelists?'''YES
Does your change require a change in compiler options?'''NO '''

If any of these apply, please document the changes required here.......


Resources

''Please ''summarize'' any changes in runtime or memory use caused by this change......''

  • Performances for ORCA2_LIM configuration for 1 year run on CCRT/NEC-SX8 machine : tvd scheme is used / yearly output
                               DEV_r1784_mid_year_merge_2010         DEV_r2006_merge_TRA_TRC  
    
      Real Time (sec)       :         1407.898499                          1339.563067  
      User Time (sec)       :         1226.085526                          1259.150240  
      Sys  Time (sec)       :           24.706370                            7.924059
      MOPS                  :        17138.798894                         17169.891895  
      MFLOPS                :         6630.166950                          6677.992531  
      A. V. Length          :          246.896814                           247.187704  
      V. Op. Ratio (%)      :           99.633384                            99.631952  
      Memory Size (MB)      :         1024.031250                          1008.031250  
    
  • Performances for ORCA2_LIM_PISCES configuration for 1 year run on CCRT/NEC-SX8 machine : tvd scheme is used for dynamics and muscl for pisces / yearly output
                                DEV_r1784_mid_year_merge_2010        DEV_r2006_merge_TRA_TRC
    
      Real Time (sec)       :         6953.589819                          6154.514676
      User Time (sec)       :         6734.863512                          5962.929426 
      Sys  Time (sec)       :           32.816470                            29.359322
      MOPS                  :        17200.156968                         17410.820487
      MFLOPS                :         6546.497939                          6520.701557
      A. V. Length          :          252.298456                           251.979919
      V. Op. Ratio (%)      :           99.640719                            99.664899
      Memory Size (MB)      :         2352.031250                          2512.031250
    
  • Performances for ORCA2_OFF_PISCES configuration for 1 year run on CCRT/NEC-SX8 machine : muscl for pisces / yearly output
                                DEV_r1784_mid_year_merge_2010        DEV_r2006_merge_TRA_TRC
    
      Real Time (sec)       :         2970.254295                          2271.950818                 
      User Time (sec)       :         2783.331507                          2105.422724 
      Sys  Time (sec)       :           25.955298                            43.583005
      MOPS                  :        14003.086260                         17359.565290
      MFLOPS                :         5247.593871                          6459.605436
      A. V. Length          :          181.676950                           252.387730
      V. Op. Ratio (%)      :           99.371104                            99.660649
      Memory Size (MB)      :         1920.031250                          2320.031250
    
    

IPR issues

Has the code been wholly (100%) produced by NEMO developers staff working exclusively on NEMO?'''YES/ NO '''

If No:

  • Identify the collaboration agreement details
  • Ensure the code routine header is in accordance with the agreement, (Copyright/Redistribution? etc).Add further details here if required..........