Changes between Version 8 and Version 9 of 2020WP/HPC-02_Daley_Tiling

Timestamp:

2020-09-24T20:06:30+02:00 (4 years ago)

Author:

hadcv

Comment:

--

2020WP/HPC-02_Daley_Tiling

@@ -26 +26 @@
 === Implementation
 
-The current approach to tiling is described below. A document describing the issues encountered to date are described in [https://forge.ipsl.jussieu.fr/nemo/attachment/wiki/2020WP/HPC-02_Daley_Tiling/Tiling_code_issues.pdf this document].
-
-Several modules have been tiled as of 18/06/20: `tra_ldf`, `tra_zdf`, `tra_adv` and `dia_ptr`.
-
-The tiling implementation has been tested using GYRE with 1 CPU. The tests comprise 10 day simulations using different tile decompositions (including no tiling) and different science options particular to the tiled modules. A test passes if the tiling does not change results at the bit level (`run.stat`) or in the diagnostics.
+As of 24/09/20, most of the code called by the "active tracers" part of the step subroutine (between `trc_stp` and `tra_atf`) has been tiled. Solutions and workarounds for the issues encountered to date are described in [https://forge.ipsl.jussieu.fr/nemo/attachment/wiki/2020WP/HPC-02_Daley_Tiling/Tiling_code_issues.pdf this document].
+
+The tiling implementation has been tested using GYRE in benchmark mode with mono-processor and MPI configurations. The tests comprise 10 day simulations using different tile decompositions (including no tiling) and different science options particular to the tiled modules. A test passes if the tiling does not change results at the bit level (`run.stat`) or in the diagnostics. 
 
 __Summary of method__
@@ -43 +41 @@
   {{{
   #!diff
-  - #define __kIs_     2
-  + #define __kIs_     ntsi
+  - #define DO_2D(B, T, L, R) DO jj = Njs0-(B), Nje0+(T)   ;   DO ji = Nis0-(L), Nie0+(R)
+  + #define DO_2D(B, T, L, R) DO jj = ntsj-(B), ntej+(T)   ;   DO ji = ntsi-(L), ntei+(R)
   }}}
 
@@ -53 +51 @@
   {{{
   #!fortran
-  ntsi_a(0) = 1 + nn_hls
-  ntsj_a(0) = 1 + nn_hls
-  ntei_a(0) = jpi - nn_hls
-  ntej_a(0) = jpj - nn_hls
-  }}}
-
-  `dom_tile` is called whenever the active tile needs to be set or if tiling needs to be suppressed:
+  ntsi_a(0) = Nis0
+  ntsj_a(0) = Njs0
+  ntei_a(0) = Nie0
+  ntej_a(0) = Nje0
+  }}}
+
+  `dom_tile` is called whenever the active tile needs to be set or if tiling needs to be disabled:
 
   {{{
@@ -69 +67 @@
 '''2. Declaring SUBROUTINE-level arrays using the tile bounds'''
 
-  A new substitution macro in `do_loop_substitute.h90`:
-
-  {{{
-  #define A2D        __kIsm1_:__kIep1_,__kJsm1_:__kJep1_
-  }}}
-
-  is used such that:
+  A new set of substitution macros in `do_loop_substitute.h90`:
+
+  {{{
+  #define ST_1Di(H) ntsi-H:ntei+H
+  #define ST_1Dj(H) ntsj-H:ntej+H
+  #define ST_2D(H) ST_1Di(H),ST_1Dj(H)
+  }}}
+
+  replaces references to the full domain in explicit shape and allocatable array declarations:
 
   {{{
   #!diff
-  - ALLOCATE(jpi,jpj)   DIMENSION(jpi,jpj)
-  + ALLOCATE(A2D)       DIMENSION(A2D)
-  }}}
-
-  and therefore operations between local working arrays (which have the dimensions of the tile) and global/input arrays (which have the dimensions of either the tile or full domain) require no further changes, unless using `:` subscripts as described below.
-
-'''3. Replacing `:` subscripts with a DO loop macro where appropriate'''
-
-  This is only necessary when step 2 would introduce conformance issues:
-
-  {{{
-  #!diff
-  - REAL(wp), DIMENSION(jpi,jpj,jpk) :: a3d
-  - REAL(wp), DIMENSION(jpi,jpj)     :: z2d
-  - z2d(:,:) = a3d(:,:,1).
-  + REAL(wp), DIMENSION(jpi,jpj,jpk) :: a3d
-  + REAL(wp), DIMENSION(A2D)         :: z2d
-  + DO_2D_11_11
-  +    z2d(ji,jj) = a3d(ji,jj,1)
-  + END_2D
-  }}}
-
-'''4. Looping over tiles at the timestepping level'''
+  - ALLOCATE(jpi,jpj      ) DIMENSION(jpi,jpj      )
+  + ALLOCATE(ST_2D(nn_hls)) DIMENSION(ST_2D(nn_hls))
+  }}}
+
+  These arrays then have the same dimensions as the tile if tiling is used, otherwise they will have the same dimensions as the full domain as before. Furthermore, the tile-sized arrays are declared with lower and upper bounds corresponding to the position of the tile in the full domain. Horizontal indices, for example in DO loops, will therefore apply to both tile- and full-sized arrays:
+
+  {{{
+  #!fortran
+  ! ntsi = 3, ntsj = 7, ntei = 5, ntej = 9
+  REAL(wp), DIMENSION(ntsi:ntei,ntsj:ntej) :: z2d
+  REAL(wp), DIMENSION(jpi,jpj) :: a2d
+
+  DO_2D(1,1,1,1)
+    z2d(ji,jj) = a2d(ji,jj)
+  END_2D
+  }}}
+
+  This substitution is made for local working arrays where possible to minimise memory consumption when using tiling. 
+  No further changes are generally required, except in specific cases described in [https://forge.ipsl.jussieu.fr/nemo/attachment/wiki/2020WP/HPC-02_Daley_Tiling/Tiling_code_issues.pdf this document] and other common cases described in steps 5 & 6 below.
+  
+'''3. Looping over tiles at the timestepping level'''
 
   A loop over tiles has been added to `stp`. The domain indices for the current tile (`ntile /= 0`) are set at the start of each iteration. After exiting the loop (and before, during initialisation) the tiling is suppressed (`ntile == 0`):
@@ -110 +108 @@
   DO jtile = 1, nijtile
      IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=jtile )
+
      CALL tra_ldf( kstp, Nbb, Nnn, ts, Nrhs )  ! lateral mixing
   END DO
+
   IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=0 )        ! Revert to full domain
   }}}
 
-  DO loops within the tiling loop therefore work on the current tile, while those outside the loop work on the full domain.
-
-'''5. A new namelist (`namtile`)'''
+  DO loops within the tiling loop therefore work on the current tile, while those outside the tiling loop work on the full domain.
+
+'''4. A new namelist (`namtile`)'''
 
 {{{
@@ -131 +131 @@
   The number of tiles is calculated from the tile lengths, `nn_ltile_i` and `nn_ltile_j`, with respect to the full domain.
 
+'''5. Replacing `:` subscripts with a DO loop macro where appropriate'''
+
+  This is only necessary when step 2 would introduce conformance issues:
+
+  {{{
+  #!diff
+  - REAL(wp), DIMENSION(jpi,jpj,jpk)   :: a3d
+  - REAL(wp), DIMENSION(jpi,jpj)       :: z2d
+  - z2d(:,:) = a3d(:,:,1).
+  + REAL(wp), DIMENSION(jpi,jpj,jpk)   :: a3d
+  + REAL(wp), DIMENSION(ST_2D(nn_hls)) :: z2d
+  + DO_2D(1,1,1,1)
+  +    z2d(ji,jj) = a3d(ji,jj,1)
+  + END_2D
+  }}}
+
+'''6. Suppressing code that should not be called more than once per timestep'''
+
+  Examples include ocean.output write statements and initialisation steps outside of an "_ini" routine.
+
 __Branch__
 
-''These branches contain a trial implementation of tiling in `tra_ldf_iso`; there is not yet a formal branch for the development.''
-
-[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12745_HPC-02_Daley_Tiling_trial_public&old=12740%40NEMO%2Ftrunk Implementation in trunk]
-
-[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo Implementation in extended haloes branch]
+[http://forge.ipsl.jussieu.fr/nemo/browser/NEMO/branches/2020/dev_r13383_HPC-02_Daley_Tiling]
 
 __New subroutines__
@@ -148 +164 @@
 * `OCE/DOM/dom_oce`- Declare tiling namelist and other tiling variables
 * `OCE/DOM/domain`- Read `namtile` namelist (`dom_nam`), calculate tiling variables and do control print (`dom_tile`)
-* `OCE/IOM/prtctl`- Add IF statement to prevent execution of `prt_ctl` by each tile
-* `OCE/TRA/traldf`- Add IF statements to prevent execution of `trd_tra` by each tile
-* `OCE/TRA/traldf_iso`- Add IF statements (as above), modify local arrays for tiling
-* `OCE/do_loop_substitute`- Modify DO loop macros to use domain indices, add `A2D` macro
+* `OCE/DOM/domutl`- `is_tile` functions
+* `OCE/do_loop_substitute`- Modify DO loop macro to use domain indices, add CPP macros
 * `OCE/par_oce`- Declare tiling variables
 * `OCE/step`- Add tiling loop
 * `OCE/step_oce`- Add USE statement for `dom_tile` in `step`
-* `OCE/timing`- Add IF statements to prevent execution of `timing_start` and `timing_stop` by each tile
+* Various others..
 
 __New variables (excluding local)__
@@ -166 +180 @@
   * `ntile`- current tile number
   * `nijtile`- number of tiles
-* Namelist
+* Namelist (`namtile`)
   * `ln_tile`- logical control on use of tiling
   * `nn_ltile_i`, `nn_ltile_j`- tile length
 * Pre-processor macros
-  * `A2D`- substitution for ALLOCATE or DIMENSION arguments
-
-__Notes__
-
-'''Issues with the tiling implementation'''
-
-See the attached [https://forge.ipsl.jussieu.fr/nemo/attachment/wiki/2020WP/HPC-02_Daley_Tiling/Tiling_code_issues.pdf document].
-
-'''Extended haloes'''
-
-The tiling trial has also been implemented in the [http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo extended haloes branch].
-There are few differences between this and the trunk implementation.
+  * `ST_*D`- substitutions for ALLOCATE or DIMENSION arguments
+  * `ST_*DT`- substitutions for ALLOCATE or DIMENSION arguments when the shape of the array is unknown
+* Functions
+  * `is_tile`- Returns 0 if the array has the dimensions of the full domain, else 1
 
 === Documentation updates