Changes between Version 6 and Version 7 of 2020WP/HPC-02_Daley_Tiling

Timestamp:

2020-06-18T15:50:18+02:00 (4 years ago)

Author:

hadcv

Comment:

--

2020WP/HPC-02_Daley_Tiling

@@ -26 +26 @@
 === Implementation
 
-A trial of horizontal tiles has been implemented in `tra_ldf_iso` and the main code changes are described. 
-
-This has been tested using GYRE with 1 CPU, without XIOS. 10 day simulations using different tile decompositions (including no tiling) have been bit compared against the trunk.
+The current approach to tiling is described below. More detailed notes, including a number of issues that prevent the full tiling of code are described in [??? this document].
+
+Several modules have been tiled as of 18/06/20: `tra_ldf`, `tra_zdf`, `tra_adv` and `dia_ptr`.
+
+The tiling implementation has been tested using GYRE with 1 CPU. The tests comprise 10 day simulations using different tile decompositions (including no tiling) and different science options particular to the tiled modules. A test passes if the tiling does not change results at the bit level (`run.stat`) or in the diagnostics.
 
 __Summary of method__
 
-The full processor domain (dimensions `jpi` x `jpj`) is split into one or more subdomains (tiles). 
+The full processor domain (dimensions `jpi` x `jpj`) is split into one or more tiles/subdomains. 
 This is implemented by: 
 
 '''1. Modifying the DO loop macros in `do_loop_substitute.h90` to use the tile bounds'''
 
-  The tile domain is defined by a new set of domain indices (`ntsi`, `ntei`, `ntsj`, `ntej`), which are arrays with lengths equal to the number of tiles (`nijtile`) plus one and represent the internal part of the domain. The tile number (`ntile`) is used to obtain the indices for the current tile:
+  The tile domain is defined by a new set of domain indices (`ntsi`, `ntei`, `ntsj`, `ntej`), which represent the internal part of the domain:
 
   {{{
   #!diff
   - #define __kIs_     2
-  + #define __kIs_     ntsi(ntile)
-  }}}
-
-  A new subroutine `dom_tile` (in `domain.F90`) sets the values of these indices during initialisation.
-  The zero index is used to store the indices for the full domain:
+  + #define __kIs_     ntsi
+  }}}
+
+  A new subroutine `dom_tile` (in `domain.F90`) sets the values of these indices. 
+
+  During initialisation, this subroutine calculates and stores the indices in global arrays (`ntsi_a`, `ntei_a`, `ntsj_a`, `ntej_a`) with lengths equal to the number of tiles (`nijtile`) plus one. The zero index is used to store the indices for the full domain:
 
   {{{
   #!fortran
-  ntsi(0) = 1 + nn_hls
-  ntsj(0) = 1 + nn_hls
-  ntei(0) = jpi - nn_hls
-  ntej(0) = jpj - nn_hls
+  ntsi_a(0) = 1 + nn_hls
+  ntsj_a(0) = 1 + nn_hls
+  ntei_a(0) = jpi - nn_hls
+  ntej_a(0) = jpj - nn_hls
+  }}}
+
+  `dom_tile` is called whenever the active tile needs to be set or if tiling needs to be suppressed:
+
+  {{{
+  #!fortran
+  CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=3 ) ! Work on tile 3
+  CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=0 ) ! Work on the full domain
   }}}
 
@@ -76 +87 @@
 '''3. Replacing `:` subscripts with a DO loop macro where appropriate'''
 
-  This is only necessary when step 2 would introduce an array shape inconsistency:
+  This is only necessary when step 2 would introduce conformance issues:
 
   {{{
@@ -92 +103 @@
 '''4. Looping over tiles at the timestepping level'''
 
-  The current tile number (`ntile`) is set within this loop in `stp`, then set to 0 after exiting the loop (and after initialisation, before the loop). 
+  A loop over tiles has been added to `stp`. The domain indices for the current tile (`ntile /= 0`) are set at the start of each iteration. After exiting the loop (and before, during initialisation) the tiling is suppressed (`ntile == 0`):
 
   {{{
@@ -98 +109 @@
   ! Loop over tile domains
   DO jtile = 1, nijtile
-     IF( ln_tile ) ntile = jtile
+     IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=jtile )
      CALL tra_ldf( kstp, Nbb, Nnn, ts, Nrhs )  ! lateral mixing
   END DO
-  IF( ln_tile ) ntile = 0                      ! Revert to tile over full domain
-  }}}
-
-  DO loops within the tiling loop therefore work on the current tile (`ntile /= 0`), while those outside the loop work on the full domain (`ntile == 0`).
+  IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=0 )        ! Revert to full domain
+  }}}
+
+  DO loops within the tiling loop therefore work on the current tile, while those outside the loop work on the full domain.
 
 '''5. A new namelist (`namtile`)'''
@@ -122 +133 @@
 __Branch__
 
-[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12945%40NEMO%2Fbranches%2FUKMO%2Fdev_r12745_HPC-02_Daley_Tiling_trial_public&old=12740%40NEMO%2Ftrunk Implementation in trunk]
-
-[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12942%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo Implementation in extended haloes branch]
+''These branches contain a trial implementation of tiling in `tra_ldf_iso`; there is not yet a formal branch for the development.''
+
+[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12745_HPC-02_Daley_Tiling_trial_public&old=12740%40NEMO%2Ftrunk Implementation in trunk]
+
+[http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo Implementation in extended haloes branch]
 
 __New subroutines__
 
-* `OCE/DOM/domain/dom_tile`- Calculate tiling variables (domain indices, number of tiles)
+* `OCE/DOM/domain/dom_tile`- Calculate/set tiling variables (domain indices, number of tiles)
 
 __Modified modules__
 
 * `cfgs/SHARED/namelist_ref`- Add `namtile` namelist
-* `OCE/DOM/dom_oce`- Declare namelist variables
-* `OCE/DOM/domain`- Read `namtile` namelist (`dom_nam`), calculate tiling variables (`dom_tile`)
+* `OCE/DOM/dom_oce`- Declare tiling namelist and other tiling variables
+* `OCE/DOM/domain`- Read `namtile` namelist (`dom_nam`), calculate tiling variables and do control print (`dom_tile`)
 * `OCE/IOM/prtctl`- Add IF statement to prevent execution of `prt_ctl` by each tile
 * `OCE/TRA/traldf`- Add IF statements to prevent execution of `trd_tra` by each tile
@@ -141 +154 @@
 * `OCE/par_oce`- Declare tiling variables
 * `OCE/step`- Add tiling loop
+* `OCE/step_oce`- Add USE statement for `dom_tile` in `step`
 * `OCE/timing`- Add IF statements to prevent execution of `timing_start` and `timing_stop` by each tile
 
@@ -148 +162 @@
   * `ntsi`, `ntsj`- start index of tile
   * `ntei`, `ntej`- end index of tile
+  * `ntsi_a`, `ntsj_a`- start indices of each tile
+  * `ntei_a`, `ntej_a`- end indices of each tile
   * `ntile`- current tile number
   * `nijtile`- number of tiles
@@ -158 +174 @@
 __Notes__
 
-'''Untiled code'''
-
-Parts of the code that should only be executed by one tile (e.g. `numout` write statements) as well as code that has not yet been tiled (e.g. timing routines) have been enclosed in IF statements.
-This code has been marked with `! TODO: TO BE TILED`.
-
-I will add some notes on this code in the near future.
+'''Untiled code and other issues'''
+
+See the attached [??? document].
 
 '''Extended haloes'''
 
-The tiling trial has also been implemented in the [http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12942%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo extended haloes branch].
+The tiling trial has also been implemented in the [http://fcm3/projects/NEMO.xm/changeset?reponame=&new=12979%40NEMO%2Fbranches%2FUKMO%2Fdev_r12866_HPC-02_Daley_Tiling_trial_extra_halo&old=12866%40NEMO%2Fbranches%2F2020%2Fdev_r12558_HPC-08_epico_Extra_Halo extended haloes branch].
 There are few differences between this and the trunk implementation.