Version 7 (modified by hadcv, 3 months ago) (diff)

Name and subject of the action

Last edition: 09/24/20 21:55:41 by hadcv

The PI is responsible to closely follow the progress of the action, and especially to contact NEMO project manager if the delay on preview (or review) are longer than the 2 weeks expected.

  1. Summary
  2. Preview
  3. Tests
  4. Review

Summary

Action Implement 2D tiling (with the LFRA version of NEMO)
PI(S) Daley Calvert, Andrew Coward
Digest Implement 2D tiling to reduce traffic between main memory and L3 cache
Dependencies DO loop macros (2020WP/KERNEL-02_Coward_Do Loop Macros_part1), extended haloes (Italo Epicoco, Seb Masson and Francesca Mele), extension of XIOS to accept 2D tiles of data (Yann Meurdesoif & Seb Masson)
Branch source:/NEMO/branches/{YEAR}/dev_r{REV}_{ACTION_NAME}
Previewer(s) Gurvan Madec
Reviewer(s) Gurvan Madec
Ticket #2365

Description

Implement loop tiling over horizontal dimensions (i and j).

Implementation

The current approach to tiling is described below. More detailed notes, including a number of issues that prevent the full tiling of code are described in this document.

Several modules have been tiled as of 18/06/20: tra_ldf, tra_zdf, tra_adv and dia_ptr.

The tiling implementation has been tested using GYRE with 1 CPU. The tests comprise 10 day simulations using different tile decompositions (including no tiling) and different science options particular to the tiled modules. A test passes if the tiling does not change results at the bit level (run.stat) or in the diagnostics.

Summary of method

The full processor domain (dimensions jpi x jpj) is split into one or more tiles/subdomains. This is implemented by:

1. Modifying the DO loop macros in do_loop_substitute.h90 to use the tile bounds

The tile domain is defined by a new set of domain indices (ntsi, ntei, ntsj, ntej), which represent the internal part of the domain:

- #define __kIs_     2
+ #define __kIs_     ntsi

A new subroutine dom_tile (in domain.F90) sets the values of these indices.

During initialisation, this subroutine calculates and stores the indices in global arrays (ntsi_a, ntei_a, ntsj_a, ntej_a) with lengths equal to the number of tiles (nijtile) plus one. The zero index is used to store the indices for the full domain:

ntsi_a(0) = 1 + nn_hls
ntsj_a(0) = 1 + nn_hls
ntei_a(0) = jpi - nn_hls
ntej_a(0) = jpj - nn_hls

dom_tile is called whenever the active tile needs to be set or if tiling needs to be suppressed:

CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=3 ) ! Work on tile 3
CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=0 ) ! Work on the full domain

2. Declaring SUBROUTINE-level arrays using the tile bounds

A new substitution macro in do_loop_substitute.h90:

#define A2D        __kIsm1_:__kIep1_,__kJsm1_:__kJep1_

is used such that:

- ALLOCATE(jpi,jpj)   DIMENSION(jpi,jpj)
+ ALLOCATE(A2D)       DIMENSION(A2D)

and therefore operations between local working arrays (which have the dimensions of the tile) and global/input arrays (which have the dimensions of either the tile or full domain) require no further changes, unless using : subscripts as described below.

3. Replacing : subscripts with a DO loop macro where appropriate

This is only necessary when step 2 would introduce conformance issues:

- REAL(wp), DIMENSION(jpi,jpj,jpk) :: a3d
- REAL(wp), DIMENSION(jpi,jpj)     :: z2d
- z2d(:,:) = a3d(:,:,1).
+ REAL(wp), DIMENSION(jpi,jpj,jpk) :: a3d
+ REAL(wp), DIMENSION(A2D)         :: z2d
+ DO_2D_11_11
+    z2d(ji,jj) = a3d(ji,jj,1)
+ END_2D

4. Looping over tiles at the timestepping level

A loop over tiles has been added to stp. The domain indices for the current tile (ntile /= 0) are set at the start of each iteration. After exiting the loop (and before, during initialisation) the tiling is suppressed (ntile == 0):

! Loop over tile domains
DO jtile = 1, nijtile
   IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=jtile )
   CALL tra_ldf( kstp, Nbb, Nnn, ts, Nrhs )  ! lateral mixing
END DO
IF( ln_tile ) CALL dom_tile( ntsi, ntsj, ntei, ntej, ktile=0 )        ! Revert to full domain

DO loops within the tiling loop therefore work on the current tile, while those outside the loop work on the full domain.

5. A new namelist (namtile)

   !-----------------------------------------------------------------------
   &namtile        !   parameters of the tiling
   !-----------------------------------------------------------------------
      ln_tile = .false.     !  Use tiling (T) or not (F)
      nn_ltile_i = 10       !  Length of tiles in i
      nn_ltile_j = 10       !  Length of tiles in j
   /

The number of tiles is calculated from the tile lengths, nn_ltile_i and nn_ltile_j, with respect to the full domain.

Branch

These branches contain a trial implementation of tiling in tra_ldf_iso; there is not yet a formal branch for the development.

Implementation in trunk

Implementation in extended haloes branch

New subroutines

  • OCE/DOM/domain/dom_tile- Calculate/set tiling variables (domain indices, number of tiles)

Modified modules

  • cfgs/SHARED/namelist_ref- Add namtile namelist
  • OCE/DOM/dom_oce- Declare tiling namelist and other tiling variables
  • OCE/DOM/domain- Read namtile namelist (dom_nam), calculate tiling variables and do control print (dom_tile)
  • OCE/IOM/prtctl- Add IF statement to prevent execution of prt_ctl by each tile
  • OCE/TRA/traldf- Add IF statements to prevent execution of trd_tra by each tile
  • OCE/TRA/traldf_iso- Add IF statements (as above), modify local arrays for tiling
  • OCE/do_loop_substitute- Modify DO loop macros to use domain indices, add A2D macro
  • OCE/par_oce- Declare tiling variables
  • OCE/step- Add tiling loop
  • OCE/step_oce- Add USE statement for dom_tile in step
  • OCE/timing- Add IF statements to prevent execution of timing_start and timing_stop by each tile

New variables (excluding local)

  • Global variables
    • ntsi, ntsj- start index of tile
    • ntei, ntej- end index of tile
    • ntsi_a, ntsj_a- start indices of each tile
    • ntei_a, ntej_a- end indices of each tile
    • ntile- current tile number
    • nijtile- number of tiles
  • Namelist
    • ln_tile- logical control on use of tiling
    • nn_ltile_i, nn_ltile_j- tile length
  • Pre-processor macros
    • A2D- substitution for ALLOCATE or DIMENSION arguments

Notes

Untiled code and other issues

See the attached document.

Extended haloes

The tiling trial has also been implemented in the extended haloes branch. There are few differences between this and the trunk implementation.

Documentation updates

Using previous parts, define the main changes to be done in the NEMO literature (manuals, guide, web pages, …).

Preview

Since the preview step must be completed before the PI starts the coding, the previewer(s) answers are expected to be completed within the two weeks after the PI has sent the request to the previewer(s).
Then an iterative process should take place between PI and previewer(s) in order to find a consensus

Possible bottlenecks:

  • the methodology
  • the flowchart and list of routines to be changed
  • the new list of variables wrt coding rules
  • the summary of updates in literature

Once an agreement has been reached, preview is ended and the PI can start the development into his branch.

Tests

Once the development is done, the PI should complete the tests section below and after ask the reviewers to start their review.

This part should contain the detailed results of SETTE tests (restartability and reproducibility for each of the reference configuration) and detailed results of restartability and reproducibility when the option is activated on specified configurations used for this test

Regular checks:

  • Can this change be shown to produce expected impact (option activated)?
  • Can this change be shown to have a null impact (option not activated)?
  • Results of the required bit comparability tests been run: are there no differences when activating the development?
  • If some differences appear, is reason for the change valid/understood?
  • If some differences appear, is the impact as expected on model configurations?
  • Is this change expected to preserve all diagnostics?
  • If no, is reason for the change valid/understood?
  • Are there significant changes in run time/memory?

Review

A successful review is needed to schedule the merge of this development into the future NEMO release during next Merge Party (usually in November).

Assessments:

  • Is the proposed methodology now implemented?
  • Are the code changes in agreement with the flowchart defined at preview step?
  • Are the code changes in agreement with list of routines and variables as proposed at preview step?
    If, not, are the discrepancies acceptable?
  • Is the in-line documentation accurate and sufficient?
  • Do the code changes comply with NEMO coding standards?
  • Is the development documented with sufficient details for others to understand the impact of the change?
  • Is the project literature (manual, guide, web, …) now updated or completed following the proposed summary in preview section?

Finding:

Is the review fully successful? If not, please indicate what is still missing


Once review is successful, the development must be scheduled for merge during next Merge Party Meeting.

Attachments (5)

Download all attachments as: .zip