Opened 6 years ago
Closed 4 years ago
#2011 closed Task (fixed)
HPC-04(2018WP)_Mocavero_mpi3
| Reported by: | mocavero | Owned by: | mocavero |
|---|---|---|---|
| Priority: | high | Milestone: | 2019 WP |
| Component: | OCE | Version: | trunk |
| Severity: | minor | Keywords: | |
| Cc: |
Description (last modified by epico)
Context
MPI-3 provides new neighbourhood collective operations (i.e. MPI_Neighbor_allgather and MPI_Neighbor_alltoall) that allow to perform halo exchange with a single MPI communication call when a 5-point stencil is used.
Collective communications will be tested on the NEMO code in order to evaluate the code performance compared with the traditional point-to-point halo exchange currently implemented in NEMO.
The replacement of point-to-point communication with new collective ones will be designed and implemented taking care of the results accuracy.
Implementation plan
The work is described in the following:
Step 1: extraction of a mini-app to be used as test case. The advection kernel has been considered as test case and a mini-app has been implemented. The parallel application performs the MUSCL advection scheme and the dimension of the subdomain as well as the number of parallel processes can be set by the user
Step 2: integration of the new MPI-3 neighbourhood collective communications in the mini-app and performance comparison with the standard MPI-2 point-to-point communications. The evaluation of the proof of concept has been performed by changing the subdomain size. Performance analysis has been executed on system available at CMCC.
Step 3: the neighbourhood collective communications have been integrated in the NEMO code. The first version of the implementation uses a cartesian topology, so it does not support 9-point stencil neither land domain exclusion and the north fold is handled as usual. The use of new collective communications has been tested on a representative kernel implementing the FCT advection scheme.
Modified files are:
- OCE/LBC/lib_mpp.F90 where the new communicator is created, taking into account the different order of MPI processes between the cartesian communicator and NEMO
- OCE/LBC/mppini.F90 where the ranks of the processes are reordered and the call to the communicator creation routine has been added
- OCE/LBC/lbclnk.F90 where generic.h90 files to introduce MPI3 neighbourhood collectives are created
- OCE/TRA/traadv_fct.F90 as example of routine where MPI3 neighbourhood collectives can be used
Two files have been added:
- OCE/LBC/lbc_lnk_nc_generic.h90 to handle multi field exchange in MPI3 case
- OCE/LBC/mpp_nc_generic.h90 where the halo exchange is implemented.
The branch is ready to be merged during 2019 Merge Party. The proposed changes do not impact on NEMO usability. Reference manual will not be changed since code modifications are transparent to the users.
Step 4 (action will be continued in 2020): integration of graph topology to support the routines that use a 9-point stencil, the land domain exclusion and the north fold exchanges through MPI3 neighbourhood collective communications
Commit History (3)
| Changeset | Author | Time | ChangeLog |
|---|---|---|---|
| 11955 | mocavero | 2019-11-22T18:44:17+01:00 | Bug fix for MPI3 neighbourhood collectives halo exchange. See ticket #2011 |
| 11940 | mocavero | 2019-11-20T22:48:28+01:00 | Add MPI3 neighbourhood collectives halo exchange in LBC and call it in tracer advection FCT scheme #2011 |
| 11496 | mocavero | 2019-09-04T10:36:21+02:00 | Create HPC-12 branch - ticket #2011 |
Change History (12)
comment:1 Changed 6 years ago by mocavero
- Owner set to mocavero
- Status changed from new to assigned
comment:2 Changed 6 years ago by francesca
- Description modified (diff)
- Milestone changed from 2018 WP to 2019 WP
- Owner changed from mocavero to francesca
- wp_comment set to Some tests on a significative kernel have been executed (using a mini-app approach). The work will be continued in 2019 and the action is postponed.
comment:3 Changed 6 years ago by nicolasmartin
- Summary changed from HPC-04_Mocavero_mpi3 to HPC-04(2018WP)_Mocavero_mpi3
comment:4 Changed 5 years ago by nemo
- Priority changed from low to high
comment:5 Changed 5 years ago by francesca
- Description modified (diff)
comment:6 Changed 5 years ago by francesca
- Owner changed from francesca to mocavero
comment:7 Changed 5 years ago by mocavero
comment:8 Changed 5 years ago by francesca
- wp_comment changed from Some tests on a significative kernel have been executed (using a mini-app approach). The work will be continued in 2019 and the action is postponed. to Some tests on a significative kernel have been executed (using a mini-app approach). The integration of the neighbourhood collective communications in the trunk is ongoing. The branch will be ready for merge party end 2019
comment:9 Changed 4 years ago by mocavero
In 11940:
comment:10 Changed 4 years ago by mocavero
In 11955:
comment:11 Changed 4 years ago by epico
- Description modified (diff)
comment:12 Changed 4 years ago by francesca
- Resolution set to fixed
- Status changed from assigned to closed
- wp_comment changed from Some tests on a significative kernel have been executed (using a mini-app approach). The integration of the neighbourhood collective communications in the trunk is ongoing. The branch will be ready for merge party end 2019 to The neighbourhood collective communications have been integrated in the NEMO code (a dedicated branch has been developed). The use of new collective communications has been tested on a representative kernel implementing the FCT advection scheme. The action will be continued in 2020 to extend new collective communications to 9-point stencil routines, land domain exclusion and north fold exchanges.
In 11496: