Version 10 (modified by andmirek, 7 months ago) (diff)

Name and subject of the action

Last edition: 11/29/19 22:57:59 by epico

The PI is responsible to closely follow the progress of the action, and especially to contact NEMO project manager if the delay on preview (or review) are longer than the 2 weeks expected.

  1. Summary
  2. Preview
  3. Tests
  4. Review

Summary

Analysis of scalability improvement using MPI3 new communications (e.g. collective neighbours communications) instead of point to point communications.

ticket: #2011

Preview

Mirek Andrejczuk

Plan outlined in ticket #2011 is OK. Most likely changes to the code will be limited to LBC/lib_mpp.F90. 
I'm happy to test the changes in MO operational configurations. May be worth considering implementation 
in which changing the number of halo points is easy.

Tests

Once the development is done, the PI should complete the tests section below and after ask the reviewers to start their review.

This part should contain the detailed results of SETTE tests (restartability and reproducibility for each of the reference configuration) and detailed results of restartability and reproducibility when the option is activated on specified configurations used for this test

Regular checks:

  • Can this change be shown to produce expected impact (option activated)?
  • Can this change be shown to have a null impact (option not activated)?
  • Results of the required bit comparability tests been run: are there no differences when activating the development?
  • If some differences appear, is reason for the change valid/understood?
  • If some differences appear, is the impact as expected on model configurations?
  • Is this change expected to preserve all diagnostics?
  • If no, is reason for the change valid/understood?
  • Are there significant changes in run time/memory?

The change improves the communication time. 
Preliminary tests show an improvement within a range of 18%-32% 
on the GYRE_PISCES configuration (with nn_GYRE=200), 
depending on the allocated number of cores

Results of the required bit comparability tests: No differences between outputs

This change preserves all diagnostics

Review

Coding standard OK.

No need for changes in documentation.

The development is using mpi3 collective neighbours communications functionality for halo exchange. It has been implemented for the case without land suppression and in one routine- traadv_fct.F90 only. I think this is much needed development. Initial results are very encouraging. I see two major issues with the current implementation.

1) Is MPI3 standard common now and all centres/users of NEMO will be able to compile and run the model (looks it was introduced in 2012, but I'n not sure if it’s implemented in all recent MPI distributions),

2) Because the implementation is limited to only one routine it will be difficult to test impact on performance, moreover in the case of problems debugging of the code will be difficult because for one routine different approach to exchange messages will be used.

3) It would be good to have a logical flag controling use of this option (for now in traadv_fct.F90).

Those two issues should be raised with the whole system team for discussion and potentially all centres should test if the code compiles/runs without any issues, BEFORE the decision to merge this development into the trunk is made. This is a change to a major component of NEMO and not only SETTE tests and test cases, but also operational configurations used in the Centres should be tested if possible.

I personally would propose a different strategy, which requires more development before merging this change into the trunk, but avoids potential problem outlined in point 1 and addresses point 2.

After implementing exchange for land suppression, a key_mpi3 should be introduce. If defined collective neighbours communications will be used for hallo exchange if not, what we have now will be used. There would be no need to name the newly added routines differently, on preprocessing stage exchange method will be set. My understanding is that the argument list in lbc_lnk will be exactly the same for both (new/old) cases. We would have two different approaches for halo exchange depending on the key key_mpi3 (in the same way XIOS1 and XIOS2 interfaces existed together in the past). By doing that number of changes in source code will be limited to files in LBC only. And if we are not happy with the performance of MPI3 we can continue using old approach until mpi3 (or newer standard) performs sufficiently good. If we are happy with the performance, we can just remove old method and key_mpi3 (after warning users about the change in advance).