Minutes Notes from NEMO HPC working group meeting: 5 Nov 2015
Attending: Mike Bell, Miguel Castrillo, Silvia Mocavero, Tim Graham, Julien le Sommer, Oriol Mula-Valls, Martin Schreiber, Kim Serradell
Appologies Mondher Chekki, Claire Levy, Marie-Alice Foujols.
- Agree who will own actions identified in Sebastien Masson’s task list (and Gurvan’s additional action(s) from last meeting)
Silvia (CMCC) is willing to take on task 1 (suppress global communications). She will do it in two steps: a) an analysis leading to a proposed approach b) implementation of approach agreed with Systems Team.
Miguel’s team (BSC) is willing to do the part of task 2 extending the mpp_lnk functionality to a 4D array. Its implementation throughout the code would need to be done by another team.
Tim (Met Office) is willing to do task 3 (suppress useless communication).
No one volunteered for tasks 4 and 5 – larger halos. These were agreed to be important.
Action: Mike and Julien to look for suitable people to take on tasks 4 and 5, the second half of task 2, and the task proposed by Gurvan in previous meeting. (item 5 below is relevant to this)
- Discuss which benchmark configurations to use for future tests (see notes from previous meeting)
Standard benchmark configurations would enable comparison of performance on different machines by the group members. The system team receives requests for such configurations by external users; benchmark configurations also need to be given to vendors. So there are good motivations for shared benchmark configurations. The costs of setting them up were thought to be modest and NEMO version 3.6 could be a good choice of version initially. However a large number of configurations could be of interest: ORCA025 without sea-ice, ORCA025 with LIM3; ORCA12; GYRE with resolution equivalent to ORCA12. We probably want to choose only 2 configurations to get started. Miguel will set up a GoogleDrive? page where each group can say which configurations they would prefer to use as benchmarks. A decision will be made at the next meeting.
Action Miguel to set up GoogleDrive? page. Sylvia, Miguel, Tim to fill in their views.
- Discussion of existing evidence about bottlenecks (Silvia’s roofline analysis) and present performance on a single node
Roofline analysis is a useful first step in analysing the code efficiency within a single node. Figures 6 and 7 of Silvia’s paper suggest that the efficiency of advection within NEMO on a single processor is poor and could be significantly improved. Evidence of performance of other subroutines, configurations and machines would be valuable.
It would be useful to exchange information on how roofline diagrams are generated – to help ensure we are all using comparable and reliable methods
Action : Tim and Sylvia to discuss what specification of this would be useful and then iterate with Martin, Oriol and Miguel.
- Discuss how to develop a longer term strategy
We are starting to implement a short-term plan to improve MPI communications between nodes. The benchmark configuration and single node bottle-neck analysis is a short-term plan to improve understanding of issues and the performance on a single node.
Julien described two actions within an AGRIF consolidiation plan which could have a major impact on performance. In particular AGRIF could enable the sea-ice or biogeochemistry components of the model to run on a coarser grid than the dynamics.
Psyclone has potential as a pre-processing tool to enable NEMO to work efficiently on a wide range of HPCs. The G-Ocean project explored its application to a 2D shallow-water version of NEMO. Psyclone would be applied to NEMO using metadata which identify the field-type of each of the arrays (e.g. ua as a field defined at u-velocity gridpoints). The main NEMO subroutines would be converted into “kernels” essentially by stripping out the i,j,k loops. Psyclone applies a series of user-directed transformations to the code (such as MPI domain decomposition, OpenMP directives, etc) for its implementation on a particular HPC. Psyclone can be implemented in stages, one subroutine at a time, and allows the vanilla code (present version) to be re-built. It was agreed that a presentation describing the Psyclone concepts and how it could be applied to NEMO would be useful. The presentation should include discussion of the steps to implementation, the effort involved and the longer term maintenance issues. Action: Mike to arrange a suitable presentation on Psyclone at the next meeting.
AGRIF and Psyclone are both pre-processors and likely to conflict with each other
Martin has started a document describing the targets for HPC performance, the present performance and options for improvement. Martin, Sylvia, Tim and Mike intend to tidy this up a bit next week.
Action: Martin to circulate his document before 15 November.
- Copernicus call within H2020 (Julien)
Julien described the outline proposal he has developed with Mike and Paolo Oddo to respond to the Horizon 2020 Space 2016-2017 (see item 5iii in
Julien has submitted his document to the NEMO steering committee. It proposes that about 2 FTE effort for 3 years is devoted to HPC work. The work will need to be done in the period 2017-2019 and have an impact on the CMEMS service before end of 2020. The deadline for responses to the call is 3 March 2016.
The document is not specific about what tasks will be done. We should discuss ideas for tasks at our next meeting
Action: Julien to circulate document to group members (done)
- Next meeting
We should discuss benchmark configurations, the strategy document, Psyclone and ideas for the Copernicus call.
The last week of November or the first week of December would be good timing. 30 Nov – 2 Dec are not suitable dates.
Action: Mike to set up a doodle call.
