NEMO HPC WG: Mon 22 Feb 2021

0. Attendees

Andy Porter (STFC), Clément Bricaud (Mercator), Claire Lev (IPSL), Daley Calvert (MetOffice?), David Gubiert (Atos), Eloi Rodriguez Gaxas (BSC), Erwan Raffin (Atos), Ioan Hadade, Italo Epicoco (CMCC), Kristian Mogensen (ECMWF), Mario Acosta (BSC), Martin Schreiber (TUM ), Maff Glover (MetOffice?),

Maicon Faria (BSC), Michael Lange (ECMWF), Miguel Castrillo (BSC), Mike Bell (MetOffice?), Mirek Andrejczuk (MetOffice?), Nils Wedi (ECMWF), Peter Dueben (ECMWF), Samuel Hatfield (ECMWF), Sebastien Masson (IPSL), Stella Paronuzzi (BSC), Vineet Soni (Mercator).

1. Presentations

During the meeting we have discussed about the adoption and the maintainability of the Mixed Precision implementation of NEMO. Two presentations have been discussed:

• Mixed precision approach (Stella Paronuzzi - BSC)
• Single-precision strategy at ECMWF (Samuel Hatfield - ECMWF)

(see attached slides)

2. Discussion

DRAFT

The discussion started recalling the undoubted benefits that the mixed precision approach has in terms of reduction of CPU time, reduction of data movement and reduction of I/O. The loss of accuracy can be fine tuned by means of the application of a tool which provides the user/developer with the list of variables to be kept in double precision.

The ECMWF has developed a single precision version of NEMO 4.0.4 keeping very few variables in double precision and has experienced good results with acceptable loss of accuracy for short term predictions with NEMO SP (single precision) even in a coupled configuration. In contrast, the mixed precision trunk from the BSC keeps more variables in double precision to reduce the loss of accuracy and support longer runs.

It was clearly stated that the double precision version of NEMO will not be affected by these changes, neither its results nor its computational performance.

According to the current approach, the application of the mixed precision in new developments requires the following procedure:

• The code needs to be compliant to a specific guidelines which include a few trivial rules
  • only as an example of trivial rules, but not exhaustive, we mention: each real variable must be declared in WP (working precision); each constant value must have the suffix '_wp'; etc.
• The user/developer has to define a configuration of interest and has to define the experiment (e.g. short-term vs long-term simulation; regional vs global domain; etc.)
• The user/developer has to execute an external tool (which we can name Mixed Precision Advice Tool which essentially consists of the execution of an ensemble run). The outcome of this tool is a list of variables that should be promoted to double precision from working precision.

The main concerns raised during the discussion were about the maintainability of the mixed precision version of NEMO considering that the NEMO ST would not be able to support every configuration in mixed precision.

From the user point of view, the only requirement to run the mixed precision version is compiling using the key_single option.

The outcomes of the meeting can be summarised as follows:

• HPC-WG agreed on and will propose to the NEMO ST to support the development of the NEMO code following the guidelines mentioned above, this will lead to a code where:
  • NEMO could be compiled in MP or in DP (as usual) by activating a compiler key
  • The Mixed Precision Advice Tool would be easily and directly applicable to the NEMO code

• HPC-WG agreed to include the Mixed Precision Advice Tool in the NEMO repository as external tool. The final user will apply this tool on his own configuration if needed.

• BSC (with the support of ECMWF) is available to take the responsibility of making the next NEMO 4.2 release fully compliant with the guidelines mentioned above. From the NEMO 4.2 on, each NEMO developer should adopt the coding guidelines in order to maintain NEMO fully compliant with the guidelines.

• BSC with the support of ECMWF is available to write the coding guidelines document to be followed by the developers after NEMO 4.2 is released

• BSC is available to apply the mixed precision and on the most common configurations

As a final comment, HPC-WG believes that the NEMO ST could maintain with a limited and small additional effort a version of NEMO that can be compiled in MP (for the most used configurations) as well as in DP, leaving to the final users the choice to apply the mixed precision only on other specific configurations that are not supported by the ST.