REPORT OF THE WORKING GROUP MEETING (March 10, 2017)

Attendees: C. Bricaud & J. Chanut (Mercator Océan); R. Benshila (LEGOS); F. Dupont (Environnement Canada); C. Rousset (LOCEAN); L. Debreu (INRIA), T. Graham & M. Bell (MetOffice).

Objectives:

• Review current status and ongoing work
• Ideas to defend at next extended developer committee (3-6 April 2017 in Barcelona)

1. Ongoing developments:

a) AGRIF library:

The library used in NEMO (including the converter) is about 2 years old although it has evolved since then on INRIA’s side. Some reproducibility issues in NEMO (e.g. mpp decomposition dependent solution) have been assigned to the way extrapolations near masked points are handled in the library. This has been partially fixed thanks to Laurent (see ticket #1855) and seems to be adequate in Clément R.’s test cases. However, Laurent reports from tests conducted in the MARS model, that further modifications to the Library are needed.

A major revision of the library is ongoing, in particular in the framework of the CROCO model (formerly ROMS-AGRIF). This will allow more flexibility in the grids definition and in particular to be able to have interaction between overlapping grids having the same refinement level (in such a case, grids communicate with their parent level only in NEMO). Time stepping will also be performed in parallel for each grid having the same refinement level. This will be available in May 2017.

The “Grandmother strategy” ( ie the capability to handle a coarsened parent grid) is already available in the current library => Clément B. to have a quick check on how this is defined as a basis for the future bgc coarsening with AGRIF.

New web page including library documentation at: ​http://agrif.imag.fr

b) Sea-ice:

Several users (Geomar, Lgge) reported bugs with LIM2 and AGRIF although it was apparently working smoothly in older versions (v3.4) . Rachid and Jérôme provided fixes but no feedback from aforementioned users so far. LIM2 is obsolete anyway and LIM3 implementation almost achieved so that the group does not investigate this further.

Clément R. has made LIM3 agrifable but this concerns the trunk only. Jerome suggests to duplicate changes in NEMO3_6_STABLE since it is likely to be the most used version for a while. Simple advection tests reveal that some difficulties remain. The group recommended to perform 1:1 test (without neither refinement nor ice dynamics) to evaluate the impact of the truncated advection stencil near refined domain edges.

Suggestion to possibly discard ice dynamical refinement. Ice model would in that case be time stepped on the parent grid only which could save some cpu or provide an alternative if LIM3 boundary problems would persist. Proper interpolation of fluxes from the parent grid below ice could do the job. Strategy already explored by Rachid.

c) Vertical coordinate:

Reminder: Agrif currently supports only a unique, time invariant and geopotential vertical grid. The use of Eulerian S coordinates does not need any change in the code itself but is not considered in the preprocessing tools. Combination of S and z coordinates (enveloping bathymetry) is not possible due to the online computation of the envelope. It has to be performed at the preprocessing stage to ensure masks matching i.e in the nesting tools (see next paragraph).

Pedro Colombo (Lgge) adapted the nesting tools in order to use s-z coordinates and is testing it in a regional configuration over the Denmark strait. Jérôme has already tested with success the s-z combination with AGRIF over the Gibraltar strait.

Tim is implementing coordinate change among grids. Tests have been performed in the double gyre configuration (hence with a flat bottom and 2 different vertical z grids). From these, vertical (conservative) remapping seems to be ok. The next step is to perform tests in a realistic setup (ORCA2-Aguhlas).

Jérôme has to finish to compatibility with lagrangian grids. One way coupling is ok. Update of vertical scale factors is still missing.

d) Other compatibility issues:

• Prognostic vertical mixing schemes variables (e.g. TKE and GLS) need to be AGRIFed. This has been done for TKE scheme by S. Masson, but still pending for GLS. J. Juanno (LEGOS) has duplicated TKE changes into GLS scheme which could be transferred into the code repository. Jérôme points out that the lagged time stepping of these schemes makes AGRIF proper time interpolation and update impossible. The group strongly advises that the time stepping scheme has to be coherent among variables to ensure a seamless integration with AGRIF. Jérôme recalls that restartability is not achieved anymore with changes made by S. Masson for TKE variables.
• The use of open boundaries with AGRIF is restricted to the parent grid only although there is no deep reason not to do so also in the nested grids. Changes made by J. Juanno to allow this could again be transferred into the code.
• Stochastic parametrization is currently not allowed with AGRIF. J. Chanut and G. Rugierro have restored this possibility wit AGRIF.
• North-Fold and cyclic East-West boundary conditions still not possible. Strong need to sort this out to use online coarsening with AGRIF.
• Nesting has been implemented with 2nd order schemes in mind which only require one line (row) of boundary data provision. This leads to unconsistancies with higher order schemes as recently reported with biharmonic diffusion. The number of ghost cell points must be increased to solve this.
• Iceberg modelling (Canadian action): Modifying iceberg model to allow for 2-way interaction of icebergs. Currently we can force icebergs in AGRIF nest, but no update/interpolation performed at boundaries.

e) Status of MPP capabilities and AGRIF:

Current status is pretty much restrictive and seems to be one of major problem to “diffuse” AGRIF in global models:

• All grids must have the same mpp decomposition, ie the same pair of (jpni, jpnj) and no land processors suppression (ie jpnij = jpni X jpnj). Removing this restriction is currently investigated in Canada. Coordination is needed here.
• Computation is done sequentially if more than one child domain is defined although AGRIF could handle parallel computation ( Agrif_Parallel_sisters=T in the library). Mpi initialization is currently not at the right place to allow this (Rachid).

f) Preprocessing tools:

S-z capability from P. Colombo to be added (Jérôme to have a look at Pedro’s code). The future is clearly to insert the nesting tools capabilities into the Configuration manager.

g) Reference configuration:

ORCA 2-LIM3 is still the only one. Tim implemented changes to use GYRE. Clément R. does simple advection tests with sea-ice that need to be advertised. Desperately need for a new one with all capabilities (sea-ice, passive tracers, s-z coordinates) and some kind of basic report of the results (from a significantly long simulation as it is done with ORCA1).

h) New configurations/users/publications:

• NEMO-AGRIF v3.4 seems to be still very popular. We feel this is mainly because it was made compatible with sea-ice or because it has been “over-advertised” as the still only AGRIF working version.
• J. Juanno is developing new configurations in the equatorial Atlantic with AGRIF and NEMO_3_6_STABLE.
• P.Colombo (Lgge) over Denmark Strait (NEMO).
• Geomar has transitioned to 3_6 and is very active with AGRIF over various regions.
• Fred collected new configurations using AGRIF in Canada (active or planned):
  • Arctic Northern Hemisphere Atlantic (ANHA) 1/4 degree with 1/12 degree nest in the subpolar gyre (ANHA4-SPG12)
  • ANHA4-Greenland12: 1/4 parent with 1/12 degree nest encompassing Greenland and surrounding ocean basins
  • ANHA4-SPG12-LS60: 1/4 parent, 1/12 degree nest in subpolar gyre, 1/60 degree nest in the Labrador Sea
  • ANHA4-Greenland12-Fjord60: 1/4 parent, 1/12 degree nest around Greenland, multiple (4 or 5) 1/60 degree nests around Greenland's major marine terminating glaciers/fjords.
• Publication list to be updated (current list inserted at the end of this document – Müller’s publication added on April 7 thanks to Fred).

2) Extended developer committee:

We discussed the evolution of NEMO-AGRIF and what should be put forward for the coming 3-5 years at next extended developer committee. The same exercise has been done in the COPPER proposal. It seems to be still valid and basically consists in:

• Developing a strategy for online biochemistry coarsening with AGRIF.
• Allowing and demonstrating the benefit of local vertical coordinate change in AGRIF zooms.
• Adapting to the upcoming time stepping scheme and coupling at barotropic level.

The group expressed the need to sustain our effort to improve the overall implementation robustness. It is envisioned that a greater flexibility in the grid definition should be part of plans as it already the case in CROCO. One of the outcome could be the design of a multi-grid global eddy resolving model at an affordable cost or systematic refinement in few identified “hot spots” (overflows). To reach that end, it is strongly advised to consider grid refinement as one of the drivers of the kernel structure.