read/write SI3 restart with XIOS

[andmirek]

Currently each processor reads and writes restart. This development will extend use of XIOS to read/write restart in SI3. This development will follow what has been done for NEMO.

With XIOS it should be easier to write multidimensional data into the restart file (more than 3 dimensions). To maintain compatibility with the old restart file two read interfaces will be necessary. One reading the data in new format and the second reading the data in the current format, where 3D slices are written for 4D data.

Workplan action

This development is part of IMMERSE project
Wikipage: wiki:2019WP/HPC-11_Mirek_Andrejczuk_IO_with_XIOS

[andmirek]

In 11815:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

In 11837:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

In 11839:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

In 11840:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

Unfortunately grids with more than 3 dimensions can not be defined using Fortran interface. As a result currently XIOS follows NEMO iom approach to write SI3 4D arrays - they are written as a set of 3D arrays.

How to implement 4D arrays in SI3 to be investigated next. It will require change in the way restart is handled by XIOS. Probably restart files should be defined in xios_context rather than in a separate restart contexts.

[andmirek]

In 11870:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

This (11870) commit is a first step for writing 4D data with XIOS.

Writing attribute in subroutine iom_delay_rst was disabled temporary, until it can be also done with XIOS.

[andmirek]

In 11893:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

In 11901:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

With the latest commit restart reading/writing with XIOS for NEMO and SI3 works (was tested) for ORCA2_ICE_PISCES_ST, ORCA2_ICE_OBS_ST, AGRIF_DEMO_ST.

In order for this functionality to work with AGRIF, for each domain nn_wxios and ln_xios_read must be set independently.

Restart written with XIOS for SI3 will have different format than the one written with default NEMO iom routines, and iom will not be able to read it (some variables are saved as 4D with XIOS). XIOS is capable of reading SI3 restart written with iom as long as restart is in a single file.

An additional xml files, keeping definition of 4D grid, are needed and were added as part of this development. This may change (those files removed) in the future when definition of 4D grid will be possible using Fortran interface.

[andmirek]

In 11923:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

In 11924:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[andmirek]

Clem advised to have the same format of SI3 restart file for files written by XIOS and NEMO iom. I'll add this functionality in XIOS, but will keep writing 4D arrays with XIOS as an option (activated by a logical parameter in the code). I don't see an easy way to write/read 4D arrays using default NEMO functionality.

Regarding moving all calls to iom_set_rstw_var_active to iom.F90, it will make making changes (adding/removing restart variables) more complex than now, because it would require to put the whole logic which is in different parts of the code dealing with restart files for different configurations into iom.F90.

[smasson]

Some years ago, when we created iom.F90, the idea was to be able to use different IO libraries as transparently as possible...

This was still the case in NEMO 3.6.
In that version, you could find 3 different IO libraries: nf90, ioipsl and dimg.
the choice between theses libraries was done in iom_def.F90 (lines 30 to 37)
https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/releases/release-3.6/NEMOGCM/NEMO/OPA_SRC/IOM/iom_def.F90

The iom routines where therefore organized in a 2 stages modules.

• The first stage is iom.F90 that contains all public subroutines, prepare all the IO work but do the IO work.

All the idea was iom.F90 is the common interface for all IO libraries.
https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/releases/release-3.4/NEMOGCM/NEMO/OPA_SRC/IOM/iom.F90

• The second stage is called by iom.F90 subroutines and consist in 3 different IO routines

https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/releases/release-3.6/NEMOGCM/NEMO/OPA_SRC/IOM/iom_nf90.F90
https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/releases/release-3.6/NEMOGCM/NEMO/OPA_SRC/IOM/iom_ioipsl.F90
https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/releases/release-3.6/NEMOGCM/NEMO/OPA_SRC/IOM/iom_rstdimg.F90

The fork toward the different libraries was done in iom.F90, with case statements.
See for example iom_rstput interface in this version of iom.F90 (from line 1046)

We think that the use of xios to read/write restarts should try follow this same logic.

-> reintroduce case in iom to be able to switch from native netcdf (nf90) to xios library
-> Ideally, the use of xios for restarts should not be visible somewhere else, no?
-> once the interface is done it is directly usable for all restarts

We know that the reality is more complex that this simple description and developing such interface wont be so easy…
We think a good stating point is to follow what is done for nf90 restarts that are performed in 2 steps:

1) 1 time step before restart: define all the restart file using a specific context (so xios_context_initialize and xios_close_context_definition cand be done anytime)
2) fill the file when is it restart time step.

Sebastien and Clement

[andmirek]

I’m afraid with XIOS it will be difficult (or even impossible) to implement the code as above.

XIOS works in a different way than approaches used currently (in the past). With XIOS there is a definition phase where within context all files/variables used in this context are defined and execution phase, where after changing to a context, data is handled by XIOS either to process and write or write only with xios_send_field. As a result all checks on input arguments done before calling iom_nf90… are not needed, simply because all definitions were already set during context definition. And this is the reason why xios_send_field is directly in iom_rp… (the same is in iom_get), outside all checks needed by iom_nf90.

Initially I had a separate context for each restart file, but as documented in the ticket I had to switch to a single context (where all restart files for writing are defined) because of the issue with writing 4D data for SI3. 4D grid can’t be defined using Fortran interface and as a result a separate context defined in XML file is needed. It would be difficult to do that when a separate context for each restart file is open. In long term I think everything should be in a single context, or as small number of contexts as possible. As reported by Sam Adams and confirmed by Rémi Lacroix there is a significant overhead when new context is created, probably associated with global communication. See ​https://forge.ipsl.jussieu.fr/mailman/private.cgi/xios-users/2017/000699.html.

Xios interface is implemented by a logical flag in iom, this is done because not all code can handle I/O with XIOS at this moment. Should definitions be present for all variables read/written, XIOS related logical flag can be hidden. Note, at this moment it’s not obvious if all I/O can be replicated with XIOS so there may be a need to keep this flag in the future.

I do have iom_set_rstw_var_active for restart in other places than iom.F90 and restart.F90 (as raised by Clement), but reading/writing restart is not only in restart.F90. I have calls to above routine from exactly the same places in the code where restart variables are read/written. This is to avoid duplcation of the code in iom routines. Should all restart read/write calls move to restart.F90 the same can be done with iom_set_rstw_var_active.

[andmirek]

To be continued in 2020.