updating NEMO and LMDZ sources with Schwarz routines

This page describes how to proceed to update the sources of NEMO and LMDZ with the Schwarz subroutines from the svn cocoa repository.

updating NEMO

Go to your config/IPSLCM6/SOURCES directory.

Get the NEMO Schwarz sources from the repository:

svn co svn+ssh://forge/ipsl/forge/projets/cocoa/svn/SCHWARZ_IPSLCM6VLRNOP/SOURCES/NEMO MY_NEMO

All the needed files to modify NEMO for the Schwarz algorithm are in this directory: MY_NEMO.

The rest of the work consist in checking the subroutines in MY_NEMO vs those in SOURCES/NEMO, NEMOGCM/CONFIG/ORCA2_LIM3_PISCES/MY_SRC or NEMOGCM/NEMO/OPA_SRC.

Review each .f90 in MY_NEMO:

1. If it is present in SOURCES/NEMO do a vimdiff between the two and make the needed changes to the file in SOURCES/NEMO to include Schwarz modifications.
2. If it is present in NEMOGCM/CONFIG/ORCA2_LIM3_PISCES/MY_SRC but not in SOURCES/NEMO copy the file to SOURCES/NEMO do a vimdiff with the MY_NEMO file and include Schwarz modifications.
3. If it is neither in ORCA2_LIM3_PISCES/MY_SRC nor in SOURCES/NEMO copy the file from NEMOGCM/NEMO/OPA_SRC review the changes in MY_NEMO with vimdiff and include them in the SOURCES/NEMO file.

When you have an updated version of every .f90 files from MY_NEMO in SOURCES/NEMO you can recompile NEMO.

Be careful before compiling that you have an additional file that is not present in the original NEMO: schwarz.F90. This file only needs to be copied.

You will still have to modify the NEMO namelist: namelist_ORCA2_cfg.

In your submission directory (where you have config.card and Job_MYEXPERIMENT files), go to PARAM and modify namelist_ORCA2_cfg.

Add mswr, ksout, kall and cplfrq to the &namrun section:

   mswr        =  010   !  number of scwharz iterations
   ksout       =  010   !  Schwarz iteration for output
   kall        =  001   !  first iteration with all physics
   ncplfrq     =  86400 !  coupling frequency in seconds

You should try to run a simulation with mswr=1 ksout=1, kall=1 and ncplfrq=<coupling frequency of your reference simulation> as a test.

It everything is ok it will give you the same result as without Schwarz.

updating LMDZ

Go to the directory modeles/LMDZ/libf.

Get the Schwarz LMDZ subroutines from the repository:

svn co svn+ssh://forge/ipsl/forge/projets/cocoa/svn/SCHWARZ_IPSLCM6VLRNOP/SOURCES/SWZ_LMDZ SWZ_LMDZ

As for NEMO you have to update the LMDZ/libf files with those from SWZ_LMDZ.

For each file in SWZ_LMDZ you get a directory which is the directory where the file is located in LMDZ/libf. Go there make a copy of the file:

cp my_initial_file.f90 my_initial_file.f90.old

Or: for File in SWZ_LMDZ/*/* ; do cp ${File#SWZ_LMDZ/} ${File#SWZ_LMDZ/}.old ; done

Use vimdiff to review the changes that need to be applied to the file LMDZ/libf from SWZ_LMDZ. Or: for File in SWZ_LMDZ/*/* ; do cp ${File} ${File#SWZ_LMDZ/} ; done Once this is over copy schwarz.F90 in the proper directory.

Compiling LMDZ should be ok.

As for NEMO you will need to update a configuration file: gcm.def_96x95_AP

Edit COMP/lmdz.car and put this line in ParametersFiles:

(${SUBMIT_DIR}/PARAM/gcm.def_${RESOL_ATM_XY}_${LMDZ_Physics}_SWZ, gcm.def), \

to get the proper gcm.def file.

In PARAM make a copy of gcm.def_96x95_AP to gcm.def_96x95_AP_SWZ and add the following lines at the end:

## Schwarz coupling parameters from config.card
## number of schwarz iterations
mswr= 001
## write output at schwarz iteration number ksout
ksout= 001
## first all physics iteration
kall= 001
## frequency of coupling in seconds
ncplfrq= 86400

As a test, run the model with mswr=1, ksout=1, kall=1 and ncplfrq=<frequency of coupling of initial code>.

It should give you the same resultst with both Schwarz modifications in NEMO and LMDZ than with the initial code.

Additional modifications for better integration of config files and Schwarz iterations

Now that everything is fine you can make these changes to config.card, namelist_ORCA2_cfg and gcm.def_96x95_AP_SWZ.

In config.card add these lines to the [UserChoices] section:

##D-- Schwarz iterations parameters -
## number of schwarz iterations
mswr= 010
### write output at schwarz iteration number ksout
ksout= 010
### first all physics iteration
kall= 001
### frequency of coupling in seconds (maybe it could be read from oasis.card)
ncplfrq= 3600

The goal is to make these parameters global for NEMO and LMDZ.

In namelist_ORCA2_cfg replace the values by _AUTOBLOCKER_:

   mswr        =  _AUTOBLOCKER_   !  number of scwharz iterations
   ksout       =  _AUTOBLOCKER_   !  Schwarz iteration for output
   kall        =  _AUTOBLOCKER_   !  first iteration with all physics
   ncplfrq     =  _AUTOBLOCKER_   !  coupling frequency in seconds

In gcm.def_96x95_AP_SWZ do the same:

## Schwarz coupling parameters from config.card
## number of schwarz iterations
mswr= _AUTOBLOCKER_
## write output at schwarz iteration number ksout
ksout= _AUTOBLOCKER_
## first all physics iteration
kall= _AUTOBLOCKER_
## frequency of coupling in seconds
ncplfrq= _AUTOBLOCKER_

Now we will modify the NEMO and LMDZ driver files to set the values in both files with config.card values.

In DRIVER/opa9.driver add these lines for Schwarz in the Update function:

    IGCM_comp_modifyNamelist blocker    namelist_cfg nn_leapy     ${OPA_NLEAPY}
    # Schwarz parameters from config.card
    IGCM_comp_modifyNamelist blocker    namelist_cfg mswr         ${config_UserChoices_mswr}
    IGCM_comp_modifyNamelist blocker    namelist_cfg ksout        ${config_UserChoices_ksout}
    IGCM_comp_modifyNamelist blocker    namelist_cfg kall         ${config_UserChoices_kall}
    IGCM_comp_modifyNamelist blocker    namelist_cfg ncplfrq      ${config_UserChoices_ncplfrq}

In DRIVER/lmdz.driver add these lines in the Update function:

    # guide.def : Activate nudging if ok_guide=y set in lmdz.card
    if [ ! X${lmdz_UserChoices_ok_guide} = X ] ; then
    IGCM_comp_modifyDefFile nonblocker guide.def ok_guide  ${lmdz_UserChoices_ok_guide}
    else
    IGCM_comp_modifyDefFile nonblocker guide.def ok_guide n
    fi

    # gcm.def : set schwarz parameters from config.card
    IGCM_comp_modifyNamelist blocker gcm.def mswr         ${config_UserChoices_mswr}
    IGCM_comp_modifyNamelist blocker gcm.def ksout        ${config_UserChoices_ksout}
    IGCM_comp_modifyNamelist blocker gcm.def kall         ${config_UserChoices_kall}
    IGCM_comp_modifyNamelist blocker gcm.def ncplfrq      ${config_UserChoices_ncplfrq}

Now your changes to config.card will apply to NEMO and LMDZ.