= updating NEMO and LMDZ sources with Schwarz routines =

This page describes how to proceed to update the sources of NEMO and LMDZ with the Schwarz subroutines from the svn cocoa repository.

== updating NEMO ==

Go to your `config/IPSLCM6/SOURCES` directory.

Get the NEMO Schwarz sources from the repository:
{{{
svn co svn+ssh://forge/ipsl/forge/projets/cocoa/svn/SCHWARZ_IPSLCM6VLRNOP/SOURCES/NEMO MY_NEMO
}}}

All the needed files to modify NEMO for the Schwarz algorithm are in this directory: `MY_NEMO`.

The rest of the work consist in checking the subroutines in `MY_NEMO` vs those in `SOURCES/NEMO`, `NEMOGCM/CONFIG/ORCA2_LIM3_PISCES/MY_SRC` or `NEMOGCM/NEMO/OPA_SRC`.

Review each `.f90` in `MY_NEMO`: 
1. If it is present in `SOURCES/NEMO` do a vimdiff between the two and make the needed changes to the file in `SOURCES/NEMO` to include Schwarz modifications.
2. If it is present in `NEMOGCM/CONFIG/ORCA2_LIM3_PISCES/MY_SRC` but not in `SOURCES/NEMO` copy the file to `SOURCES/NEMO` do a vimdiff with the `MY_NEMO` file and include Schwarz modifications.
3. If it is neither in `ORCA2_LIM3_PISCES/MY_SRC` nor in `SOURCES/NEMO` copy the file from `NEMOGCM/NEMO/OPA_SRC` review the changes in `MY_NEMO` with vimdiff and include them in the `SOURCES/NEMO` file.

When you have an updated version of every `.f90` files from `MY_NEMO` in `SOURCES/NEMO` you can recompile NEMO.

Be careful before compiling that you have an additional file that is not present in the original NEMO: `schwarz.F90`. This file only needs to be copied.

You will still have to modify the NEMO namelist: `namelist_ORCA2_cfg`.

In your submission directory (where you have `config.card` and `Job_MYEXPERIMENT` files), go to `PARAM` and modify `namelist_ORCA2_cfg`.

Add `mswr`, `ksout`, `kall` and `cplfrq` to the `&namrun` section:
{{{
   mswr        =  010   !  number of scwharz iterations
   ksout       =  010   !  Schwarz iteration for output
   kall        =  001   !  first iteration with all physics
   ncplfrq     =  86400 !  coupling frequency in seconds
}}}

You should try to run a simulation with `mswr=1` `ksout=1`, `kall=1` and `ncplfrq=<coupling frequency of your reference simulation>` as a test.

It everything is ok it will give you the same result as without Schwarz.

== updating LMDZ ==

Go to the directory `modeles/LMDZ/libf`.

Get the Schwarz LMDZ subroutines from the repository:
{{{
svn co svn+ssh://forge/ipsl/forge/projets/cocoa/svn/SCHWARZ_IPSLCM6VLRNOP/SOURCES/SWZ_LMDZ SWZ_LMDZ
}}}

As for NEMO you have to update the `LMDZ/libf` files with those from `SWZ_LMDZ`.

For each file in `SWZ_LMDZ` you get a directory which is the directory where the file is located in `LMDZ/libf`.
Go there make a copy of the file:
{{{
cp my_initial_file.f90 my_initial_file.f90.old
}}}

Use vimdiff to review the changes that need to be applied to the file `LMDZ/libf` from `SWZ_LMDZ`.

Once this is over copy `schwarz.F90` in the proper directory.

Compiling LMDZ should be ok.

As for NEMO you will need to update a configuration file: `gcm.def_96x95_AP`

Edit `COMP/lmdz.car` and put this line in `ParametersFiles`:
{{{
(${SUBMIT_DIR}/PARAM/gcm.def_${RESOL_ATM_XY}_${LMDZ_Physics}_SWZ, gcm.def), \
}}}
to get the proper gcm.def file.

In `PARAM` make a copy of `gcm.def_96x95_AP` to `gcm.def_96x95_AP_SWZ` and add the following lines at the end:
{{{
## Schwarz coupling parameters from config.card
## number of schwarz iterations
mswr= 001
## write output at schwarz iteration number ksout
ksout= 001
## first all physics iteration
kall= 001
## frequency of coupling in seconds
ncplfrq= 86400
}}}

As a test, run the model with `mswr=1`, `ksout=1`, `kall=1` and `ncplfrq=<frequency of coupling of initial code>`.

It should give you the same resultst with both Schwarz modifications in NEMO and LMDZ than with the initial code.

== Additional modifications for better integration of config files and Schwarz iterations ==

Now that everything is fine you can make these changes to `config.card`, `namelist_ORCA2_cfg` and `gcm.def_96x95_AP_SWZ`.

In `config.card` add these lines to the `[UserChoices]` section:
{{{
##D-- Schwarz iterations parameters -
## number of schwarz iterations
mswr= 010
### write output at schwarz iteration number ksout
ksout= 010
### first all physics iteration
kall= 001
### frequency of coupling in seconds (maybe it could be read from oasis.card)
ncplfrq= 3600
}}}

The goal is to make these parameters global for NEMO and LMDZ.

In `namelist_ORCA2_cfg` replace the values by `_AUTOBLOCKER_`:
{{{
   mswr        =  _AUTOBLOCKER_   !  number of scwharz iterations
   ksout       =  _AUTOBLOCKER_   !  Schwarz iteration for output
   kall        =  _AUTOBLOCKER_   !  first iteration with all physics
   ncplfrq     =  _AUTOBLOCKER_   !  coupling frequency in seconds
}}}

In `gcm.def_96x95_AP_SWZ` do the same:
{{{
## Schwarz coupling parameters from config.card
## number of schwarz iterations
mswr= _AUTOBLOCKER_
## write output at schwarz iteration number ksout
ksout= _AUTOBLOCKER_
## first all physics iteration
kall= _AUTOBLOCKER_
## frequency of coupling in seconds
ncplfrq= _AUTOBLOCKER_
}}}

Now we will modify the NEMO and LMDZ driver files to set the values in both files with `config.card` values.

In `DRIVER/opa9.driver` add these lines for Schwarz in the Update function:
{{{
    IGCM_comp_modifyNamelist blocker    namelist_cfg nn_leapy     ${OPA_NLEAPY}
    # Schwarz parameters from config.card
    IGCM_comp_modifyNamelist blocker    namelist_cfg mswr         ${config_UserChoices_mswr}
    IGCM_comp_modifyNamelist blocker    namelist_cfg ksout        ${config_UserChoices_ksout}
    IGCM_comp_modifyNamelist blocker    namelist_cfg kall         ${config_UserChoices_kall}
    IGCM_comp_modifyNamelist blocker    namelist_cfg ncplfrq      ${config_UserChoices_ncplfrq}
}}}

In `DRIVER/lmdz.driver` add these lines in the Update function:
{{{
    # guide.def : Activate nudging if ok_guide=y set in lmdz.card
    if [ ! X${lmdz_UserChoices_ok_guide} = X ] ; then
    IGCM_comp_modifyDefFile nonblocker guide.def ok_guide  ${lmdz_UserChoices_ok_guide}
    else
    IGCM_comp_modifyDefFile nonblocker guide.def ok_guide n
    fi

    # gcm.def : set schwarz parameters from config.card
    IGCM_comp_modifyNamelist blocker gcm.def mswr         ${config_UserChoices_mswr}
    IGCM_comp_modifyNamelist blocker gcm.def ksout        ${config_UserChoices_ksout}
    IGCM_comp_modifyNamelist blocker gcm.def kall         ${config_UserChoices_kall}
    IGCM_comp_modifyNamelist blocker gcm.def ncplfrq      ${config_UserChoices_ncplfrq}
}}}

Now your changes to `config.card` will apply to NEMO and LMDZ.