New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
ticket/1367_NOC2_ZTS – NEMO
wiki:ticket/1367_NOC2_ZTS

Version 6 (modified by acc, 9 years ago) (diff)

--

2014 NOC2 sub- time stepping of vertical advection to ease numerical stability problems with high vertical resolution and options which reduce vertical metrics

Stage 1. Create branch and checkout a working copy:

svn copy svn+ssh://acc@forge.ipsl.jussieu.fr/ipsl/forge/projets/nemo/svn/trunk    svn+ssh://acc@forge.ipsl.jussieu.fr/ipsl/forge/projets/nemo/svn/branches/2014/dev_r4743_NOC2_ZTS \
-m "Create branch for NOCS development of a sub-timestepping scheme for vertical advection to ease numerical stability issues when vertical metrics become small. See ticket #1367"

svn co svn+ssh://acc@forge.ipsl.jussieu.fr/ipsl/forge/projets/nemo/svn/branches/2014/dev_r4743_NOC2_ZTS

Stage 2. Import changes from test directory and check in

svn ci -m"Branch 2014/dev_r4743_NOC2_ZTS, #1367. Code changes to introduce optional sub-timestepping for vertical advection. \
          Changes in dynadv.F90, dynzad.F90, traadv.F90, traadv_tvd.F90 and namelist_ref only."
Password: 
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is en_GB.utf-8
svnserve: warning: please check that your locale name is correct
Sending        NEMOGCM/CONFIG/SHARED/namelist_ref
Sending        NEMOGCM/NEMO/OPA_SRC/DYN/dynadv.F90
Sending        NEMOGCM/NEMO/OPA_SRC/DYN/dynzad.F90
Sending        NEMOGCM/NEMO/OPA_SRC/TRA/traadv.F90
Sending        NEMOGCM/NEMO/OPA_SRC/TRA/traadv_tvd.F90
Transmitting file data .....
Committed revision 4736.

Enabling unsupported (external) configurations

A new option to enable makenemo to compile externally supported configurations has been implemented on this branch. This is a beta test facility which is fully functional on the test system but untested elsewhere. The new option takes the form of a "-u" (for unsupported) command-line argument to makenemo which can be used in place of the -r option to set up a new configuration based on an externally supported set of code and ancillary files. For the user, operation is of the form:

./makenemo -n MY_ORCA1_LIM -u ORCA1_LIM -m X64_MOBILIS -j 8

This will create and compile a configuration called MY_ORCA1_LIM in the local CONFIG directory with details of the configuration imported from an external site. These details are determined by a single line of settings in the uspcfg.txt file (now found in the CONFIG directory). For each external configuration, this is just one line of text, such as:

ORCA1_LIM # ORCA2_LIM # OPA_SRC LIM_SRC_2 NST_SRC # ftp://ftp.noc.soton.ac.uk/msmftp/ORCA1/ORCA1_LIM_ctl.txt

which contains (delimited by #'s): conf_name , nearest reference conf, source directories and a URL to a list of external files. The other changes use this information to:

  1. Set up a configuration structure; initially based on "the nearest reference configuration" (in this case ORCA2_LIM)
  2. Retrieve the remote file and retrieve any files listed there, overwriting local equivalents if necessary. Internally this uses wget so, if you don't get the result you expect, check your system and path for a working version of wget with access to the internet.

The rest of makenemo should proceed as normal.

Currently the example remote list only has one entry:

ftp://ftp.noc.soton.ac.uk/msmftp/ORCA1/Z75/bathy_level75_050308_UKMO.nc EXP00/bathy_meter.nc

In this case the remote URL for the bathymetry file followed by the local name relative to the new config directory. Obviously everything else needed could be added here, including cpp keys and any source for the MY_SRC directory. Any lines beginning with "#" in the remote list file will be ignored so it can (and should) be annotated with full descriptions and references for the files listed.

This first version was submitted at revision 4837:

svn ci -m"Branch 2014/dev_r4743_NOC2_ZTS. Added makenemo changes and associated COMPILE scripts to provide \
         options for unsupported (external) configurations. This is not yet widely tested but should not interfer with existing \
         options. See ticket #1367." NEMOGCM/CONFIG/makenemo NEMOGCM/CONFIG/uspcfg.txt NEMOGCM/TOOLS/COMPILE/Fcopy_extdir.sh \
         NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh NEMOGCM/TOOLS/COMPILE/Fmake_extconfig.sh
 
Sending        NEMOGCM/CONFIG/makenemo
Adding         NEMOGCM/CONFIG/uspcfg.txt
Adding         NEMOGCM/TOOLS/COMPILE/Fcopy_extdir.sh
Adding         NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh
Adding         NEMOGCM/TOOLS/COMPILE/Fmake_extconfig.sh
Transmitting file data .....
Committed revision 4837.

with some minor updates added at revision 4838:

svn ci -m"Branch 2014/dev_r4743_NOC2_ZTS. Correct a few typos in the comments and add facility for basic comments in the \
       remote file list for unsupported (external) configurations. See ticket #1367" NEMOGCM/CONFIG/makenemo NEMOGCM/TOOLS/COMPILE/Fadd_keys.sh\
       NEMOGCM/TOOLS/COMPILE/Fcheck_config.sh NEMOGCM/TOOLS/COMPILE/Fclean_var.sh NEMOGCM/TOOLS/COMPILE/Fcopy_dir.sh \
       NEMOGCM/TOOLS/COMPILE/Fdel_keys.sh NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh NEMOGCM/TOOLS/COMPILE/Fmake_WORK.sh 
Password: 
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is en_GB.utf-8
svnserve: warning: please check that your locale name is correct
Sending        NEMOGCM/CONFIG/makenemo
Sending        NEMOGCM/TOOLS/COMPILE/Fadd_keys.sh
Sending        NEMOGCM/TOOLS/COMPILE/Fcheck_config.sh
Sending        NEMOGCM/TOOLS/COMPILE/Fclean_var.sh
Sending        NEMOGCM/TOOLS/COMPILE/Fcopy_dir.sh
Sending        NEMOGCM/TOOLS/COMPILE/Fdel_keys.sh
Sending        NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh
Sending        NEMOGCM/TOOLS/COMPILE/Fmake_WORK.sh
Transmitting file data ........
Committed revision 4838.

A full working example has now been set up with a permanent accompanying web-resource: http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/

Additional functionality was required to enable this test case to work correctly. This was the ability to direct the fetching process to create destination directories on the local system if required. This circumvents a limitation of wget which would otherwise return a flat file if the directory did not exist. The remote file list can now contain an instruction to create a directory before downloading content. Here is a snippet from the new, full, ORCA1-CICE example:

create_directory                                                                                                EXP00/RUNOFF
create_directory                                                                                                EXP00/FORCING
http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/cpp_ORCA1_CICE.fcm                        cpp_ORCA1_CICE.fcm
http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/EXP00-files/ahmcoef                       EXP00/ahmcoef
http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/EXP00-files/bathy_meter.nc                EXP00/bathy_meter.nc

http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/EXP00-files/FORCING/q2_y2000.nc           EXP00/FORCING/q2_y2000.nc
http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/EXP00-files/FORCING/rad_lw_down_y2000.nc  EXP00/FORCING/rad_lw_down_y2000.nc

http://gws-access.ceda.ac.uk/public/nemo/uspconfigs/ORCA1_CICE/v3.6.0/EXP00-files/RUNOFF/runoff_1m_y2000.nc     EXP00/RUNOFF/runoff_1m_y2000.nc
svn ci -m "Branch 2014/dev_r4743_NOC2_ZTS. Added option to create directories for receiving files associated with unsupported (external) configurations.\
              This circumvents a limitation in the use of wget to retrieve remote file hierarchies. See ticket #1367." \
                 NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh NEMOGCM/CONFIG/uspcfg.txt
Sending        NEMOGCM/CONFIG/uspcfg.txt
Sending        NEMOGCM/TOOLS/COMPILE/Ffetch_extdir.sh
Transmitting file data ..
Committed revision 4876.

Fixing Icebergs across the north-fold

A third set of orthogonal changes have been applied to this branch to fix problems encountered with the ICB package around the north fold. There were several aspects to the problems:

  1. The extra halos required for the iceberg interpolation were not exchanged correctly leading to 'missing rows' and artificial barriers to iceberg transport.
  2. An assumption about communication symmetry and message order that was found wanting at higher resolutions leading to dead-lock situations
  3. A lack of support for land-suppression.

The first two issues requires a recoding of the north-fold exchange of iceberg information which was extensive enough to warrant testing on a branch prior to submission. The current, tested solution is universal but has not yet been optimised for the ln_nnogather option. This will need to be addressed if ICB is to be used at very high processor counts. With a robust exchange mechanism in place, enabling land suppression proved to be a trivial task. Testing and identifying problems proved difficult initially due to the difficulty of working with the trajectory data spread throughout distributed datasets as ragged arrays. A basic python script has been added which collates trajectory data into contiguous streams for analysis.

svn ci -m 'Branch 2014/dev_r4743_NOC2_ZTS. Added fixes for ICB (icebergs) option to enable correct exchange of icb arrays across the north fold. \
These fixes also enable ICB to be used with land suppression. Optimisation of the exchanges for the ln_nnogather option has not yet been done. \
A Python utility (TOOLS/MISCELLANEOUS/icb_pp.py) is included to help collate iceberg trajectory output.' NEMOGCM/NEMO/OPA_SRC/ICB/icb_oce.F90 \
NEMOGCM/NEMO/OPA_SRC/ICB/icbdyn.F90 NEMOGCM/NEMO/OPA_SRC/ICB/icbini.F90 NEMOGCM/NEMO/OPA_SRC/ICB/icblbc.F90 NEMOGCM/NEMO/OPA_SRC/ICB/icbrst.F90 \
NEMOGCM/NEMO/OPA_SRC/ICB/icbstp.F90 NEMOGCM/NEMO/OPA_SRC/ICB/icbutl.F90 NEMOGCM/NEMO/OPA_SRC/LBC/lbclnk.F90 NEMOGCM/NEMO/OPA_SRC/LBC/lib_mpp.F90 \
NEMOGCM/TOOLS/MISCELLANEOUS/icb_pp.py

Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icb_oce.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icbdyn.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icbini.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icblbc.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icbrst.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icbstp.F90
Sending        NEMOGCM/NEMO/OPA_SRC/ICB/icbutl.F90
Sending        NEMOGCM/NEMO/OPA_SRC/LBC/lbclnk.F90
Sending        NEMOGCM/NEMO/OPA_SRC/LBC/lib_mpp.F90
Adding         NEMOGCM/TOOLS/MISCELLANEOUS/icb_pp.py
Transmitting file data ..........
Committed revision 4891.

illustration of the ice problems at the north fold before fix Icebergs trajectories near the north-fold before fix illustration of the fixed icebergs trajectories at the north fold Icebergs trajectories near the north-fold after fix More comprehensive test of the fixed icb code (ORCA1) More comprehensive test (ORCA1) of icebergs trajectories in the Arctic Basin

Preparing for 2014 merge

This branch has been prepared for the 2014 merge by the addition of trunk chanes between revision 4743 and 4879:

svn merge  svn+ssh://acc@forge.ipsl.jussieu.fr/ipsl/forge/projets/nemo/svn/trunk@4743 \
            svn+ssh://acc@forge.ipsl.jussieu.fr/ipsl/forge/projets/nemo/svn/trunk@4879 ./ 
svn ci -m 'Branch 2014/dev_r4743_NOC2_ZTS. Merged in trunk changes from r4743 to r4879 in \
                  preparation for the annual merge. See ticket #1367 and https://forge.ipsl.jussieu.fr/nemo/wiki/ticket/1367_NOC2_ZTS'

Sending    DOC/TexFiles/Chapters/Chap_DYN.tex
Sending    NEMOGCM/ARCH/CMCC/arch-PW6_calypso.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-PW6_calypso_debug.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-PW6_calypso_tools.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-gfortran_athena_xios.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-ifort_athena.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-ifort_athena_debug.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-ifort_athena_tools.fcm
Sending    NEMOGCM/ARCH/CMCC/arch-ifort_athena_xios.fcm
Sending    NEMOGCM/ARCH/INGV/arch-IBM_EKMAN_INGV.fcm
Sending    NEMOGCM/ARCH/arch-ALTIX_NAUTILUS_MPT.fcm
Sending    NEMOGCM/ARCH/arch-PW7_C2A_XIO.fcm
Sending    NEMOGCM/ARCH/arch-PW7_METO.fcm
Sending    NEMOGCM/ARCH/arch-X64_CURIE.fcm
Sending    NEMOGCM/ARCH/arch-X64_MOBILIS.fcm
Sending    NEMOGCM/ARCH/arch-X64_VAYU.fcm
Sending    NEMOGCM/ARCH/arch-X64_YELLOWSTONE.fcm
Sending    NEMOGCM/ARCH/arch-XC_ARCHER.fcm
Sending    NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm
Sending    NEMOGCM/ARCH/arch-macport_osx.fcm
Sending    NEMOGCM/ARCH/arch-openmpi_NAVITI_MERCATOR.fcm
Sending    NEMOGCM/ARCH/arch-x3750_ADA.fcm
Sending    NEMOGCM/CONFIG/AMM12/EXP00/namelist_cfg
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/README
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_IceFrac.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_all.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_vardef.sh
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_vardef.sh.old_names
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_ts_ICE.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_ts_ICE_FRAM.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_ts_ICE_Vol.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_ts_all.pro
Sending    NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_ts_vardef.sh
Sending    NEMOGCM/CONFIG/ORCA2_LIM3/EXP00/iodef.xml
Sending    NEMOGCM/CONFIG/SHARED/1_namelist_ref
Sending    NEMOGCM/CONFIG/SHARED/field_def.xml
Sending    NEMOGCM/CONFIG/SHARED/namelist_ice_lim3_ref
Sending    NEMOGCM/CONFIG/SHARED/namelist_ref
Sending    NEMOGCM/NEMO/LIM_SRC_3/ice.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/iceini.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limcons.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limdiahsb.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limdyn.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limistate.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limitd_me.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limitd_th.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limsbc.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd_dh.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd_dif.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd_ent.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd_lac.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limthd_sal.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limtrp.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limupdate1.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limupdate2.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limvar.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/limwri.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/par_ice.F90
Sending    NEMOGCM/NEMO/LIM_SRC_3/thd_ice.F90
Sending    NEMOGCM/NEMO/OFF_SRC/nemogcm.F90
Sending    NEMOGCM/NEMO/OPA_SRC/BDY/bdyice_lim.F90
Sending    NEMOGCM/NEMO/OPA_SRC/BDY/bdytides.F90
Sending    NEMOGCM/NEMO/OPA_SRC/DIA/diawri.F90
Sending    NEMOGCM/NEMO/OPA_SRC/DOM/domvvl.F90
Sending    NEMOGCM/NEMO/OPA_SRC/DYN/dynspg_ts.F90
Sending    NEMOGCM/NEMO/OPA_SRC/SBC/cpl_oasis3.F90
Sending    NEMOGCM/NEMO/OPA_SRC/SBC/fldread.F90
Sending    NEMOGCM/NEMO/OPA_SRC/SBC/sbcice_lim.F90
Sending    NEMOGCM/NEMO/OPA_SRC/SBC/sbcice_lim_2.F90
Sending    NEMOGCM/NEMO/OPA_SRC/SBC/sbcmod.F90
Sending    NEMOGCM/NEMO/OPA_SRC/TRA/traadv_mle.F90
Sending    NEMOGCM/NEMO/OPA_SRC/TRA/traqsr.F90
Sending    NEMOGCM/NEMO/OPA_SRC/ZDF/zdfgls.F90
Sending    NEMOGCM/NEMO/OPA_SRC/oce.F90
Sending    NEMOGCM/NEMO/OPA_SRC/step.F90
Sending    NEMOGCM/NEMO/TOP_SRC/PISCES/P4Z/p4zfechem.F90
Sending    NEMOGCM/NEMO/TOP_SRC/PISCES/P4Z/p4zmeso.F90
Sending    NEMOGCM/NEMO/TOP_SRC/PISCES/P4Z/p4zmicro.F90
Sending    NEMOGCM/NEMO/TOP_SRC/PISCES/P4Z/p4zsed.F90
Sending    NEMOGCM/SETTE/BATCH_TEMPLATE/batch-x3750_ADA
Sending    NEMOGCM/SETTE/iodef_sette.xml
Sending    NEMOGCM/SETTE/prepare_exe_dir.sh
Sending    NEMOGCM/SETTE/prepare_job.sh
Sending    NEMOGCM/SETTE/sette.sh
Sending    NEMOGCM/SETTE/sette_beginner.sh
Sending    NEMOGCM/TOOLS/COMPILE/bld.cfg
Sending    NEMOGCM/TOOLS/COMPILE/bld_preproagr.cfg
Sending    NEMOGCM/TOOLS/COMPILE/bld_tools.cfg
Sending    NEMOGCM/TOOLS/COMPILE/bldxag.cfg
Adding     NEMOGCM/ARCH/arch-x3750_ADA_O0.fcm
Adding     NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_MLD_ortho.pro
Adding     NEMOGCM/CONFIG/ORCA2_LIM/IDL_scripts/std_plot_vardef.sh.pstar4-testht
Adding     NEMOGCM/SETTE/BATCH_TEMPLATE/batch-XC_ARCHER_INTEL
Adding     NEMOGCM/SETTE/BATCH_TEMPLATE/batch-x3750_ADA_O0
Adding     NEMOGCM/SETTE/input_ORCA2_LIM3.cfg

Committed revision 4899.

And finally by confirmation that the branch passes SETTE tests:

 ./sette_rpt
GYRE        restartability  passed
ORCA2LIMPIS restartability  passed
ORCA2OFFPIS restartability  passed
AMM12       restartability  passed
SAS         restartability  inconclusive (  solver.stat  not found )

GYRE        reproducibility passed
ORCA2LIMPIS reproducibility passed
ORCA2OFFPIS reproducibility passed
AMM12       reproducibility passed
SAS         reproducibility inconclusive (  solver.stat  not found )

Attachments (3)

Download all attachments as: .zip