#2239 closed Enhancement (fixed)
SETTE improvement
Reported by: | mathiot | Owned by: | mathiot |
---|---|---|---|
Priority: | low | Milestone: | |
Component: | CI | Version: | trunk |
Severity: | minor | Keywords: | SETTE, |
Cc: |
Description
BE CAREFUL !!! Due to dynamic behaviour of this ticket creation page, it is highly recommend to set first all other fields before writing the ticket description below.
If you have lost your draft after an unwanted reload, you can click on the link 'Restore Form' in the contextual menu upper right to recover it.
Remove these lines after reading.
Context
- The detection of the revision is not properly working (depend on use of 'git svn' or 'svn', french/italian/english output of svn info and svn version)
- The comparison of the result with a reference version/revision is in a separate script almost a copy paste of sette_rpt
- The personal/institute/computer parameter are not only contained in to param.cfg (need to change llsubmit, computer target and add key_nosignedzero if needed in SETTE
- Running the full sette tests need 3 steps: sette, sette_test-case and sette_rpt and setting it up need to modify 3 files (param, sette and sette_test-cases.sh)
Proposal
- Limit the number of file to edit to 1
- Fix revision detection
- Add result comparison with a reference (usually trunk in a development process)
Actions
- force svn info to be done using en_US and simplify the extraction of path/rev number.
- Add in oparam.cfg:
- the batch command
- the arch name
- if we need key_nosignedzero
- the reference VALIDATION_DIR and reference revision (if default no comparison of the results is done)
- SVN command to use (default svn) in case you need to use 'git svn'
- rename sette.sh to sette_cfg-ref.sh.
- change in sette_*.sh accordingly to the change in param.cfg.
- Add a sette.sh script which will run sette_cfg-ref.sh, then sette_test-cases.sh and then generate the report.
- merge sette_chk_results and sette_rpt and remove sette_chk_results.
- update of the README.
- build a Branch for testing.
Commit History (17)
Changeset | Author | Time | ChangeLog |
---|---|---|---|
11165 | mathiot | 2019-06-21T10:51:26+02:00 | delete branch fix_sette_ticket2239 (ticket #2239) |
11161 | mathiot | 2019-06-20T18:55:23+02:00 | merge fix_sette_ticket2239 : major simplification/improvement of SETTE (see README.rst for all the details), tested with the latest version of the trunk (r11160) (ticket #2239) |
11158 | mathiot | 2019-06-20T17:34:31+02:00 | replace hard coded svn command by ${SVN_CMD} (ticket #2239) |
10820 | mathiot | 2019-04-01T15:17:13+02:00 | update README (ticket #2239) |
10819 | mathiot | 2019-04-01T15:10:17+02:00 | update README and add comments in param.cfg (ticket #2239) |
10750 | mathiot | 2019-03-13T19:06:16+01:00 | update nn_itend and restart name to fit new time step of ORCA2, add usage to sette.sh, add OVERFLOW in the cfg list in param.cfg and update README (ticket #2239) |
10747 | smueller | 2019-03-12T18:00:52+01:00 | Addition of configurability of the set of active test configurations, see comment 14 of ticket #2239 |
10732 | mathiot | 2019-03-01T14:44:40+01:00 | restore default parameter (compiler, batch cmd ...) (ticket #2239) |
10718 | mathiot | 2019-02-22T19:09:57+01:00 | cosmetic changes (ticket #2239) |
10717 | mathiot | 2019-02-22T16:52:47+01:00 | change related to discussion with Simon (ticket #2239) |
10713 | mathiot | 2019-02-21T21:29:46+01:00 | changes related to Simon comments (ticket #2239) |
10709 | mathiot | 2019-02-20T20:14:25+01:00 | cosmetic change in the output (sette ticket #2239) |
10708 | mathiot | 2019-02-20T20:09:56+01:00 | add possibility to generate report on older revision + no more mix of various revision in the report output (ticket #2239) |
10704 | mathiot | 2019-02-20T11:10:07+01:00 | apply Simon suggestion + minor change in report display (ticket #2239) |
10703 | mathiot | 2019-02-20T11:07:06+01:00 | reverse change in BATCH_TEMPLATE (ticket #2239) |
10699 | mathiot | 2019-02-19T16:29:38+01:00 | revert last change on param.cfg (ticket #2239) |
10698 | mathiot | 2019-02-19T16:23:31+01:00 | check exit code from NEMO (do not copy anything if it failed on exit code other than 0/123456/999) + rename run.stat/tracer.stat/ocean.output to *_old in the current validation dir (ticket #2239) |
Change History (22)
comment:1 Changed 6 years ago by mathiot
comment:2 Changed 6 years ago by mathiot
In 10698:
comment:3 Changed 6 years ago by mathiot
In 10699:
comment:4 Changed 6 years ago by smueller
The exit status from the model run should be available at the start of function post_test_tidyup in variable $?. Therefore, in contrast to the implementation of changeset [10698], it should be possible to implement the exit-status check by modifying one file only (source:/utils/CI/sette/all_functions.sh). For example, with the modification
-
all_functions.sh
154 154 # function to tidy up after each test and populate the NEMO_VALIDATION store 155 155 post_test_tidyup () { 156 156 # 157 # Save current exit status of caller script 158 RUN_STATUS=$? 159 echo "Exit status: ${RUN_STATUS}" 157 160 # requires the following variables defined and exported from the calling script: 158 161 # SETTE_DIR 159 162 # INPUT_DIR … … 174 177 echo "TOOLS directory is : ${TOOLS_DIR}" 175 178 ################################################################ 176 179 # SMALL DEBUG 180 EXIT_STATUS=${RUN_STATUS} 177 181 if [ ! -r ${EXE_DIR}/ocean.output ] 178 182 then 179 183 grep "E R R O R" ${EXE_DIR}/ocean.output && echo "Some ERRORS at execution time, see ${EXE_DIR}/ocean.output" 180 exit2184 EXIT_STATUS=2 181 185 fi 182 186 183 187 if [ ! -r ${EXE_DIR}/time.step ] 184 188 then 185 189 echo "file time.step does not exist" >> ${SETTE_DIR}/output.sette 186 190 echo "some problems during execution of model" >> ${SETTE_DIR}/output.sette 187 exit1191 EXIT_STATUS=1 188 192 else 189 193 echo "file time.step exists" >> ${SETTE_DIR}/output.sette 190 194 echo "execution of model time step loop started" >> ${SETTE_DIR}/output.sette … … 216 220 echo "EXIT," 217 221 exit 1 218 222 fi 223 224 # Exit before populating validation directory if the model run has 225 # returned a non-zero exit status 226 [ ${EXIT_STATUS} -ne 0 ] && exit ${EXIT_STATUS} 227 219 228 # Save output & debug files in NEMO_VALIDATION tree 220 229 echo "saving ocean & ice output, run.stat, tracer.stat files ...." >> ${SETTE_DIR}/output.sette 221 230 echo " " >> ${SETTE_DIR}/output.sette
function post_test_tidyup would create a directory in the validation tree in any case, but for model runs with non-zero exit status it would leave the directory empty (thus tests that involve model runs with non-zero exit status would be reported as incomplete); the modification would also bring the existing checks for run-time problems of the model run in line with the exit-status check (i.e., it would postpone the exit from the script until after the creation of an empty entry for the test run in the validation directory).
comment:5 Changed 6 years ago by mathiot
In 10703:
comment:6 Changed 6 years ago by mathiot
In 10704:
comment:7 Changed 6 years ago by mathiot
In 10708:
comment:8 Changed 6 years ago by mathiot
In 10709:
comment:9 Changed 6 years ago by mathiot
In 10713:
comment:10 Changed 6 years ago by mathiot
After an extra round of discussion with Simon:
- add a script to find all available revision for each config in the validation directory
- possibility to run the report on the current directory (will set the status to FAILED if a directory is present in an older revision), an other specified revision or the old behavior (ie the lastest revision available in the validation directory is shown)
- in param.cfg, there is the possibility to use the variable defined in your bashrc file most of the variable here (now the only remaining value are the compiler and the reference directory used to compare results from)
At the moment:
- when starting a new sette old file are rename on valid dir and workdir
- missing directory (exit status from NEMO other than 0)/missing ocean.output/ocean.output with error in it seems to be properly catch by the new sette
comment:11 Changed 6 years ago by mathiot
In 10717:
comment:12 Changed 6 years ago by mathiot
In 10718:
comment:13 Changed 6 years ago by mathiot
In 10732:
comment:14 Changed 6 years ago by smueller
In addition to the enhancements listed above, it would be useful to enhance SETTE by facilitating the restriction of the test configurations to a subset of those available. This could be achieved through modifications of source:/NEMO/branches/2019/fix_sette_ticket2239/sette_reference-configurations.sh and source:/NEMO/branches/2019/fix_sette_ticket2239/sette_test-cases.sh and by assigning the names of the active test configurations to a variable in source:/NEMO/branches/2019/fix_sette_ticket2239/param.cfg:
-
sette_reference-configurations.sh
140 140 # Copy job_batch_COMPILER file for specific compiler into job_batch_template 141 141 cd ${SETTE_DIR} 142 142 cp BATCH_TEMPLATE/${JOB_PREFIX}-${COMPILER} job_batch_template || exit 143 # Description of configuration tested:144 # GYRE_PISCES : 1145 # ORCA2_ICE_PISCES : 2146 # ORCA2_OFF_PISCES : 3147 # AMM12 : 4148 # SAS : 5149 # ORCA2_ICE_OBS : 6150 # AGRIF : 7 & 8test AGRIF in a double zoom configuration in the nordic seas + 1 zoom in the eq. Pacific (AGRIF_DEMO)151 # 152 # SPITZ12 : 9regional configuration including sea-ice and tides (Spitzbergen)143 # Description of available configurations: 144 # GYRE_PISCES : 145 # ORCA2_ICE_PISCES : 146 # ORCA2_OFF_PISCES : 147 # AMM12 : 148 # SAS : 149 # ORCA2_ICE_OBS : 150 # AGRIF : test AGRIF in a double zoom configuration in the nordic seas + 1 zoom in the eq. Pacific (AGRIF_DEMO) 151 # and check that key_agrif without zoom = no key_agrif 152 # SPITZ12 : regional configuration including sea-ice and tides (Spitzbergen) 153 153 154 for config in 1 2 3 4 5 6 7 8 9154 for config in ${TEST_CONFIGS} 155 155 do 156 156 157 157 # ----------- 158 158 # GYRE_PISCES 159 159 # ----------- 160 if [ ${config} -eq 1] ; then160 if [ ${config} == "GYRE_PISCES" ] ; then 161 161 ## Restartability tests for GYRE_PISCES 162 162 export TEST_NAME="LONG" 163 163 cd ${MAIN_DIR} … … 293 293 # ----------------- 294 294 # ORCA2_ICE_PISCES 295 295 # ----------------- 296 if [ ${config} -eq 2] ; then296 if [ ${config} == "ORCA2_ICE_PISCES" ] ; then 297 297 ## Restartability tests for ORCA2_ICE_PISCES 298 298 export TEST_NAME="LONG" 299 299 cd ${MAIN_DIR} … … 530 530 # ---------------- 531 531 # ORCA2_OFF_PISCES 532 532 # ---------------- 533 if [ ${config} -eq 3] ; then533 if [ ${config} == "ORCA2_OFF_PISCES" ] ; then 534 534 ## Restartability tests for ORCA2_OFF_PISCES 535 535 export TEST_NAME="LONG" 536 536 cd ${MAIN_DIR} … … 716 716 # ----- 717 717 # AMM12 718 718 # ----- 719 if [ ${config} -eq 4] ; then719 if [ ${config} == "AMM12" ] ; then 720 720 ## Restartability tests for AMM12 721 721 export TEST_NAME="LONG" 722 722 cd ${MAIN_DIR} … … 839 839 # --------- 840 840 # ORCA2_SAS 841 841 # --------- 842 if [ ${config} -eq 5] ; then842 if [ ${config} == "SAS" ] ; then 843 843 ## Restartability tests 844 844 export TEST_NAME="LONG" 845 845 cd ${MAIN_DIR} … … 969 969 # -------------- 970 970 ## Test assimilation interface code, OBS and ASM for reproducibility 971 971 ## Restartability not tested (ASM code not restartable while increments are being applied) 972 if [ ${config} -eq 6] ; then972 if [ ${config} == "ORCA2_ICE_OBS" ] ; then 973 973 ## Reproducibility tests 974 974 export TEST_NAME="REPRO_4_8" 975 975 cd ${MAIN_DIR} … … 1085 1085 # ------------ 1086 1086 # AGRIF ICE 1087 1087 # ----------- 1088 if [ ${config} -eq 7] ; then1088 if [ ${config} == "AGRIF" ] ; then 1089 1089 ## Restartability tests 1090 1090 export TEST_NAME="LONG" 1091 1091 cd ${MAIN_DIR} … … 1350 1350 1351 1351 1352 1352 ## test code corruption with AGRIF (phase 2) ==> Compile without key_agrif (to be compared with AGRIF_DEMO_ST/ORCA2) 1353 if [ ${config} -eq 8] ; then1353 if [ ${config} == "AGRIF" ] ; then 1354 1354 export TEST_NAME="ORCA2" 1355 1355 cd ${MAIN_DIR} 1356 1356 . ./makenemo -m ${CMP_NAM} -n AGRIF_DEMO_NOAGRIF_ST -r AGRIF_DEMO -j 8 add_key ${ADD_KEYS} del_key "key_agrif" … … 1385 1385 # ------- 1386 1386 # SPITZ12 1387 1387 # ------- 1388 if [ ${config} -eq 9] ; then1388 if [ ${config} == "SPITZ12" ] ; then 1389 1389 ## Restartability tests 1390 1390 export TEST_NAME="LONG" 1391 1391 cd ${MAIN_DIR}
-
sette_test-cases.sh
142 142 cd ${SETTE_DIR} 143 143 cp BATCH_TEMPLATE/${JOB_PREFIX}-${COMPILER} job_batch_template || exit 144 144 # Description of configuration tested: 145 # OVERFLOW : 1TEST s-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs146 # 147 # 148 # 149 # LOCK_EXCHANGE : 2150 # VORTEX : 3151 # ICE_AGRIF : 4152 # ISOMIP : 5145 # OVERFLOW : TEST s-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs 146 # & (dynamics) advection schemes: flux form (ubs, centered), vector form (een) 147 # zps-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs 148 # & (dynamics) advection schemes: flux form (ubs, centered), vector form (een, and een + Hollingsworth correction) 149 # LOCK_EXCHANGE : 150 # VORTEX : 151 # ICE_AGRIF : 152 # ISOMIP : 153 153 # WAD 154 154 155 155 156 for config in 1 2 3 4 5156 for config in ${TEST_CONFIGS} 157 157 do 158 158 159 159 # --------- 160 160 # OVERFLOW 161 161 # --------- 162 if [ ${config} -eq 1] ; then162 if [ ${config} == "OVERFLOW" ] ; then 163 163 ## Restartability tests for OVERFLOW 164 164 export TEST_NAME="LONG" 165 165 cd ${MAIN_DIR} … … 253 253 # -------------- 254 254 # LOCK_EXCHANGE 255 255 # -------------- 256 if [ ${config} -eq 2] ; then256 if [ ${config} == "LOCK_EXCHANGE" ] ; then 257 257 ## Restartability tests for LOCK_EXCHANGE 258 258 export TEST_NAME="LONG" 259 259 cd ${MAIN_DIR} … … 348 348 # --------- 349 349 # VORTEX 350 350 # --------- 351 if [ ${config} -eq 3] ; then351 if [ ${config} == "VORTEX" ] ; then 352 352 ## Restartability tests for VORTEX 353 353 export TEST_NAME="LONG" 354 354 cd ${MAIN_DIR} … … 521 521 # --------- 522 522 # ICE_AGRIF 523 523 # --------- 524 if [ ${config} -eq 4] ; then524 if [ ${config} == "ICE_AGRIF" ] ; then 525 525 ## Restartability tests for ICE_AGRIF 526 526 export TEST_NAME="LONG" 527 527 cd ${MAIN_DIR} … … 702 702 # ------ 703 703 # ISOMIP 704 704 # ------ 705 if [ ${config} -eq 5] ; then705 if [ ${config} == "ISOMIP" ] ; then 706 706 ## Restartability tests 707 707 export TEST_NAME="LONG" 708 708 cd ${MAIN_DIR}
-
param.cfg
18 18 BATCH_STAT=${SETTE_BATCH_STAT:-llq} 19 19 # generic sette job name 20 20 BATCH_NAME=${SETTE_BATCH_NAME:-sette} 21 # List of test configurations 22 TEST_CONFIGS=${SETTE_TEST_CONFIGS:-"GYRE_PISCES ORCA2_ICE_PISCES ORCA2_OFF_PISCES AMM12 SAS ORCA2_ICE_OBS AGRIF SPITZ12 LOCK_EXCHANGE VORTEX ICE_AGRIF ISOMIP"} 21 23 # 22 24 #- FORCING files storing 23 25 FORCING_DIR=${SETTE_FORCING_DIR:-$WORKDIR/FORCING}
Further, such restriction of the active test configurations could be made accessible via the command line, for example with the additional modification
-
sette.sh
2 2 # initialise user dependent variable 3 3 SETTE_DIR=$(cd $(dirname "$0"); pwd) 4 4 MAIN_DIR=$(dirname $SETTE_DIR) 5 while getopts t: option; do [ $option == "t" ] && export SETTE_TEST_CONFIGS=$OPTARG; done 6 shift $((OPTIND - 1)) 5 7 . ./param.cfg 6 8 7 9 # run sette on reference configuration
comment:15 Changed 6 years ago by smueller
In 10747:
comment:16 Changed 6 years ago by mathiot
In 10750:
comment:17 Changed 6 years ago by mathiot
In 10819:
comment:18 Changed 6 years ago by mathiot
In 10820:
comment:19 Changed 6 years ago by mathiot
In 11158:
comment:20 Changed 6 years ago by mathiot
In 11161:
comment:21 Changed 6 years ago by mathiot
- Resolution set to fixed
- Status changed from new to closed
comment:22 Changed 6 years ago by mathiot
In 11165:
In 10687: