Opened 18 months ago

Closed 14 months ago

Last modified 14 months ago

#2239 closed Enhancement (fixed)

SETTE improvement

Reported by: mathiot Owned by: mathiot
Priority: low Milestone:
Component: CI Version: trunk
Severity: minor Keywords: SETTE,
Cc: Review:
MP ready?:
Progress:

Description

BE CAREFUL !!! Due to dynamic behaviour of this ticket creation page, it is highly recommend to set first all other fields before writing the ticket description below.
If you have lost your draft after an unwanted reload, you can click on the link 'Restore Form' in the contextual menu upper right to recover it.
Remove these lines after reading.

Context

  • The detection of the revision is not properly working (depend on use of 'git svn' or 'svn', french/italian/english output of svn info and svn version)
  • The comparison of the result with a reference version/revision is in a separate script almost a copy paste of sette_rpt
  • The personal/institute/computer parameter are not only contained in to param.cfg (need to change llsubmit, computer target and add key_nosignedzero if needed in SETTE
  • Running the full sette tests need 3 steps: sette, sette_test-case and sette_rpt and setting it up need to modify 3 files (param, sette and sette_test-cases.sh)

Proposal

  • Limit the number of file to edit to 1
  • Fix revision detection
  • Add result comparison with a reference (usually trunk in a development process)

Actions

  • force svn info to be done using en_US and simplify the extraction of path/rev number.
  • Add in oparam.cfg:
    • the batch command
    • the arch name
    • if we need key_nosignedzero
    • the reference VALIDATION_DIR and reference revision (if default no comparison of the results is done)
    • SVN command to use (default svn) in case you need to use 'git svn'
  • rename sette.sh to sette_cfg-ref.sh.
  • change in sette_*.sh accordingly to the change in param.cfg.
  • Add a sette.sh script which will run sette_cfg-ref.sh, then sette_test-cases.sh and then generate the report.
  • merge sette_chk_results and sette_rpt and remove sette_chk_results.
  • update of the README.
  • build a Branch for testing.

Commit History (17)

ChangesetAuthorTimeChangeLog
11165mathiot2019-06-21T10:51:26+02:00

delete branch fix_sette_ticket2239 (ticket #2239)

11161mathiot2019-06-20T18:55:23+02:00

merge fix_sette_ticket2239 : major simplification/improvement of SETTE (see README.rst for all the details), tested with the latest version of the trunk (r11160) (ticket #2239)

11158mathiot2019-06-20T17:34:31+02:00

replace hard coded svn command by ${SVN_CMD} (ticket #2239)

10820mathiot2019-04-01T15:17:13+02:00

update README (ticket #2239)

10819mathiot2019-04-01T15:10:17+02:00

update README and add comments in param.cfg (ticket #2239)

10750mathiot2019-03-13T19:06:16+01:00

update nn_itend and restart name to fit new time step of ORCA2, add usage to sette.sh, add OVERFLOW in the cfg list in param.cfg and update README (ticket #2239)

10747smueller2019-03-12T18:00:52+01:00

Addition of configurability of the set of active test configurations, see comment 14 of ticket #2239

10732mathiot2019-03-01T14:44:40+01:00

restore default parameter (compiler, batch cmd …) (ticket #2239)

10718mathiot2019-02-22T19:09:57+01:00

cosmetic changes (ticket #2239)

10717mathiot2019-02-22T16:52:47+01:00

change related to discussion with Simon (ticket #2239)

10713mathiot2019-02-21T21:29:46+01:00

changes related to Simon comments (ticket #2239)

10709mathiot2019-02-20T20:14:25+01:00

cosmetic change in the output (sette ticket #2239)

10708mathiot2019-02-20T20:09:56+01:00

add possibility to generate report on older revision + no more mix of various revision in the report output (ticket #2239)

10704mathiot2019-02-20T11:10:07+01:00

apply Simon suggestion + minor change in report display (ticket #2239)

10703mathiot2019-02-20T11:07:06+01:00

reverse change in BATCH_TEMPLATE (ticket #2239)

10699mathiot2019-02-19T16:29:38+01:00

revert last change on param.cfg (ticket #2239)

10698mathiot2019-02-19T16:23:31+01:00

check exit code from NEMO (do not copy anything if it failed on exit code other than 0/123456/999) + rename run.stat/tracer.stat/ocean.output to *_old in the current validation dir (ticket #2239)

Change History (22)

comment:1 Changed 18 months ago by mathiot

In 10687:

first commit on sette changes suggested in ticket2239

comment:2 Changed 18 months ago by mathiot

In 10698:

check exit code from NEMO (do not copy anything if it failed on exit code other than 0/123456/999) + rename run.stat/tracer.stat/ocean.output to *_old in the current validation dir (ticket #2239)

comment:3 Changed 18 months ago by mathiot

In 10699:

revert last change on param.cfg (ticket #2239)

comment:4 Changed 18 months ago by smueller

The exit status from the model run should be available at the start of function post_test_tidyup in variable $?. Therefore, in contrast to the implementation of changeset [10698], it should be possible to implement the exit-status check by modifying one file only (source:/utils/CI/sette/all_functions.sh). For example, with the modification

  • all_functions.sh

     
    154154# function to tidy up after each test and populate the NEMO_VALIDATION store 
    155155post_test_tidyup () { 
    156156# 
     157# Save current exit status of caller script 
     158RUN_STATUS=$? 
     159echo "Exit status: ${RUN_STATUS}" 
    157160# requires the following variables defined and exported from the calling script: 
    158161#  SETTE_DIR 
    159162#  INPUT_DIR 
     
    174177echo "TOOLS directory is : ${TOOLS_DIR}" 
    175178################################################################ 
    176179# SMALL DEBUG 
     180    EXIT_STATUS=${RUN_STATUS} 
    177181    if [ ! -r ${EXE_DIR}/ocean.output ] 
    178182        then 
    179183        grep "E R R O R" ${EXE_DIR}/ocean.output && echo "Some ERRORS at execution time, see ${EXE_DIR}/ocean.output" 
    180         exit 2 
     184        EXIT_STATUS=2 
    181185    fi 
    182186 
    183187    if [ ! -r ${EXE_DIR}/time.step ] 
    184188        then 
    185189        echo "file time.step does not exist"   >> ${SETTE_DIR}/output.sette 
    186190        echo "some problems during execution of model"  >> ${SETTE_DIR}/output.sette  
    187         exit 1 
     191        EXIT_STATUS=1 
    188192    else 
    189193        echo "file time.step exists"  >> ${SETTE_DIR}/output.sette 
    190194        echo "execution of model time step loop started"   >> ${SETTE_DIR}/output.sette 
     
    216220   echo "EXIT," 
    217221   exit 1 
    218222    fi 
     223 
     224    # Exit before populating validation directory if the model run has 
     225    # returned a non-zero exit status 
     226    [ ${EXIT_STATUS} -ne 0 ] && exit ${EXIT_STATUS} 
     227 
    219228    # Save output & debug files in NEMO_VALIDATION tree 
    220229    echo "saving ocean & ice output, run.stat, tracer.stat files ...." >> ${SETTE_DIR}/output.sette 
    221230    echo "            " >> ${SETTE_DIR}/output.sette 

function post_test_tidyup would create a directory in the validation tree in any case, but for model runs with non-zero exit status it would leave the directory empty (thus tests that involve model runs with non-zero exit status would be reported as incomplete); the modification would also bring the existing checks for run-time problems of the model run in line with the exit-status check (i.e., it would postpone the exit from the script until after the creation of an empty entry for the test run in the validation directory).

comment:5 Changed 18 months ago by mathiot

In 10703:

reverse change in BATCH_TEMPLATE (ticket #2239)

comment:6 Changed 18 months ago by mathiot

In 10704:

apply Simon suggestion + minor change in report display (ticket #2239)

comment:7 Changed 18 months ago by mathiot

In 10708:

add possibility to generate report on older revision + no more mix of various revision in the report output (ticket #2239)

comment:8 Changed 18 months ago by mathiot

In 10709:

cosmetic change in the output (sette ticket #2239)

comment:9 Changed 18 months ago by mathiot

In 10713:

changes related to Simon comments (ticket #2239)

comment:10 Changed 18 months ago by mathiot

After an extra round of discussion with Simon:

  • add a script to find all available revision for each config in the validation directory
  • possibility to run the report on the current directory (will set the status to FAILED if a directory is present in an older revision), an other specified revision or the old behavior (ie the lastest revision available in the validation directory is shown)
  • in param.cfg, there is the possibility to use the variable defined in your bashrc file most of the variable here (now the only remaining value are the compiler and the reference directory used to compare results from)

At the moment:

  • when starting a new sette old file are rename on valid dir and workdir
  • missing directory (exit status from NEMO other than 0)/missing ocean.output/ocean.output with error in it seems to be properly catch by the new sette

comment:11 Changed 18 months ago by mathiot

In 10717:

change related to discussion with Simon (ticket #2239)

comment:12 Changed 18 months ago by mathiot

In 10718:

cosmetic changes (ticket #2239)

comment:13 Changed 17 months ago by mathiot

In 10732:

restore default parameter (compiler, batch cmd …) (ticket #2239)

comment:14 Changed 17 months ago by smueller

In addition to the enhancements listed above, it would be useful to enhance SETTE by facilitating the restriction of the test configurations to a subset of those available. This could be achieved through modifications of source:/NEMO/branches/2019/fix_sette_ticket2239/sette_reference-configurations.sh and source:/NEMO/branches/2019/fix_sette_ticket2239/sette_test-cases.sh and by assigning the names of the active test configurations to a variable in source:/NEMO/branches/2019/fix_sette_ticket2239/param.cfg:

  • sette_reference-configurations.sh

     
    140140# Copy job_batch_COMPILER file for specific compiler into job_batch_template 
    141141cd ${SETTE_DIR} 
    142142cp BATCH_TEMPLATE/${JOB_PREFIX}-${COMPILER} job_batch_template || exit 
    143 # Description of configuration tested: 
    144 # GYRE_PISCES       :  1  
    145 # ORCA2_ICE_PISCES  :  2  
    146 # ORCA2_OFF_PISCES  :  3  
    147 # AMM12             :  4  
    148 # SAS               :  5 
    149 # ORCA2_ICE_OBS     :  6 
    150 # AGRIF             :  7 & 8  test AGRIF in a double zoom configuration in the nordic seas + 1 zoom in the eq. Pacific (AGRIF_DEMO) 
    151 #                               and check that key_agrif without zoom = no key_agrif 
    152 # SPITZ12           :  9      regional configuration including sea-ice and tides (Spitzbergen) 
     143# Description of available configurations: 
     144# GYRE_PISCES       : 
     145# ORCA2_ICE_PISCES  : 
     146# ORCA2_OFF_PISCES  : 
     147# AMM12             : 
     148# SAS               : 
     149# ORCA2_ICE_OBS     : 
     150# AGRIF             : test AGRIF in a double zoom configuration in the nordic seas + 1 zoom in the eq. Pacific (AGRIF_DEMO) 
     151#                       and check that key_agrif without zoom = no key_agrif 
     152# SPITZ12           : regional configuration including sea-ice and tides (Spitzbergen) 
    153153 
    154 for config in 1 2 3 4 5 6 7 8 9 
     154for config in ${TEST_CONFIGS} 
    155155do 
    156156 
    157157# ----------- 
    158158# GYRE_PISCES 
    159159# ----------- 
    160 if [ ${config} -eq 1 ] ;  then 
     160if [ ${config} == "GYRE_PISCES" ] ;  then 
    161161## Restartability tests for GYRE_PISCES 
    162162    export TEST_NAME="LONG" 
    163163    cd ${MAIN_DIR} 
     
    293293# ----------------- 
    294294# ORCA2_ICE_PISCES 
    295295# ----------------- 
    296 if [ ${config} -eq 2 ] ;  then 
     296if [ ${config} == "ORCA2_ICE_PISCES" ] ;  then 
    297297## Restartability tests for ORCA2_ICE_PISCES 
    298298    export TEST_NAME="LONG" 
    299299    cd ${MAIN_DIR} 
     
    530530# ---------------- 
    531531# ORCA2_OFF_PISCES 
    532532# ---------------- 
    533 if [ ${config} -eq 3 ] ;  then 
     533if [ ${config} == "ORCA2_OFF_PISCES" ] ;  then 
    534534## Restartability tests for ORCA2_OFF_PISCES 
    535535    export TEST_NAME="LONG" 
    536536    cd ${MAIN_DIR} 
     
    716716# ----- 
    717717# AMM12 
    718718# ----- 
    719 if [ ${config} -eq 4 ] ;  then 
     719if [ ${config} == "AMM12" ] ;  then 
    720720    ## Restartability tests for AMM12 
    721721    export TEST_NAME="LONG" 
    722722    cd ${MAIN_DIR} 
     
    839839# --------- 
    840840# ORCA2_SAS 
    841841# --------- 
    842 if [ ${config} -eq 5 ] ;  then 
     842if [ ${config} == "SAS" ] ;  then 
    843843## Restartability tests 
    844844    export TEST_NAME="LONG" 
    845845    cd ${MAIN_DIR} 
     
    969969# -------------- 
    970970## Test assimilation interface code, OBS and ASM for reproducibility 
    971971## Restartability not tested (ASM code not restartable while increments are being applied) 
    972 if [ ${config} -eq 6 ] ; then 
     972if [ ${config} == "ORCA2_ICE_OBS" ] ; then 
    973973## Reproducibility tests 
    974974    export TEST_NAME="REPRO_4_8" 
    975975    cd ${MAIN_DIR} 
     
    10851085# ------------ 
    10861086# AGRIF ICE 
    10871087# ----------- 
    1088 if [ ${config} -eq 7 ] ;  then 
     1088if [ ${config} == "AGRIF" ] ;  then 
    10891089## Restartability tests 
    10901090    export TEST_NAME="LONG" 
    10911091    cd ${MAIN_DIR} 
     
    13501350 
    13511351 
    13521352## test code corruption with AGRIF (phase 2) ==> Compile without key_agrif (to be compared with AGRIF_DEMO_ST/ORCA2) 
    1353 if [ ${config} -eq 8 ] ;  then 
     1353if [ ${config} == "AGRIF" ] ;  then 
    13541354    export TEST_NAME="ORCA2" 
    13551355    cd ${MAIN_DIR} 
    13561356    . ./makenemo -m ${CMP_NAM} -n AGRIF_DEMO_NOAGRIF_ST -r AGRIF_DEMO -j 8 add_key ${ADD_KEYS} del_key "key_agrif" 
     
    13851385# ------- 
    13861386# SPITZ12 
    13871387# ------- 
    1388 if [ ${config} -eq 9 ] ;  then 
     1388if [ ${config} == "SPITZ12" ] ;  then 
    13891389## Restartability tests 
    13901390    export TEST_NAME="LONG" 
    13911391    cd ${MAIN_DIR} 
  • sette_test-cases.sh

     
    142142cd ${SETTE_DIR} 
    143143cp BATCH_TEMPLATE/${JOB_PREFIX}-${COMPILER} job_batch_template || exit 
    144144# Description of configuration tested: 
    145 # OVERFLOW       : TEST s-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs  
    146 #                                        & (dynamics) advection schemes: flux form (ubs, centered), vector form (een) 
    147 #                          zps-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs 
    148 #                                        & (dynamics) advection schemes: flux form (ubs, centered), vector form (een, and een + Hollingsworth correction) 
    149 # LOCK_EXCHANGE  : 2 
    150 # VORTEX         : 3 
    151 # ICE_AGRIF      : 4 
    152 # ISOMIP         : 5 
     145# OVERFLOW       : TEST s-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs  
     146#                                     & (dynamics) advection schemes: flux form (ubs, centered), vector form (een) 
     147#                       zps-coordinates : (tracers) Advection schemes: FCT2, FCT4, ubs 
     148#                                     & (dynamics) advection schemes: flux form (ubs, centered), vector form (een, and een + Hollingsworth correction) 
     149# LOCK_EXCHANGE  :  
     150# VORTEX         :  
     151# ICE_AGRIF      :  
     152# ISOMIP         :  
    153153# WAD 
    154154 
    155155 
    156 for config in 1 2 3 4 5  
     156for config in ${TEST_CONFIGS} 
    157157do 
    158158 
    159159# --------- 
    160160#  OVERFLOW 
    161161# --------- 
    162 if [ ${config} -eq 1 ] ;  then 
     162if [ ${config} == "OVERFLOW" ] ;  then 
    163163    ## Restartability tests for OVERFLOW 
    164164    export TEST_NAME="LONG" 
    165165    cd ${MAIN_DIR} 
     
    253253# -------------- 
    254254#  LOCK_EXCHANGE 
    255255# -------------- 
    256 if [ ${config} -eq 2 ] ;  then 
     256if [ ${config} == "LOCK_EXCHANGE" ] ;  then 
    257257    ## Restartability tests for LOCK_EXCHANGE 
    258258    export TEST_NAME="LONG" 
    259259    cd ${MAIN_DIR} 
     
    348348# --------- 
    349349# VORTEX 
    350350# --------- 
    351 if [ ${config} -eq 3 ] ;  then  
     351if [ ${config} == "VORTEX" ] ;  then  
    352352## Restartability tests for VORTEX 
    353353    export TEST_NAME="LONG" 
    354354    cd ${MAIN_DIR} 
     
    521521# --------- 
    522522# ICE_AGRIF 
    523523# --------- 
    524 if [ ${config} -eq 4 ] ;  then  
     524if [ ${config} == "ICE_AGRIF" ] ;  then  
    525525## Restartability tests for ICE_AGRIF 
    526526    export TEST_NAME="LONG" 
    527527    cd ${MAIN_DIR} 
     
    702702# ------ 
    703703# ISOMIP 
    704704# ------ 
    705 if [ ${config} -eq 5 ] ;  then 
     705if [ ${config} == "ISOMIP" ] ;  then 
    706706## Restartability tests 
    707707    export TEST_NAME="LONG" 
    708708    cd ${MAIN_DIR} 
  • param.cfg

     
    1818BATCH_STAT=${SETTE_BATCH_STAT:-llq} 
    1919# generic sette job name 
    2020BATCH_NAME=${SETTE_BATCH_NAME:-sette} 
     21# List of test configurations 
     22TEST_CONFIGS=${SETTE_TEST_CONFIGS:-"GYRE_PISCES ORCA2_ICE_PISCES ORCA2_OFF_PISCES AMM12 SAS ORCA2_ICE_OBS AGRIF SPITZ12 LOCK_EXCHANGE VORTEX ICE_AGRIF ISOMIP"} 
    2123# 
    2224#- FORCING files storing 
    2325FORCING_DIR=${SETTE_FORCING_DIR:-$WORKDIR/FORCING} 

Further, such restriction of the active test configurations could be made accessible via the command line, for example with the additional modification

  • sette.sh

     
    22# initialise user dependent variable 
    33SETTE_DIR=$(cd $(dirname "$0"); pwd) 
    44MAIN_DIR=$(dirname $SETTE_DIR) 
     5while getopts t: option; do [ $option == "t" ] && export SETTE_TEST_CONFIGS=$OPTARG; done 
     6shift $((OPTIND - 1)) 
    57. ./param.cfg 
    68 
    79# run sette on reference configuration 

comment:15 Changed 17 months ago by smueller

In 10747:

Addition of configurability of the set of active test configurations, see comment 14 of ticket #2239

comment:16 Changed 17 months ago by mathiot

In 10750:

update nn_itend and restart name to fit new time step of ORCA2, add usage to sette.sh, add OVERFLOW in the cfg list in param.cfg and update README (ticket #2239)

comment:17 Changed 16 months ago by mathiot

In 10819:

update README and add comments in param.cfg (ticket #2239)

comment:18 Changed 16 months ago by mathiot

In 10820:

update README (ticket #2239)

comment:19 Changed 14 months ago by mathiot

In 11158:

replace hard coded svn command by ${SVN_CMD} (ticket #2239)

comment:20 Changed 14 months ago by mathiot

In 11161:

merge fix_sette_ticket2239 : major simplification/improvement of SETTE (see README.rst for all the details), tested with the latest version of the trunk (r11160) (ticket #2239)

comment:21 Changed 14 months ago by mathiot

  • Resolution set to fixed
  • Status changed from new to closed

Commited at r11161 and in phase with trunk (r11160) and NEMO_4.0 (r11162)

comment:22 Changed 14 months ago by mathiot

In 11165:

delete branch fix_sette_ticket2239 (ticket #2239)

Note: See TracTickets for help on using tickets.