#2226 closed Defect (fixed)

Incorrect SETTE test report

Reported by: smueller Owned by: systeam
Priority: high Milestone:
Component: CI Version:
Severity: major Keywords: SETTE, validation
Cc:

Description

Context

Following a source-code update that brought about the failure of the reference AMM12 model run, the corresponding AMM12 restartability and reproducibility tests were reported as being successful.

Analysis

The output files which SETTE compares in restartability and reproducibility tests (ocean.output, run.stat, tracer.stat) are created during the initial phase of a model run. If a model run fails before all of these output files have been created and a set of output files preexists in the test execution directory from a previous run, the post_test_tidyup function (source:/utils/CI/sette/all_functions.sh#L155) copies extant files from the previous test to the validation directory. As a result, source:/utils/CI/sette/sette_rpt.sh compares inappropriate output files and possibly reports an incorrect test outcome.

Recommendation

When the test execution directory is being prepared for the new test using script source:/utils/CI/sette/prepare_exe_dir.sh, any preexisting ocean.output, run.stat, or tracer.stat file could be removed or renamed.

Commit History (2)

ChangesetAuthorTimeChangeLog
10632smueller2019-02-04T17:40:20+01:00

Reversal of the modifications that have been inadvertently included in changeset [10631] when addressing ticket #2226

10631smueller2019-02-04T17:19:48+01:00

Slight SETTE enhancement to improve the reliability of restartability and reproducibility tests; see ticket #2226

Change History (4)

comment:1 Changed 14 months ago by smueller

  • Component changed from tools to CI

comment:2 Changed 14 months ago by smueller

In 10631:

Slight SETTE enhancement to improve the reliability of restartability and reproducibility tests; see ticket #2226

comment:3 Changed 14 months ago by smueller

In 10632:

Reversal of the modifications that have been inadvertently included in changeset [10631] when addressing ticket #2226

comment:4 Changed 14 months ago by smueller

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.