Opened 5 years ago
Closed 5 years ago
#2226 closed Defect (fixed)
Incorrect SETTE test report
| Reported by: | smueller | Owned by: | systeam |
|---|---|---|---|
| Priority: | high | Milestone: | |
| Component: | CI | Version: | |
| Severity: | major | Keywords: | SETTE, validation |
| Cc: |
Description
Context
Following a source-code update that brought about the failure of the reference AMM12 model run, the corresponding AMM12 restartability and reproducibility tests were reported as being successful.
Analysis
The output files which SETTE compares in restartability and reproducibility tests (ocean.output, run.stat, tracer.stat) are created during the initial phase of a model run. If a model run fails before all of these output files have been created and a set of output files preexists in the test execution directory from a previous run, the post_test_tidyup function (source:/utils/CI/sette/all_functions.sh#L155) copies extant files from the previous test to the validation directory. As a result, source:/utils/CI/sette/sette_rpt.sh compares inappropriate output files and possibly reports an incorrect test outcome.
Recommendation
When the test execution directory is being prepared for the new test using script source:/utils/CI/sette/prepare_exe_dir.sh, any preexisting ocean.output, run.stat, or tracer.stat file could be removed or renamed.
Commit History (2)
| Changeset | Author | Time | ChangeLog |
|---|---|---|---|
| 10632 | smueller | 2019-02-04T17:40:20+01:00 | Reversal of the modifications that have been inadvertently included in changeset [10631] when addressing ticket #2226 |
| 10631 | smueller | 2019-02-04T17:19:48+01:00 | Slight SETTE enhancement to improve the reliability of restartability and reproducibility tests; see ticket #2226 |
Change History (4)
comment:1 Changed 5 years ago by smueller
- Component changed from tools to CI
comment:2 Changed 5 years ago by smueller
comment:3 Changed 5 years ago by smueller
In 10632:
comment:4 Changed 5 years ago by smueller
- Resolution set to fixed
- Status changed from new to closed
In 10631: