New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
#2226 (Incorrect SETTE test report) – NEMO

Opened 6 years ago

Closed 6 years ago

#2226 closed Defect (fixed)

Incorrect SETTE test report

Reported by: smueller Owned by: systeam
Priority: high Milestone:
Component: CI Version:
Severity: major Keywords: SETTE, validation
Cc:

Description

Context

Following a source-code update that brought about the failure of the reference AMM12 model run, the corresponding AMM12 restartability and reproducibility tests were reported as being successful.

Analysis

The output files which SETTE compares in restartability and reproducibility tests (ocean.output, run.stat, tracer.stat) are created during the initial phase of a model run. If a model run fails before all of these output files have been created and a set of output files preexists in the test execution directory from a previous run, the post_test_tidyup function (source:/utils/CI/sette/all_functions.sh#L155) copies extant files from the previous test to the validation directory. As a result, source:/utils/CI/sette/sette_rpt.sh compares inappropriate output files and possibly reports an incorrect test outcome.

Recommendation

When the test execution directory is being prepared for the new test using script source:/utils/CI/sette/prepare_exe_dir.sh, any preexisting ocean.output, run.stat, or tracer.stat file could be removed or renamed.

Commit History (2)

ChangesetAuthorTimeChangeLog
10632smueller2019-02-04T17:40:20+01:00

Reversal of the modifications that have been inadvertently included in changeset [10631] when addressing ticket #2226

10631smueller2019-02-04T17:19:48+01:00

Slight SETTE enhancement to improve the reliability of restartability and reproducibility tests; see ticket #2226

Change History (4)

comment:1 Changed 6 years ago by smueller

  • Component changed from tools to CI

comment:2 Changed 6 years ago by smueller

In 10631:

Error: Failed to load processor CommitTicketReference
No macro or processor named 'CommitTicketReference' found

comment:3 Changed 6 years ago by smueller

In 10632:

Error: Failed to load processor CommitTicketReference
No macro or processor named 'CommitTicketReference' found

comment:4 Changed 6 years ago by smueller

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.