This wiki relates to the ticket on NEMO reproducibility fails with land domains exclusion (#1704)

Author: Tomas Lovato

Institution: CMCC

Proposed actions, after the confirmation from other ST members:

• revise the code and the log of mppini subroutines
• evaluate the modified tool for offline mpp decomposition

Additional details not included in the ticket

I tried to use configuration 4 (ORCA2_LIM_PISCES) to test the reproducibility for a run with a total of 88 PEs (80 ocean+8 land) and one with only the ocean domains (80 PEs). Here below a diff file of changes used in the test.

@@ -385,7 +386,7 @@ if [ ${config} -eq 4 ] ;  then
     . ./all_functions.sh
     . ./prepare_exe_dir.sh
     JOB_FILE=${EXE_DIR}/run_job.sh
-    NPROC=16
+    NPROC=88
     if [ -f ${JOB_FILE} ] ; then \rm ${JOB_FILE} ; fi
     cd ${EXE_DIR}
     set_namelist namelist_cfg nn_it000 1
@@ -393,9 +394,9 @@ if [ ${config} -eq 4 ] ;  then
     set_namelist namelist_cfg nn_fwb 0
     set_namelist namelist_cfg ln_ctl .false.
     set_namelist namelist_cfg ln_clobber .true.
-    set_namelist namelist_cfg jpni 4
-    set_namelist namelist_cfg jpnj 4
-    set_namelist namelist_cfg jpnij 16
+    set_namelist namelist_cfg jpni 8
+    set_namelist namelist_cfg jpnj 11
+    set_namelist namelist_cfg jpnij 88
     set_namelist namelist_cfg nn_solv 2
     set_namelist namelist_top_cfg ln_trcdta .false.
     set_namelist namelist_top_cfg ln_diatrc .false.
@@ -425,16 +426,16 @@ if [ ${config} -eq 4 ] ;  then
     export TEST_NAME="REPRO_2_8"
     . ./prepare_exe_dir.sh
     JOB_FILE=${EXE_DIR}/run_job.sh
-    NPROC=16
+    NPROC=80
     if [ -f ${JOB_FILE} ] ; then \rm ${JOB_FILE} ; fi
     cd ${EXE_DIR}
     set_namelist namelist_cfg nn_it000 1
     set_namelist namelist_cfg nn_itend 75
     set_namelist namelist_cfg ln_clobber .true.
     set_namelist namelist_cfg nn_fwb 0
-    set_namelist namelist_cfg jpni 2
-    set_namelist namelist_cfg jpnj 8
-    set_namelist namelist_cfg jpnij 16
+    set_namelist namelist_cfg jpni 8
+    set_namelist namelist_cfg jpnj 11
+    set_namelist namelist_cfg jpnij 80
     set_namelist namelist_cfg nn_solv 2
     set_namelist namelist_top_cfg ln_trcdta .false.
     set_namelist namelist_top_cfg ln_diatrc .false.

The test for reproducibility failed.

I made the following change in the mppini_2.h subroutine to modify the exclusion criteria, which now consider a domain as land when all points within the region (inner+overlap) are zero (I also modified the offline decomposition tool accordingly):

isurf = 0
- DO jj = 1+jprecj, ilj-jprecj
- DO ji = 1+jpreci, ili-jpreci
+ DO jj = 1, ilj
+ DO ji = 1, ili
IF( imask(ji+iimppt(ii,ij)-1, jj+ijmppt(ii,ij)-1) == 1)
isurf = isurf+1
END DO
END DO

After these modifications, I rerun SETTE using the configuration 4 test always using 88 PEs, but this time the decomposition was 81 ocean + 7 land. The reproducibility test passed.