Issues with trunk when testing beyond SETTE (ORCA2_ICE_PISCES)

[acc]

Context

Two issues have arisen with the trunk whilst using the SETTE ORCA2_ICE_PISCES configuration for preliminary tests on a new cluster. These both arise when using domain decompositions beyond the standard 32-core, 8x4 or 4x8 used by SETTE.

1. Decompositions with land suppression give different run.stat results. This may only be an issue with this new cluster but all successful, fully-populated partitions up to 105 cores give consistent results. Decompositions with land suppression differ. First example is 8x4 with 30 cores; run.stat values differ from the 32 core equivalent after 651 timesteps.

2. Some larger decompositions fail to run and are stopped by checks on the validity of the specific humidity input in sbcblk.F90:

         ztmp = SUM(tmask(:,:,1)) ! number of ocean points on local proc domain
         IF( ztmp > 8._wp ) THEN ! test only on proc domains with at least 8 ocean points!
            ztmp = SUM(sf(jp_humi)%fnow(:,:,1)*tmask(:,:,1))/ztmp ! mean humidity over ocean on proc
            SELECT CASE( nhumi )
            CASE( np_humi_sph ) ! specific humidity => expect: 0. <= something < 0.065 [kg/kg] (0.061 is saturation at 45degC !!!)
               IF(  (ztmp < 0._wp) .OR. (ztmp > 0.065)  ) ztmp = -1._wp
            CASE( np_humi_dpt ) ! dew-point temperature => expect: 110. <= something < 320. [K]
               IF( (ztmp < 110._wp).OR.(ztmp > 320._wp) ) ztmp = -1._wp
            CASE( np_humi_rlh ) ! relative humidity => expect: 0. <= something < 100. [%]
               IF(  (ztmp < 0._wp) .OR.(ztmp > 100._wp) ) ztmp = -1._wp
            END SELECT
            IF(ztmp < 0._wp) THEN
               IF (lwp) WRITE(numout,'("   Mean humidity value found on proc #",i6.6," is: ",f10.5)') narea, ztmp
               CALL ctl_stop( 'STOP', 'Something is wrong with air humidity!!!', &
                  &   ' ==> check the unit in your input files'       , &
                  &   ' ==> check consistence of namelist choice: specific? relative? dew-point?', &
                  &   ' ==> ln_humi_sph -> [kg/kg] | ln_humi_rlh -> [%] | ln_humi_dpt -> [K] !!!' )
            END IF
         END IF

This is genuine but arises because the CORE2 input data has small regions of negative specific humidity (mainly arctic). With a 19x14, 223 core decomposition, for example, there is one 36 point domain that has a negative ztmp. With the check suppressed, this decomposition will complete the LONG integration in 74 seconds. A 223 decomposition for ORCA2 may look excessive but the new cluster has 128 core nodes and the charging granularity is per node.

Recommendation

The land suppression tests will be repeated on a different platform to confirm any issue.

The validation of specific humidity should be given an optional override.

...

[smasson]

I will do some land suppression tests too... I am afraid of what I will find...

[smasson]

I confirm that using X64_IRENE with intel 19, I have different results with jpni = 8, jpnj = 4 running on 30 cores.:

-bash-4.2$ cmp LONG/run.stat LONG30/run.stat
LONG/run.stat LONG30/run.stat differ: byte 95597, line 651

However, the problem disappears when I specifiy ln_icebergs = .False.
So, nothing really new in fact... ;-)

[acc]

Ah yes, I forgot we knew about the icebergs already in relation to this. On the other point, how do folks feel about making the minimum number of ocean points for the specific humidity test a namelist variable? The 8 that is hard-wired seems arbitrary. At least then the test can be suppressed by setting a large number.

[gsamson]

Concerning negative humidities in the Arctic, it should be better to set these unphysical values to zero instead.
It can be easily done with cdo for exemple:

cdo mul q_10.15JUNE2009_fill.nc -gec,0. q_10.15JUNE2009_fill.nc q_10.15JUNE2009_ok.nc

If we do so, inputs files also need to be updated on Zenodo.

[laurent]

Hello,
Hmmm, I'm behind the introduction of the possibility to use these different forms of humidity (for example, ECMWF provides dew-point temperature at 2m), and I also wrote this sanity check on humidity.
I agree with Guillaume, the bug is clearly in the CORE forcing and NEMO is right in this case to complain, negative humidity cannot be considered as okay even with and insignificantly small magnitude. Since I'm NCO-dude, input CORE forcing netcdf files could be cured with a "ncap2", using something like this:

ncap2 -O -s 'where(Q_10 < 0.) Q_10=0.' q_10.15JUNE2009_fill.nc -o q_10.15JUNE2009_fill_ok.nc

I'm about to test SETTE on my branch which is synchronized with the trunk, I may discover the other issue you mention, but cool to know about the "ln_iceberg" workaround!

Cheers, /laurent

Cheers, /laurent

[acc]

Won't fix: The first issue is known and related to icebergs. This will be investigated in a 2021WP entry. The issue with negative humidities should be addressed at source.