Changes between Version 6 and Version 7 of 2019WP/ENHANCE-04_AndrewC-reporting

Timestamp:

2019-01-16T15:26:45+01:00 (5 years ago)

Author:

acc

Comment:

--

2019WP/ENHANCE-04_AndrewC-reporting

@@ -8 +8 @@
 
 == Summary
+
 Investigate ways of improving the code's reporting facilities. Currently, errors away from the lead process are not fully reported and the ln_ctl mechanism produces an overwhelming volume of output for large processor counts. Options are needed to produce more selective output (either by output type or processor range). This is an un-started task from the 2018WP (formerly ROBUST-06_AndrewC-reporting) that has been carried forward to 2019.
 ''#2167''
 
 == Preview 
-
+'''(version 2 following initial preview by Sebastien ( see his comments below))'''
 {{{#!box help
 [[Include(wiki:Developers/DevProcess#preview_)]]
@@ -25 +26 @@
 mpp.output_XXXX
 mpp.top.output_XXX     <---- correct this to 4-digits for consistency
-EMPave.dat_XXXX
+EMPave.dat_XXXX        <---- These are all identical, suppress writing to write-master only
 icebergs.stat_XXXX
 }}}
@@ -36 +37 @@
 3. OCE/stpctl.F90
 4. TOP/prtctl_trc.F90 TOP/trcini.F90 TOP/trcstp.F90
-5. OCE/LBC/mppini.F90
+5. OCE/LBC/mppini.F90  OCE/SBC/sbcfwb.F90
 6. sette/sette.sh
 }}}
+
+To reiterate, these changes allow run.stat and tracer.stat to be produced even when ln_ctl is .false.. They also introduce additional controls such as updating run.stat, tracer.stat and time.step at integer multiples of time step rather  than every time step and for restricting the production of some types of output (e.g. layout.dat) to a subset of processing regions. There are good arguments for separating out the multiple uses that ln_ctl has grown to support (see Sebastien's comments on its history) but I propose these changes , as a quick, pre-release addition.
+
 ''' 1. OCE/IOM/in_out_manager.F90 cfgs/SHARED/namelist_ref '''
 
@@ -48 +52 @@
 * l_config                        Activates use of the settings in the rest of the structure (specifically for when ln_ctl is false)
 * l_runstat, l_trcstat            Activates production of global stats files. Only a single file of each of these is ever produced.
-* l_oceout, l_layout, l_EMPave    Normal operation is to produce a single version of each of these. If true then a version for each area is produced
+* l_oceout, l_layout              Normal operation is to produce a single version of each of these. If true then a version for each area is produced
 * l_mppout, l_mpptrc, l_icbstat   Suppressed if false, otherwise produce a version for each area.
 * procmin, procmax, procincr      Allow subsetting of areas when producing output in the previous two categories. Default values will ensure all areas report.
+* ptimincr                        Timestep increment for outputting of time step status information less frequently (affects run.stat, tracer.stat and time.step only)
 }}}
 
@@ -56 +61 @@
 Index: OCE/IOM/in_out_manager.F90
 ===================================================================
---- OCE/IOM/in_out_manager.F90  (revision 10459)
+--- OCE/IOM/in_out_manager.F90  (revision 10530)
 +++ OCE/IOM/in_out_manager.F90  (working copy)
 @@ -99,6 +99,27 @@
@@ -71 +76 @@
 +      LOGICAL :: l_oceout  = .FALSE.  !: Produce all ocean.outputs    (T) or just one (F)
 +      LOGICAL :: l_layout  = .FALSE.  !: Produce all layout.dat files (T) or just one (F)
-+      LOGICAL :: l_EMPave  = .FALSE.  !: Produce all EMPave.dat files (T) or just one (F) (if active)
 +      LOGICAL :: l_mppout  = .FALSE.  !: Produce/do not produce mpp.output_XXXX files (T/F)
 +      LOGICAL :: l_mpptop  = .FALSE.  !: Produce/do not produce mpp.top.output_XXXX files (T/F)
@@ -81 +85 @@
 +      INTEGER :: procmax   = 1000000  !: Maximum narea to output
 +      INTEGER :: procincr  = 1        !: narea increment to output
++      INTEGER :: ptimincr  = 1        !: timestep increment to output (time.step and run.stat)
 +   END TYPE
 +   TYPE (sn_ctl) :: sn_cfctl     !: run control structure for selective output
@@ -89 +94 @@
 Index: cfgs/SHARED/namelist_ref
 ===================================================================
---- cfgs/SHARED/namelist_ref    (revision 10459)
+--- cfgs/SHARED/namelist_ref    (revision 10530)
 +++ cfgs/SHARED/namelist_ref    (working copy)
-@@ -1301,7 +1301,19 @@
+@@ -1303,7 +1303,19 @@
  !-----------------------------------------------------------------------
  &namctl        !   Control prints                                       (default: OFF)
@@ -102 +107 @@
 +       sn_cfctl%l_oceout  = .FALSE. ! that  all areas report.
 +       sn_cfctl%l_layout  = .FALSE. !
-+       sn_cfctl%l_EMPave  = .FALSE. !
 +       sn_cfctl%l_mppout  = .FALSE. !
 +       sn_cfctl%l_mpptop  = .FALSE. !
@@ -109 +113 @@
 +       sn_cfctl%procmax   = 1000000 ! Maximum area number for reporting [default:1000000]
 +       sn_cfctl%procincr  = 1       ! Increment for optional subsetting of areas [default:1]
++       sn_cfctl%ptimincr  = 1       ! Timestep increment for writing time step progress info
     nn_print    =    0      !  level of print (0 no extra print)
     nn_ictls    =    0      !  start i indice of control sum (use to compare mono versus
     nn_ictle    =    0      !  end   i indice of control sum        multi processor runs
+
 }}}
 
@@ -121 +127 @@
 Index: OCE/nemogcm.F90
 ===================================================================
---- OCE/nemogcm.F90     (revision 10459)
+--- OCE/nemogcm.F90     (revision 10530)
 +++ OCE/nemogcm.F90     (working copy)
 @@ -256,8 +256,8 @@
@@ -134 +140 @@
        NAMELIST/namcfg/ ln_read_cfg, cn_domcfg, ln_closea, ln_write_cfg, cn_domcfg_out, ln_use_jattr
        !!----------------------------------------------------------------------
-@@ -327,6 +327,17 @@
-
+@@ -327,6 +327,18 @@
+ 
        narea = narea + 1                                     ! mynode return the rank of proc (0 --> jpnij -1 )
-
+ 
 +      IF( sn_cfctl%l_config ) THEN
 +         ! Activate finer control of report outputs
@@ -146 +152 @@
 +           &   CALL nemo_set_cfctl( sn_cfctl, .FALSE., .FALSE. )
 +      ELSE
++         ! Use ln_ctl to turn on or off all options.
 +         CALL nemo_set_cfctl( sn_cfctl, ln_ctl, .TRUE. )
 +      ENDIF
@@ -151 +158 @@
        lwm = (narea == 1)                                    ! control of output namelists
        lwp = (narea == 1) .OR. ln_ctl                        ! control of all listing output print
-
-@@ -489,6 +500,18 @@
+ 
+@@ -503,6 +515,18 @@
           WRITE(numout,*) '~~~~~~~~'
           WRITE(numout,*) '   Namelist namctl'
@@ -161 +168 @@
 +         WRITE(numout,*) '                              sn_cfctl%l_oceout  = ', sn_cfctl%l_oceout
 +         WRITE(numout,*) '                              sn_cfctl%l_layout  = ', sn_cfctl%l_layout
-+         WRITE(numout,*) '                              sn_cfctl%l_EMPave  = ', sn_cfctl%l_EMPave
 +         WRITE(numout,*) '                              sn_cfctl%l_mppout  = ', sn_cfctl%l_mppout
 +         WRITE(numout,*) '                              sn_cfctl%l_mpptop  = ', sn_cfctl%l_mpptop
@@ -168 +174 @@
 +         WRITE(numout,*) '                              sn_cfctl%procmax   = ', sn_cfctl%procmax  
 +         WRITE(numout,*) '                              sn_cfctl%procincr  = ', sn_cfctl%procincr 
++         WRITE(numout,*) '                              sn_cfctl%ptimincr  = ', sn_cfctl%ptimincr 
           WRITE(numout,*) '      level of print                  nn_print   = ', nn_print
           WRITE(numout,*) '      Start i indice for SUM control  nn_ictls   = ', nn_ictls
           WRITE(numout,*) '      End i indice for SUM control    nn_ictle   = ', nn_ictle
-@@ -635,6 +658,35 @@
+@@ -649,6 +673,34 @@
        !
     END SUBROUTINE nemo_alloc
-
+ 
 +   SUBROUTINE nemo_set_cfctl(sn_cfctl, setto, for_all )
 +      !!----------------------------------------------------------------------
@@ -198 +205 @@
 +      sn_cfctl%l_oceout  = setto
 +      sn_cfctl%l_layout  = setto
-+      sn_cfctl%l_EMPave  = setto
 +      sn_cfctl%l_mppout  = setto
 +      sn_cfctl%l_mpptop  = setto
@@ -215 +221 @@
 
 {{{#!diff
- 
 Index: OCE/stpctl.F90
 ===================================================================
---- OCE/stpctl.F90      (revision 10459)
+--- OCE/stpctl.F90      (revision 10530)
 +++ OCE/stpctl.F90      (working copy)
-@@ -33,7 +33,7 @@
-    PUBLIC stp_ctl           ! routine called by step.F90
-
-    INTEGER  ::   idrun, idtime, idssh, idu, ids1, ids2, idt1, idt2, idc1, idw1, istatus
--   LOGICAL  ::   lsomeoce
-+   LOGICAL  ::   lsomeoce, lcolruns, lwrtruns
-    !!----------------------------------------------------------------------
-    !! NEMO/OCE 4.0 , NEMO Consortium (2018)
-    !! $Id$
-@@ -69,6 +69,8 @@
+@@ -66,17 +66,22 @@
+       INTEGER, DIMENSION(3)  ::   iu, is1, is2        ! min/max loc indices
+       REAL(wp)               ::   zzz                 ! local real 
+       REAL(wp), DIMENSION(9) ::   zmax
++      LOGICAL                ::   ll_wrtstp, ll_colruns, ll_wrtruns
        CHARACTER(len=20) :: clname
        !!----------------------------------------------------------------------
        !
-+      IF( kt == nit000 )   lcolruns = ln_ctl .OR. ( sn_cfctl%l_config .AND. sn_cfctl%l_runstat )
-+      IF( kt == nit000 )   lwrtruns = lcolruns .AND. lwm
++      ll_wrtstp  = ( MOD( kt, sn_cfctl%ptimincr ) == 0 ) .OR. ( kt == nitend )
++      ll_colruns = ll_wrtstp .AND. ( ln_ctl .OR. sn_cfctl%l_runstat )
++      ll_wrtruns = ll_colruns .AND. lwm
        IF( kt == nit000 .AND. lwp ) THEN
-          WRITE(numout,*) 
+          WRITE(numout,*)
           WRITE(numout,*) 'stp_ctl : time-stepping control'
-@@ -76,7 +78,7 @@
+          WRITE(numout,*) '~~~~~~~'
           !                                ! open time.step file
           IF( lwm ) CALL ctl_opn( numstp, 'time.step', 'REPLACE', 'FORMATTED', 'SEQUENTIAL', -1, numout, lwp, narea )
-          !                                ! open run.stat file
+-         !                                ! open run.stat file
 -         IF( ln_ctl .AND. lwm ) THEN
-+         IF( lwrtruns ) THEN
++         !                                ! open run.stat file(s) at start whatever
++         !                                ! the value of sn_cfctl%ptimincr
++         IF( lwm .AND. ( ln_ctl .OR. sn_cfctl%l_runstat ) ) THEN
              CALL ctl_opn( numrun, 'run.stat', 'REPLACE', 'FORMATTED', 'SEQUENTIAL', -1, numout, lwp, narea )
-             clname = 'run.stat.nc' 
+             clname = 'run.stat.nc'
              IF( .NOT. Agrif_Root() )   clname = TRIM(Agrif_CFixed())//"_"//TRIM(clname)
-@@ -120,12 +122,12 @@
+@@ -98,7 +103,7 @@
+       ENDIF
+       IF( kt == nit000 )   lsomeoce = COUNT( ssmask(:,:) == 1._wp ) > 0
+       !
+-      IF(lwm) THEN                        !==  current time step  ==!   ("time.step" file)
++      IF(lwm .AND. ll_wrtstp) THEN        !==  current time step  ==!   ("time.step" file)
+          WRITE ( numstp, '(1x, i8)' )   kt
+          REWIND( numstp )
+       ENDIF
+@@ -120,12 +125,12 @@
           zmax(9) = MAXVAL(   Cu_adv(:,:,:)   , mask = tmask(:,:,:) == 1._wp ) !       cell Courant no. max
        ENDIF
        !
 -      IF( ln_ctl ) THEN
-+      IF( lcolruns ) THEN
++      IF( ll_colruns ) THEN
           CALL mpp_max( "stpctl", zmax )          ! max over the global domain
           nstop = NINT( zmax(7) )                 ! nstop indicator sheared among all local domains
@@ -258 +270 @@
        !                                   !==  run statistics  ==!   ("run.stat" files)
 -      IF( ln_ctl .AND. lwm ) THEN
-+      IF( lwrtruns ) THEN
++      IF( ll_wrtruns ) THEN
           WRITE(numrun,9500) kt, zmax(1), zmax(2), -zmax(3), zmax(4)
           istatus = NF90_PUT_VAR( idrun, idssh, (/ zmax(1)/), (/kt/), (/1/) )
@@ -266 +278 @@
 ''' 4. TOP/prtctl_trc.F90 TOP/trcini.F90 TOP/trcstp.F90 '''
 
-Changes to control the production of tracer.stat follow similar lines with the introduction of a lltrcstat local logical. Note also changes to prtctl_trc.F90 to make mpp.top.output filenames compatible  with other similar filenames (i.e. use I4.4 for area number).
+Changes to control the production of tracer.stat follow similar lines with the introduction of a lltrcstat local logical. Note also changes to prtctl_trc.F90 to make mpp.top.output filenames compatible  with other similar filenames (i.e. use I4.4 for area number). This one is a little off message because the control for this TOP output has been read in OCE/nemogcm.F90. I think this makes sense and moving it to the TOP name list just for the sake of keeping OCE and TOP fully independent seems unnecessary; TBD.
 {{{#!diff
 Index: TOP/prtctl_trc.F90
@@ -309 +321 @@
 Index: TOP/trcstp.F90
 ===================================================================
---- TOP/trcstp.F90      (revision 10459)
+--- TOP/trcstp.F90      (revision 10530)
 +++ TOP/trcstp.F90      (working copy)
-@@ -31,6 +31,7 @@
-    PUBLIC   trc_stp    ! called by step
-
-    LOGICAL  ::   llnew                   ! ???
-+   LOGICAL  ::   lltrcstat               ! ???
-    REAL(wp) ::   rdt_sampl               ! ???
-    INTEGER  ::   nb_rec_per_day, ktdcy   ! ???
-    REAL(wp) ::   rsecfst, rseclast       ! ???
-@@ -67,6 +68,7 @@
+@@ -56,6 +56,7 @@
+       !
+       INTEGER ::   jk, jn   ! dummy loop indices
+       REAL(wp)::   ztrai    ! local scalar
++      LOGICAL ::   ll_trcstat ! local logical
+       CHARACTER (len=25) ::   charout   !
+       !!-------------------------------------------------------------------
+       !
+@@ -67,6 +68,8 @@
           r2dttrc = 2. * rdttrc       ! = 2 rdttrc (leapfrog)
        ENDIF
        !
-+      IF( kt == nittrc000 )  lltrcstat = ln_ctl .OR. (sn_cfctl%l_config .AND. sn_cfctl%l_trcstat)
++      ll_trcstat  = ( ln_ctl .OR. sn_cfctl%l_trcstat ) .AND. &
++     &              ( ( MOD( kt, sn_cfctl%ptimincr ) == 0 ) .OR. ( kt == nitend ) )
        IF( kt == nittrc000 .AND. lk_trdmxl_trc )  CALL trd_mxl_trc_init    ! trends: Mixed-layer
        !
        IF( .NOT.ln_linssh ) THEN                                           ! update ocean volume due to ssh temporal evolution
-@@ -108,7 +110,7 @@
+@@ -108,7 +111,7 @@
           !
        ENDIF
        !
 -      IF (ln_ctl ) THEN
-+      IF (lltrcstat) THEN
++      IF (ll_trcstat) THEN
           ztrai = 0._wp                                                   !  content of all tracers
           DO jn = 1, jptra
@@ -340 +353 @@
 ''' 5. OCE/LBC/mppini.F90 '''
 
-Control over the creation of multiple layout.dat files is easy to implement and so has been done as an example of how output in this category will be handled in future. For this category, the area subsetting can be used to restrict which areas produce files. The standard layout.dat file, produced by narea = 1 is always produced.
+Control over the creation of multiple layout.dat files is easy to implement and so has been done as an example of how output in this category will be handled in future. For this category, the area subsetting can be used to restrict which areas produce files. The standard layout.dat file, produced by narea = 1 is always produced. Also, OCE/SBC/sbcfwb.F90 has been fixed so that only one EMPave.dat file is ever read or produced. Ultimately, only one processor should read this file and MPI_BCAST the values to all others as a more scalable solution (for name lists too?).
 
 {{{#!diff
@@ -399 +412 @@
        DEALLOCATE(iin, ijn, ii_nono, ii_noea, ii_noso, ii_nowe,    &
           &       iimppt, ijmppt, ibondi, ibondj, ipproc, ipolj,   &
+
+
+Index: OCE/SBC/sbcfwb.F90
+===================================================================
+--- OCE/SBC/sbcfwb.F90  (revision 10530)
++++ OCE/SBC/sbcfwb.F90  (working copy)
+@@ -143,7 +143,7 @@
+             qns(:,:) = qns(:,:) - zcoef * sst_m(:,:) * tmask(:,:,1) ! account for change to the heat budget due to fw correction
+          ENDIF
+          !
+-         IF( kt == nitend .AND. lwp ) THEN            ! save fwfold value in a file
++         IF( kt == nitend .AND. lwm ) THEN            ! save fwfold value in a file (only one required)
+             CALL ctl_opn( inum, 'EMPave.dat', 'REPLACE', 'FORMATTED', 'SEQUENTIAL', -1, numout, .FALSE., narea )
+             WRITE( inum, "(24X,I8,2ES24.16)" ) nyear, a_fwb_b, a_fwb
+             CLOSE( inum )
 }}}
 
@@ -423 +451 @@
      set_namelist namelist_cfg ln_cdgw .true.
 }}}
+
+== Sebastien's comments on the first draft, many of which have already been incorporated into version2 above: ==
+Maybe you know this story, but it case you don’t know… :-)
+
+1) A long time ago  - in a galaxy far far away - ... ln_ctl was there (since ever?, at least in rev3) and was a control print of the trend over the first core.
+
+2) At rev258 https://forge.ipsl.jussieu.fr/nemo/changeset/258 the concept was modified and extended to be able to debug mpi reproducibility. 
+prtctl was added but, to my knowledge, never documented. 
+The idea is to compare the mean value of every trend in the code over (1) the local mpi domain and (2) the same domain but in a 1 core simulation (or n core as soon as the domain you want to test is included in your mpi subdomain). You compare you mpp.output_xxxx and mono.output_xxxx files and the first difference tells you which core and which trend is the first to diverge.
+
+3) Next, as prtctl was dealing with mpp.output_xxxx and mono.output_xxxx files, we again extended the use of ln_ctl to control the creation of any file trough the definition of lwp: https://forge.ipsl.jussieu.fr/nemo/changeset/1579
+This was a bad idea as, with 1 variable (ln_ctl), we control 2 different concepts : (1) mpp debug which requires the production of mpp.output_xxxx and mono.output_xxxx files and (2) the control of all other outputs files (ocean.output, run.stat, layout.dat…)
+
+4) For Performance issues, with Eric, we wanted to be able to remove the globsum in stpctl. At some point we thought that we should introduce a new namelist variable something like in_fast or ln_prod or ln_debug or nn_debug which would allow us to reduce the printed informations and use faster code (for example without globsum in stpctl). 
+After some discussions, we thought that our idea was not clear enough to be introduced in the code before the release in December. We therefore decided to postpone this and simply use ln_ctl to switch on/off the globsum (because if ln_ctl = T, you don’t care of the performances so you can do globsum in stpctl). Once you switch off the globsum, the data in run.stat are useless, so we also switch off run.stat files.
+
+
+So Today, ln_ctl is mixing different functionalities coming from theses 4 layers of developments.
+I think we should take advantage of your development to clean-up this mess!
+
+One proposition (to be discussed) could be to split ln_ctl into 3 functionalities: 
+ - mpp debug associated to prtctl. That could be renamed, for example in prttrd or trdprt for "trend print". Or mppdbg for mpp debug? this part requires the creation of the mpp.output_xxxx and mono.output_xxxx files which, to me, should be controlled only by the activation or not of prtctl
+ - something (a logical, an integer, a structure) related to the balance between verbosity-control-debug and performance. This would control/replace/be linked with the creation of run.stat files, nn_print, ln_timing maybe also layout.dat
+ - use your sn_ctl to control other files 
+
+
+Other minor points :
+ - EMPave.dat : I had a quick look but it seems that this file is the same for each core (contains only the year and global mean). So we should not offer the possibility to create EMPave.dat_xxxx files. Either all cores read the same file (as we do for the nameliste) or only core 0 read it and use mpi_broadcast to send the informations to all other cores (as, I think, we should also do for the name lists). '''First part done, will consider use of mpi_bcast in 2019'''
+ - We have files created by oce, si3 and top. Maybe, It is strange if a part of OCE, control the creation of top files like tracer.stat, no? '''To be discussed '''
+ - instead of having to test sn_cfctl%l_config everywhere as for example in "sn_cfctl%l_config .AND. sn_cfctl%l_trcstat”, I would, in nemo_set_cfctl, do sn_cfctl%l_trcstat = setto .AND. sn_cfctl%l_config ''' Actually the test of l_config isn't needed at the lower levels, removed '''
+ - there is also time.step file ''' Not sure we ever want to switch this off but it can now be updated less frequently according to ptimincr '''
+ - do we really need the sn_ctl structure? why no a simple list of logical and integer? ''' To be discussed '''
+ - I would like to “promote” the use of ln_timing, except in production mode, so people become used to look at this file to see the main bottlenecks in computational coast and communications… ''' Think this is a separate issue unless multiple files are produced? '''
 == Tests