Version 2 (modified by acc, 13 months ago) (diff)

Options for improving namelist file access

This page outlines some ideas for reducing the sequential disk access to namelist files at start-up. The current method, where every process repeatedly reads and rewinds a few small text files, is only as scalable as the file-system allows and is unlikely to be fit for purpose when moving to higher core counts.

A simple option, that requires very little code change, is to replace the current unit number variables with character arrays that contain the whole of the corresponding namelist file. I.e. use F90 internal files so that disk access is performed only once in a single sequential read by each process.

If necessary, disk access could be reduced further by having only one process read each namelist and broadcast the array to all others. I believe this is now feasible since Seb's recent updates remove any namelist access before MPI initialisation. This should be considered at a future stage.

Stage 1

  • Replace namelist unit variables with allocatable character arrays, e.g. in in_out_manger.F90:

   INTEGER ::   numnam_ref      =   -1      !: logical unit for reference namelist
   INTEGER ::   numnam_cfg      =   -1      !: logical unit for configuration specific namelist

become:

   CHARACTER(:), ALLOCATABLE :: numnam_ref    !: character buffer for reference namelist
   CHARACTER(:), ALLOCATABLE :: numnam_cfg    !: character buffer for configuration namelist
  • Replace ctl_opn statements with calls to a new routine that will fill the buffer, e.g.: in nemogcm.F90:
                  CALL ctl_opn( numnam_ref,        'namelist_ref',     'OLD', 'FORMATTED', 'SEQUENTIAL', -1, -1, .FALSE. )
                  CALL ctl_opn( numnam_cfg,        'namelist_cfg',     'OLD', 'FORMATTED', 'SEQUENTIAL', -1, -1, .FALSE. )

become:

                  CALL load_nml( numnam_ref,       'namelist_ref',                                           -1, .FALSE. )
                  CALL load_nml( numnam_cfg,       'namelist_cfg',                                           -1, .FALSE. )
  • Remove all REWIND statements involving the old unit variables (REWIND is not permitted (or necessary) on internal files)
  • Remove any CLOSE statements involving the old unit numbers. the load_nml routine will have closed the namelist file after reading.

Job done.

There are however a few caveats:

  • To be memory efficient, the character arrays should be allocated with the minimum length required to accommodate the namelist information. This requires each file to be read twice: once to determine and sum up the trimmed length of each line (comments can be stripped at this stage); and once to fill the newly allocated buffer. The example load_nml routine shown below achieves this. As an indication of requirements, this routine compacts the namelist_ref file into about 33KB. Further savings could be made by reducing multiple spaces to single spaces but the need to preserve any character strings within the namelists would complicate the code and would only save around 20%.
  • Reads from internal files always start at the beginning of the array (although substrings of the array can be provided to alter the starting point). This means the current practise in BDY of having multiple, identically named namelist blocks depending on the number of boundary segments will not work with internal files. Scanning the array to determine starting points for each block is possible but messy because of the need to support FORTRAN's case-independent attitude. A simpler solution will be to insist on appending a count to second and subsequent occurrences of the same block.
  • To be considered: will this work with AGRIF? load_nml still uses ctl_opn to open the namelist file so the correct file will be opened but will each nest get independent copies of the character arrays? What does Agrif_Get_Unit currently do?

The prototype load_nml routine (added to lib_mpp.F90)

   SUBROUTINE load_nml( cdnambuff , cdnamfile, kout, ldwp)
      CHARACTER(:)    , ALLOCATABLE, INTENT(INOUT) :: cdnambuff
      CHARACTER(LEN=*), INTENT(IN )                :: cdnamfile
      CHARACTER(LEN=256)                           :: chline
      INTEGER, INTENT(IN)                          :: kout
      LOGICAL, INTENT(IN)                          :: ldwp
      INTEGER                                      :: itot, iun, iltc, inl, ios
      !
      ! Check if the namelist buffer has already been allocated. Return if it has.
      !
      IF ( ALLOCATED( cdnambuff ) ) RETURN
      !
      ! Open namelist file
      !
      CALL ctl_opn( iun, cdnamfile, 'OLD', 'FORMATTED', 'SEQUENTIAL', -1, kout, ldwp )
      !
      ! First pass: count characters excluding comments and trimable white space
      !
      itot=0
  10  READ(iun,'(A256)',END=20,ERR=20) chline
      iltc = LEN_TRIM(chline)
      IF ( iltc.GT.0 ) THEN
       inl = INDEX(chline, '!')
       IF( inl.eq.0 ) THEN
        itot = itot + iltc + 1                             ! +1 for the newline character
       ELSEIF( inl.GT.0 .AND. LEN_TRIM( chline(1:inl-1) ).GT.0 ) THEN
        itot = itot + inl                                  !  includes +1 for the newline character
       ENDIF
      ENDIF
      GOTO 10
  20  CONTINUE
      !
      ! Allocate text cdnambuff for condensed namelist
      !
      ALLOCATE( CHARACTER(itot) :: cdnambuff )
      WRITE(*,*) 'ALLOCATED ', itot
      !
      ! Second pass: read and transfer pruned characters into cdnambuff
      !
      REWIND(iun)
      itot=1
  30  READ(iun,'(A256)',END=40,ERR=40) chline
      iltc = LEN_TRIM(chline)
      IF ( iltc.GT.0 ) THEN
       inl = INDEX(chline, '!')
       IF( inl.eq.0 ) THEN
        inl = iltc
       ELSE
        inl = inl - 1
       ENDIF
       IF( inl.GT.0 .AND. LEN_TRIM( chline(1:inl) ).GT.0 ) THEN
          cdnambuff(itot:itot+inl-1) = chline(1:inl)
          WRITE( cdnambuff(itot+inl:itot+inl), '(a)' ) NEW_LINE('A')
          itot = itot + inl + 1
       ENDIF
      ENDIF
      GOTO 30
  40  CONTINUE
      WRITE(*,*) 'ASSIGNED ',itot - 1
      !
      ! Close namelist file
      !
      CLOSE(iun)
      !write(*,'(32A)') cdnambuff
  END SUBROUTINE load_nml

Full development branch

This approach has been implemented and tested on a 2019 development branch:

https://forge.ipsl.jussieu.fr/nemo/browser/NEMO/branches/2019/dev_r11613_ENHANCE-04_namelists_as_internalfiles

This branch contains only the substantive changes; to compile and run tests all REWIND and CLOSE operations on the (no longer) units have to be removed. These changes affect many more files but can be scripted so are not included the branch in order to make a later merge easier. The scripts used to prepare code for testing are included below. With these additional changes this code passes most SETTE tests but the AGRIF preprocessor does not currently accept the new allocatable character strings. The conversion routine first bails on:

fcm_internal compile:F nemo /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/ppsrc/nemo/in_out_manager.f90 in_out_manager.f90
/home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/mk/agrifpp.sh  in_out_manager.f90 /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/inc  /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/ppsrc/nemo/in_out_manager.f90
syntax error line 135, file in_out_manager.f90 motclef = |:|
fcm_internal compile failed (256)

which is the line:

   CHARACTER(LEN=:), ALLOCATABLE :: numnam_ref      !: character buffer for reference namelist

AGRIF will need to be made to understand the (LEN=: ) syntax before this solution can be progressed.

All non-AGRIF SETTE tests are passing but note none of the current tests use multiple BDY namelists; a solution for that issue has yet to be coded.

Lists and scripts To minimise changes that need to be merged later, only the substantive changes have been checked into the development branch. These changes include the new routine in lib_mpp.F90, buffer declarations and replacement of any ctl_opn calls for namelist input. The list of files in this category is held in src/substantive.list and is:

ICE/icestp.F90 
OCE/IOM/in_out_manager.F90 
OCE/LBC/lib_mpp.F90 
OCE/nemogcm.F90 
OFF/nemogcm.F90 
SAO/nemogcm.F90 
SAS/nemogcm.F90 
TOP/PISCES/SED/sedini.F90 
TOP/PISCES/sms_pisces.F90 
TOP/PISCES/trcnam_pisces.F90 
TOP/trc.F90 
TOP/trcnam.F90

A list of files potentially containing REWIND and CLOSE statements (but excluding files in the substantive list) can be built by running the following script in the src directory:

#!/bin/bash
#
INUNITS=( numnam_ref numnam_cfg numnat_ref numnat_cfg numtrc_ref numtrc_cfg numnam_ice_ref numnam_ice_cfg numnamsed_ref numnamsed_cfg numnatp_cfg numnatp_ref )
RM_CMDS=( REWIND  CLOSE  )
#
# First build a list of files likely to need alteration
#
 if [ -f all_rewfiles.list ] ; then rm all_rewfiles.list; fi
 for n in `seq 0 1 $(( ${#INUNITS[*]} - 1 ))`
  do
   grep -l -i ${INUNITS[$n]} `find ./ -name '*.[Ffh]90'` >> all_rewfiles.list
   grep -l -i ${INUNITS[$n]} `find ../tests -name '*.[Ffh]90'` | grep -v WORK | grep -v BLD | grep -v NEMOFILES >> all_rewfiles.list
   grep -l -i ${INUNITS[$n]} `find ../cfgs -name '*.[Ffh]90'`  | grep -v WORK | grep -v BLD | grep -v NEMOFILES >> all_rewfiles.list
 done
 sort -u all_rewfiles.list > alluniq_rewfiles.list
 for f in `cat substantive.list`
 do
  ff=`echo $f | sed -e 's:/:\\\/:g'`
  echo $ff
  ed - alluniq_rewfiles.list << EOF
/$ff/d
w
q
EOF
 done

and this list can be used to target an editing script which removes any REWIND or CLOSE statements on converted units (also to be run in src):

#!/bin/bash
#
INUNITS=( numnam_ref numnam_cfg numnat_ref numnat_cfg numtrc_ref numtrc_cfg numnam_ice_ref numnam_ice_cfg numnamsed_ref numnamsed_cfg numnatp_cfg numnatp_ref )
RM_CMDS=( REWIND  CLOSE  )
#
for f in `cat alluniq_rewfiles.list`
do
 n=0
 for n in `seq 0 1 $(( ${#INUNITS[*]} - 1 ))`
 do
  for m in `seq 0 1 $(( ${#RM_CMDS[*]} - 1 ))`
  do
   perl -ni.bak -e 'print unless m@.*\s*'${RM_CMDS[$m]}'\s*\(\s*'${INUNITS[$n]}'\s*\).*@i'  $f
  done
 done
done
cd ../
rm `find ./ -name '*.bak'`   

.