Opened 2 months ago

Last modified 5 weeks ago

#2579 new Defect

PGI compiler does allow read values from namelist using buffers

Reported by: mcastril Owned by: systeam
Priority: low Milestone: Unscheduled
Component: DIA Version: trunk
Severity: minor Keywords: GPU, diagnostics, compilation, PGI,
Cc: Branch review: failed
MP ready?: no Task progress: Unspecified

Description (last modified by nemo)

Context

Compiler arch:

%NCDF_HOME           /apps/NETCDF/4.4.1.1/PGI/OPENMPI
%HDF5_HOME           /apps/HDF5/1.8.20/PGI/OPENMPI
%XIOS_HOME           /home/bsc99/bsc99214/trunk/xios-2.5.pw9
%NCDF_INC            -I%NCDF_HOME/include 
%NCDF_LIB            -L%NCDF_HOME/lib -lnetcdff -lnetcdf -L%HDF5_HOME/lib -lhdf5_hl -lhdf5 -lhdf5
%XIOS_INC            -I%XIOS_HOME/inc 
%XIOS_LIB            -L%XIOS_HOME/lib -lxios -lstdc++ -L/apps/GCC/8.1.0/lib/gcc/ppc64le-redhat-linux/lib64/
%OASIS_INC           
%OASIS_LIB           
%CPP                 cpp -P -traditional -Dkey_nosignedzero
%FC                  mpif90 
%FCFLAGS             -i4 -r8 -g -fast 
%FFLAGS              %FCFLAGS
%LD                  mpif90
%LDFLAGS             -i4 -r8 -g -fast  
%FPPFLAGS            -P -Uvector
%AR                  ar
%ARFLAGS             rs
%MK                  gmake
%USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB
%CC                  cc
%CFLAGS              -O0 -g

The bug activates a warning:

  ===>>> : W A R N I N G
 
          ===============
 
 end of record or file while reading namelist nammpp in reference namelist iostat =    -1

Analysis

investigating further there is a error in reading values from namelist files:

 FIO-F-228/namelist read/internal file/end of file reached without finding group.

Sample code to identify the problem:

       integer :: jpni,jpnj
       CHARACTER(LEN=104)    :: buff
       namelist/nammpp/ jpni,jpnj

       buff = ' /&nammpp jpni = 10 jpnj =  4   /&namctl /&namsto / EOF '

       read(buff,nml=nammpp)
       print *, jpni

       end

Recommendation

Example of workaround we implement to manage trunk

integer :: i,j, a, b, pos
CHARACTER(LEN=80)    :: buff
namelist/nammpp/ i,j
namelist/namctl/ a,b

buff=' /  &nammpp i = 10 j =  4 $ / &namctl a = 5 b = 3 / '
pos=index(buff,'&nammpp')
read(buff(pos:),nammpp)
print *, i, j
pos=index(buff,'&namctl')
read(buff(pos:),namctl)
print *, a,b
end

Commit History (0)

(No commits)

Change History (16)

comment:1 Changed 8 weeks ago by acc

Are you sure the problem here isn't just the leading / in your example namelist? A / without a preceding ampersand tends to halt processing on other compilers too; as reported in #2565

comment:2 Changed 8 weeks ago by nemo

  • Description modified (diff)

comment:3 Changed 8 weeks ago by nemo

  • Description modified (diff)

comment:4 Changed 8 weeks ago by mcastril

Thank you Andrew, and sorry because I didn't have the other issue in the radar. I told Maicon (our colleague working in the GPUs development) to check that.

Thanks for editing the description too. I tried to modify the ticket's description after I created it but I could modify everything but the description…

comment:5 Changed 7 weeks ago by mcastril

Hi Andrew,

We tested the proposed script which indeed detected a problem in the namelist_ref namelist. However, after fixing the namelist_ref our problem still persists, so it seems it is unrelated to the slashes

comment:6 Changed 7 weeks ago by mcastril

This script also shows a failure even without the trailing slash:

[bsc99214@p9login1 ~]$ cat test.f90
       integer :: jpni,jpnj,think
       CHARACTER(LEN=104)    :: buff
       namelist/nammpp/ jpni,jpnj
       namelist/some/ thing
       buff = '&some thing = 1 / &nammpp jpni = 10 jpnj =  4 / &namctl / &namsto / EOF '

       read(buff,nml=nammpp)
       print *, jpni

       end
[bsc99214@p9login1 ~]$ pgfortran test.f90 
[bsc99214@p9login1 ~]$ ./a.out 
FIO-F-228/namelist read/internal file/end of file reached without finding group.
 In source file test.f90, at line number 7
[bsc99214@p9login1 ~]$ gfortran test.f90 
[bsc99214@p9login1 ~]$ ./a.out 
          10

Last edited 7 weeks ago by nemo (previous) (diff)

comment:7 Changed 7 weeks ago by acc

I don't have access to a PGI compiler so could you try this:

integer :: i,j, a, b
CHARACTER(LEN=132)    :: buff
CHARACTER(LEN=1)    :: csp
namelist/nammpp/ i,j
namelist/namctl/ a,b
csp = NEW_LINE('A')

buff='&nammpp i = 10 j =  4  /'//csp//' &namctl a = 5 b = 3 / '

print *, buff
read(buff,nml=nammpp)
print *, i,j
read(buff,nml=namctl)
print *, a,b
end

If this works, we can fix this for the PGI compiler by swapping csp definition in LBC/lib_mpp.F90 (load_nml.F90):

      !csp = NEW_LINE('A')
      ! a new line character is the best seperator but some systems (e.g.Cray)
      ! seem to terminate namelist reads from internal files early if they
      ! encounter new-lines. Use a single space for safety.
      csp = ' '
Last edited 7 weeks ago by nemo (previous) (diff)

comment:8 Changed 7 weeks ago by andmirek

It didn't work for me (output):

 &nammpp i = 10 j =  4  /
 &namctl a = 5 b = 3 /                                                                                     
           10            4
FIO-F-228/namelist read/internal file/end of file reached without finding group.
 In source file test3.F90, at line number 13

comment:9 Changed 7 weeks ago by acc

Ok, There is an issue with some compilers that will need a work-around. The 'big 3' all work and when it doesn't work it is because the behaviour is different between internal and external files. Regardless of the exact wording of the standard, this should not be acceptable. Here is a better test program and results with the compilers I've been able to test. This option has been in the trunk for at least 6 months (possibly longer) so clearly not affecting most users.

integer :: i,j, a, b, pos
CHARACTER(LEN=80)   :: buff
CHARACTER(LEN=1)    :: csp
namelist/nammpp/ i,j
namelist/namctl/ a,b

csp=NEW_LINE('A')
buff='&nammpp i = 10 j =  4/'//csp//' &namctl a = 5 b = 3 / '
open(unit=15,file='tryme.nl')
write(15,*) buff
close(15)
!
open(unit=15,file='tryme.nl')
read(15,nammpp) ; print *, 'nammpp from external file: ',i, j
read(15,namctl) ; print *, 'namctl from external file: ',a,b
close(15)
!
read(buff,nammpp) ; print *, 'nammpp from internal file: ',i, j
read(buff,namctl) ; print *, 'namctl from internal file: ',a,b
end
----------------------------
gfortran test
----------------------------
gfortran -o test_gf test.f90
./test_gf
 nammpp from external file:           10           4
 namctl from external file:            5           3
 nammpp from internal file:           10           4
 namctl from internal file:            5           3
----------------------------
crayftn test
----------------------------
ftn -o test_cftn test.f90
./test_cftn
 nammpp from external file:  10,  4
 namctl from external file:  5,  3
 nammpp from internal file:  10,  4
 namctl from internal file:  5,  3
----------------------------
ifort test
----------------------------
ifort -o test test.f90
./test
 nammpp from external file:           10           4
 namctl from external file:            5           3
 nammpp from internal file:           10           4
 namctl from internal file:            5           3
----------------------------
allinea test
----------------------------
armflang -o test_arm test.f90
./test_arm
 nammpp from external file:            10            4
 namctl from external file:             5            3
 nammpp from internal file:            10            4
FIO-F-228/namelist read/internal file/end of file reached without finding group.
 In source file test.f90, at line number 19
 &nammpp i = 10 j =  4/
 &namctl a = 5 b = 3 /
Last edited 7 weeks ago by nemo (previous) (diff)

comment:10 Changed 7 weeks ago by acc

This might suffice as a workaround. But use with caution because it will edit files in-place; therefore only use it on freshly checked out code with no local modifications AND check the results before using. This hasn't been tested so its more of a proposed work-around at the moment. Please confirm if it works (or can be made to work with a few modifications).

The proposal is this BASH/PERl script:

internal_nl_workaround.pl

INUNITS=( numnam_ref numnam_cfg numnat_ref numnat_cfg numtrc_ref numtrc_cfg numnam_ice_ref numnam_ice_cfg numnamsed_ref numnamsed_cfg numnatp_cfg numnatp_ref )
#
# build a list of files that need to be changed
#
listfile=tmplistfile$$.txt
for iunit in ${INUNITS[@]}
do
  grep -l $iunit `find ./ -name '*.[fFh]90'` >>  $listfile
done
allfiles=`cat $listfile | sort -u`
echo $allfiles
#
if [ -f $listfile ] ; then rm $listfile; fi
for f in  $allfiles
do
 echo "Working on " $f
 n=0
 for n in `seq 0 1 $(( ${#INUNITS[*]} - 1 ))`
 do
   numnam=${INUNITS[$n]}
   perl -ni -e 'unless ( m@.*\s*READ\s*\(\s*'${INUNITS[$n]}'\s*,\s*[a-z,0-9]*.*@) { print  } else { $line= $_ ; $line=~s@(.*\s*READ\s*\()(\s*)('${INUNITS[$n]}')(\s*,\s*)([a-z0-9_]*)(.*)@\1\2\3(INDEX(\3,"\5")-1:)\4\5\6@i ; print $line }' $f
 done
done

which can be run in the top directory (on freshly checked out code).

It should locate all the files containing the listed internal files and edit all the READ statements from the likes of:

$ sdiff -s  -w 190 ../trunk/src/OCE/nemogcm.F90 src/OCE/nemogcm.F90
                            Before                                                                              After
      READ  ( numnam_ref, namctl, IOSTAT = ios, ERR = 901 )  |   READ  ( numnam_ref(INDEX(numnam_ref,"namctl")-1:), namctl, IOSTAT = ios, ERR = 901 )
      READ  ( numnam_cfg, namctl, IOSTAT = ios, ERR = 902 )  |   READ  ( numnam_cfg(INDEX(numnam_cfg,"namctl")-1:), namctl, IOSTAT = ios, ERR = 902 )
      READ  ( numnam_ref, namcfg, IOSTAT = ios, ERR = 903 )  |   READ  ( numnam_ref(INDEX(numnam_ref,"namcfg")-1:), namcfg, IOSTAT = ios, ERR = 903 )
      READ  ( numnam_cfg, namcfg, IOSTAT = ios, ERR = 904 )  |   READ  ( numnam_cfg(INDEX(numnam_cfg,"namcfg")-1:), namcfg, IOSTAT = ios, ERR = 904 )

If it works, I'll put together the reverse transformation which will have to be used before commiting any code back to the repository.
This isn't bulletproof; one obvious assumption is that the ampersand is tight with the namelist name (hence the -1 after the index). I think we've been consistent with that but it needs verifying.

Last edited 7 weeks ago by nemo (previous) (diff)

comment:11 Changed 7 weeks ago by acc

I note Maicon has got a guarded but positive response from the nvidia developers of pgfortran:

https://forums.developer.nvidia.com/t/issue-with-namelist-read/156116

The best argument would seem to be that they are already supporting the 'extension' because it is possible to read multiple namelists from a single external file. All we are asking for is that the behaviour is consistent across internal and external files.
Just to complete the list of compilers with the test program from comment 9:

--------------------------------------
pgi compiler
--------------------------------------
pgfortran -v
Export NVCOMPILER=/lustre/projects/metoffice/pgi-compiler/Linux_x86_64/20.9

pgfortran -o test_pg test.f90
./test_pg
 nammpp from external file:            10            4
 namctl from external file:             5            3
 nammpp from internal file:            10            4
FIO-F-228/namelist read/internal file/end of file reached without finding group.
 In source file test.f90, at line number 19

comment:12 Changed 7 weeks ago by maicon

The script compiles fine with Intel and PGI, and only had to add these two modifications to PGI. I was not able to test because the namefiles and inputs are not matching.

  • sbc_phy.F90

    old new  
    10691069      REAL(wp), DIMENSION(jpi,jpj), INTENT(in) :: pz0       !: roughness length (for momentum) [m] 
    10701070      !------------------------------------------------------------------- 
    10711071      ! Scalar Re_r relation from Liu et al. 
     1072      REAL(wp), DIMENSION(8,2) :: dim8x2 
    10721073      REAL(wp), DIMENSION(8,2), PARAMETER :: & 
    1073          & XA = (/ 0.177, 1.376, 1.026, 1.625, 4.661, 34.904, 1667.19, 5.88e5,  & 
    1074          &         0.292, 1.808, 1.393, 1.956, 4.994, 30.709, 1448.68, 2.98e5 /) 
     1074         & XA = reshape((/ 0.177, 1.376, 1.026, 1.625, 4.661, 34.904, 1667.19, 5.88e5,  & 
     1075         &         0.292, 1.808, 1.393, 1.956, 4.994, 30.709, 1448.68, 2.98e5           & 
     1076         & /), shape(dim8x2)) 
    10751077      !! 
    10761078      REAL(wp), DIMENSION(8,2), PARAMETER :: & 
    1077          & XB = (/ 0., 0.929, -0.599, -1.018, -1.475, -2.067, -2.907, -3.935,  & 
    1078          &         0., 0.826, -0.528, -0.870, -1.297, -1.845, -2.682, -3.616 /) 
     1079         & XB = reshape((/ 0., 0.929, -0.599, -1.018, -1.475, -2.067, -2.907, -3.935,  & 
     1080         &         0., 0.826, -0.528, -0.870, -1.297, -1.845, -2.682, -3.616           & 
     1081         & /), shape(dim8x2)) 
    10791082      !! 
    10801083      REAL(wp), DIMENSION(0:8),   PARAMETER :: & 
    10811084         & XRAN = (/ 0., 0.11, 0.825, 3.0, 10.0, 30.0, 100., 300., 1000. /) 
  • stpctl.F90

     
    2525   USE lib_mpp         ! distributed memory computing 
    2626   USE zdf_oce ,  ONLY : ln_zad_Aimp       ! ocean vertical physics variables 
    2727   USE wet_dry,   ONLY : ll_wd, ssh_ref    ! reference depth for negative bathy 
    28  
     28   USE, INTRINSIC :: IEEE_ARITHMETIC 
    2929   USE netcdf          ! NetCDF library 
    3030   IMPLICIT NONE 
    3131   PRIVATE 
     
    152152         &  zmax(3) >=   0._wp .OR.   &                    ! negative or zero sea surface salinity 
    153153         &  zmax(4) >= 100._wp .OR.   &                    ! too large sea surface salinity ( > 100 ) 
    154154         &  zmax(4) <    0._wp .OR.   &                    ! too large sea surface salinity (keep this line for sea-ice) 
    155          &  ISNAN( zmax(1) + zmax(2) + zmax(3) ) ) ) THEN   ! NaN encounter in the tests 
     155         &  IEEE_IS_NAN( zmax(1) + zmax(2) + zmax(3) ) ) ) THEN   ! NaN encounter in the tests 
    156156         IF( lk_mpp .AND. ln_ctl ) THEN 
    157157            CALL mpp_maxloc( 'stpctl', ABS(sshn)        , ssmask(:,:)  , zzz, ih  ) 
    158158            CALL mpp_maxloc( 'stpctl', ABS(un)          , umask (:,:,:), zzz, iu  ) 
Last edited 7 weeks ago by nemo (previous) (diff)

comment:13 Changed 7 weeks ago by maicon

I tested the adjust script with PGI and still there is something wrong in the read of namelists. It works well with Intel compiler, I'm checking what is missing to add to the script.

Runtime error:

  ===>>> : E R R O R
 
          ===========
 
 All TOP tracers disabled: change namtrc setting or check if key_top is active

Again, in the same conditions, Intel compilers runs well.

Last edited 7 weeks ago by maicon (previous) (diff)

comment:14 Changed 7 weeks ago by maicon

The issue was because INDEX looks for the first match string, in the namelist_top_cfg, the INDEX(numnat_cfg,"namtrc") goes to namtcr_run. Adding a space solves the issue.

  • trcnam.F90

    old new  
    146146      IF(lwp) WRITE(numout,*) 'trc_nam_trc : read the passive tracer namelists' 
    147147      IF(lwp) WRITE(numout,*) '~~~~~~~~~~~' 
    148148 
    149       READ  ( numnat_ref(INDEX(numnat_ref,"namtrc")-1:), namtrc, IOSTAT = ios, ERR = 901) 
     149      READ  ( numnat_ref(INDEX(numnat_ref,"namtrc ")-1:), namtrc, IOSTAT = ios, ERR = 901) 
    150150901   IF( ios /= 0 )   CALL ctl_nam ( ios , 'namtrc in reference namelist' ) 
    151       READ  ( numnat_cfg(INDEX(numnat_cfg,"namtrc")-1:), namtrc, IOSTAT = ios, ERR = 902 ) 
     151      READ  ( numnat_cfg(INDEX(numnat_cfg,"namtrc ")-1:), namtrc, IOSTAT = ios, ERR = 902 ) 
    152152902   IF( ios >  0 )   CALL ctl_nam ( ios , 'namtrc in configuration namelist' ) 
    153153      IF(lwm) WRITE( numont, namtrc ) 
    154154 
Last edited 7 weeks ago by maicon (previous) (diff)

comment:15 Changed 6 weeks ago by maicon

The code proposals were tested with SETTE successfully.

comment:16 Changed 5 weeks ago by mcastril

So as I see it the space mentioned in comment 14 could be added to the perl tool. And the changes mentioned in 12 (sbc_phy.F90 and stpctl.F90) in the code.

Note: See TracTickets for help on using tickets.