NEMO version 3.3

NEW FEATURE: Support for NetCDF4 chunking and compression

NEW key: key_netcdf4

NetCDF4 libraries can be used to create "classic" NetCDF3 format data files or data files based on an underlying HDF5 format. This new format is not backwards compatible with NetCDF3 but programs linked with NetCDF4 libraries are able to read both formats through the same user interface. One feature of the new format which makes it an attractive option for use with NEMO is the ability to chunk and compress data. Users unfamiliar with the concepts are referred to the following published paper and the NetCDF4 documentation:

​http://www.unidata.ucar.edu/software/netcdf/papers/AMS_2008.pdf
​http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Chunking.html#Chunking

Briefly, chunking enables the user to define storage blocks for each array. When an extraction request requires data within a block, the whole block is read into memory cache. Chunking can be used to improve i/o performance when subseting large arrays by leveling out the access times for each axial extraction. For example with the standard (i,j,k,t) arrangement of NEMO volume data, horizontal slices (i,j) can be extracted optimally since data is stored contiguously. However, the extraction of E-W slices and N-S slices will involve greater strides through the volume and performance will suffer. With a judicious choice of chunking, however, E-W and N-S extractions need only address the blocks involved and much smaller strides through memory or across disk are required. Note that performance of the previously preferred direction will suffer so the optimal choice of chunk sizes depends critically on the expected access pattern to the data.

The real advantage of chunking, however, is that it enables the compression of data on a chunk by chunk basis. Several compression algorithms are available but good compression ratios can be achieved using standard gzip routines which are often available in optimised forms on supercomputing platforms. When enabled the compression occurs invisibly to the user and with only a small runtime overhead (although this may vary depending on model size and machine architecture). On some MPP platforms the runtime overhead is more than compensated by the reduction in I/O bottlenecks due to the reduced volume of I/O passing through the I/O subsystem.

Pages 211 to 213 of the NEMO reference manual have been updated with notes on the key_netcdf4 option. These pages are attached to this page.

Implementation in NEMO v3.3

Implementing support for new NetCDF4 features in NEMO has been done in a way which:

1. Continues to support users of NetCDF3 libraries
   
2. Optionally allows codes linked with NetCDF4 libraries to produce NetCDF3-compatible files
   
3. Allows control over chunk sizes via namelist parameters
   
4. Propagates netcdf4 control parameters to IOIPSL routines and I/O server routines (if active)
   

For this first implementation the parameters for the compression algorithm have been fixed to a level 1 gzip compression with the "shuffle" option activated. Tests have shown that increasing the compression level above 1 (it can range from 0 to 9) significantly increases the computational overhead for very little reduction in file size. The NetCDF4 control parameters that are under user control are simply:

   nn_nchunks_i   =   4       !  number of chunks in i-dimension
   nn_nchunks_j   =   4       !  number of chunks in j-dimension
   nn_nchunks_k   =   31      !  number of chunks in k-dimension
                              !  setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which
   ln_nc4zip      =   .TRUE.  !  (T) use netcdf4 chunking and compression
                              !  (F) ignore chunking information and produce netcdf3-compatible files

These values are read from the namelist file (namelist namnc4) and passed to ioipsl routines in a simple structure, defined in in_out_manager.F90:

!$AGRIF_DO_NOT_TREAT
   TYPE, PUBLIC :: snc4_ctl     !: netcdf4 chunking control structure (always needed for decision making)
      SEQUENCE
      INTEGER :: ni
      INTEGER :: nj
      INTEGER :: nk
      LOGICAL :: luse
   END TYPE snc4_ctl
   TYPE(snc4_ctl) :: snc4set    !: netcdf4 chunking control structure (always needed for decision making)
!$AGRIF_END_DO_NOT_TREAT

This structure could be easily extended if user control over other aspects of the compression settings are required but note the structure is defined in three separate locations:

./NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90
./NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90
./NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90

and would need to be updated in all three locations. For use with IO server processes, the namelist parameters also need to be set in the xmlio_server.def namelist. This duplication is unfortunate but appropriate since the IO server domains may have different sizes to the individual processor domains.

The structure is filled in routine dom_nam (if defined key_netcdf4) and passed as an optional argument to histbeg and histend routines. The structure is also available to iom_nf90 for use with restart files. If key_netcdf4 is not defined then the luse logical within the structure is set to false and the new features will not be used.

For IO server use, similar namelist parameters need to be supplied in the xmlio_server.def namelist. Such duplication is unfortunate but appropriate since the io server domains may well be different sizes to the individual processor domains.

Compiling without netcdf4 libraries

In order to accommodate the new code in histcom.f90 (which will not be used if either key_netcdf4 is not defined or ln_nc4zip is false) when compiling with netcdf3 libraries, dummy versions of the netcdf4-specific routines have been included in a new routine in IOIPSL/src: nc4dummy.F90. The F90 suffix is significant because the file needs to be preprocessed. All existing applications should be unaffected by these changes.

Exceptions to this claim may occur if users are already linking to NetCDF4 libraries (possibly without even being aware that they are). In this case many compilers will complain about duplicate modules and stop. Any such users will have to define key_netcdf4 and set ln_nc4zip false in the namelist to continue working as before.

Compiling with netcdf4 libraries

Generally, compiling with NetCDF4 and with support for the new features should simply involve defining key_netcdf4 and swapping:

-L%NCDF_HOME/lib -lnetcdf

in the FCM arch file, with:

-L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdf -lhdf5_fortran -lhdf5_hl -lhdf5 -lz

The following files are affected by changes associated with support for NetCDF4:

M      DOC/TexFiles/Chapters/Chap_MISC.tex
A      DOC/TexFiles/Namelist/namnc4

M      NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90
A      NEMOGCM/EXTERNAL/IOIPSL/src/nc4dummy.F90

M      NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90
M      NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_interface_ioipsl.f90

M      NEMOGCM/NEMO/OPA_SRC/DOM/domain.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/restart.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/iom_nf90.F90
M      NEMOGCM/NEMO/OPA_SRC/TRD/trdmld.F90
M      NEMOGCM/NEMO/OPA_SRC/TRD/trdvor.F90
M      NEMOGCM/NEMO/OPA_SRC/DIA/diaptr.F90
M      NEMOGCM/NEMO/OPA_SRC/DIA/diawri.F90

M      NEMOGCM/NEMO/LIM_SRC_2/limwri_2.F90

M      NEMOGCM/NEMO/TOP_SRC/trcdia.F90
M      NEMOGCM/NEMO/TOP_SRC/TRP/trdmld_trc.F90
M      NEMOGCM/NEMO/TOP_SRC/SED/sedwri.F90

M      NEMOGCM/NEMO/LIM_SRC_3/limwri.F90

M      NEMOGCM/NEMO/C1D_SRC/diawri_c1d.F90

M      NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/namelist
M      NEMOGCM/CONFIG/GYRE/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/GYRE/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/1_namelist
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_LIM/cpp_ORCA2_LIM.fcm   <---- Oops this shouldn't have been checked in. Reverted subsequently at revision 2365
M      NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/namelist
M      NEMOGCM/CONFIG/POMME/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/POMME/EXP00/namelist

A      NEMOGCM/ARCH/arch-ALTIX_NAUTILUS4.fcm

FAQ

Q. I get compile time error messages similar to these:

fortcom: Error: ./NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90, line 430: This name does not have a type, and must have an explicit type.   [NF90_HDF5]
    IF (snc4chunks%luse) m_c = NF90_HDF5  ! will conflict if compiling with netcdf4 libraries without "key_netcdf4"
-------------------------------^
fortcom: Error: ./NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90, line 1566: This name does not have a type, and must have an explicit type.   [NF90_DEF_VAR_CHUNKING]
           iret = NF90_DEF_VAR_CHUNKING(nfid, nvid, ichunkalg, ichunksz) ! will conflict if compiling with netcdf4 libraries without "key_netcdf4"
------------------^
fortcom: Error: ./nemo_v3_3_beta/NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90, line 1568: This name does not have a type, and must have an explicit type.   [NF90_DEF_VAR_DEFLATE]
           iret = NF90_DEF_VAR_DEFLATE(nfid, nvid, ishuffle, ideflate, ideflate_level) ! will conflict if compiling with netcdf4 libraries without "key_netcdf4"
------------------^

what is wrong?

A. This can occur for one of two reasons:

1. You are picking up a NetCDF3 netcdf.mod module file (via the USE netcdf statement) but you have key_netcdf4 defined. Undefine key_netcdf4 or check include paths in the FCM arch file.
   
2. You are picking up a NetCDF4 netcdf.mod module file (via the USE netcdf statement) but you have NOT defined key_netcdf4. Add the key to the makenemo command using: add_key "key_netcdf4" or check include paths in the FCM arch file.