NEMO version 3.3

NEW FEATURE: Support for NetCDF4 chunking and compression

NEW key: key_netcdf4

Authors: Andrew Coward and Steven Alderson (NOCS)

Page updated to reflect modifications made at revision 2364 to fix compilation problems (#755)

NetCDF4 libraries can be used to create "classic" NetCDF3 format data files or data files based on an underlying HDF5 format. This new format is not backwards compatible with NetCDF3 but programs linked with NetCDF4 libraries are able to read both formats through the same user interface. One feature of the new format which makes it an attractive option for use with NEMO is the ability to chunk and compress data. Users unfamiliar with the concepts are referred to the following published paper and the NetCDF4 documentation:

​http://www.unidata.ucar.edu/software/netcdf/papers/AMS_2008.pdf
​http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Chunking.html#Chunking

Briefly, chunking enables the user to define storage blocks for each array. When an extraction request requires data within a block, the whole block is read into memory cache. Chunking can be used to improve i/o performance when subseting large arrays by leveling out the access times for each axial extraction. For example with the standard (i,j,k,t) arrangement of NEMO volume data, horizontal slices (i,j) can be extracted optimally since data is stored contiguously. However, the extraction of E-W slices and N-S slices will involve greater strides through the volume and performance will suffer. With a judicious choice of chunking, however, E-W and N-S extractions need only address the blocks involved and much smaller strides through memory or across disk are required. Note that performance of the previously preferred direction will suffer so the optimal choice of chunk sizes depends critically on the expected access pattern to the data.

The real advantage of chunking, however, is that it enables the compression of data on a chunk by chunk basis. Several compression algorithms are available but good compression ratios can be achieved using standard gzip routines which are often available in optimised forms on supercomputing platforms. When enabled the compression occurs invisibly to the user and with only a small runtime overhead (although this may vary depending on model size and machine architecture). On some MPP platforms the runtime overhead is more than compensated by the reduction in I/O bottlenecks due to the reduced volume of I/O passing through the I/O subsystem.

Pages 211 to 213 of the NEMO reference manual have been updated with notes on the key_netcdf4 option. These pages are attached to this page.

Implementation in NEMO v3.3

Implementing support for new NetCDF4 features in NEMO has been done in a way which:

1. Continues to support users of NetCDF3 libraries
   
2. Optionally allows codes linked with NetCDF4 libraries to produce NetCDF3-compatible files
   
3. Allows control over chunk sizes via namelist parameters
   
4. Propagates netcdf4 control parameters to IOIPSL routines and I/O server routines (if active)
   

For this first implementation the parameters for the compression algorithm have been fixed to a level 1 gzip compression with the "shuffle" option activated. Tests have shown that increasing the compression level above 1 (it can range from 0 to 9) significantly increases the computational overhead for very little reduction in file size. The NetCDF4 control parameters that are under user control are simply:

   nn_nchunks_i   =   4       !  number of chunks in i-dimension
   nn_nchunks_j   =   4       !  number of chunks in j-dimension
   nn_nchunks_k   =   31      !  number of chunks in k-dimension
                              !  setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which
   ln_nc4zip      =   .TRUE.  !  (T) use netcdf4 chunking and compression
                              !  (F) ignore chunking information and produce netcdf3-compatible files

These values are read from the namelist file (namelist namnc4) and passed to ioipsl routines in a simple structure:

!$AGRIF_DO_NOT_TREAT
   TYPE, PUBLIC :: snc4_ctl     !: netcdf4 chunking control structure (always needed for decision making)
      SEQUENCE
      INTEGER :: ni
      INTEGER :: nj
      INTEGER :: nk
      LOGICAL :: luse
   END TYPE snc4_ctl
!$AGRIF_END_DO_NOT_TREAT

This structure could be easily extended if user control over other aspects of the compression settings are required. The structure is defined in:

./NEMOGCM/EXTERNAL/IOIPSL/src/nc4interface.F90

Note that, although this module resides in IOIPSL/src, it should be considered as sitting above NEMO, IOIPSL and IOSERVER. It is used in:

./NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90
./NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90
./NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90

and will need to be retained if IOIPSL is ever removed from the system. in_out_manager.F90 and mod_ioserver_namelist.f90 define independent variables of type snc4_ctl.

The structure is filled in routine dom_nam (if defined key_netcdf4) and passed as an optional argument to histbeg and histend routines. The structure is also available to iom_nf90 for use with restart files. If key_netcdf4 is not defined then the luse logical within the structure is set to false and the new features will not be used.

If key_iomput is defined then the namelist parameters also need to be set in the xmlio_server.def namelist. This duplication is unfortunate but appropriate since the IO server domains may have different sizes to the individual processor domains. In this case, the parameters read from the main namelist will apply to the restart files whilst those read from xmlio_server.def will apply to the mean files.

Compiling without netcdf4 libraries

In order to accommodate the new code in histcom.f90 (which will not be used if either key_netcdf4 is not defined or ln_nc4zip is false) when compiling with netcdf3 libraries, interface routines to the netcdf4-specific routines have been included in IOIPSL/src/nc4interface.F90. The F90 suffix is significant because the file needs to be preprocessed. All existing applications should be unaffected by these changes. Linking to NetCDF3 libraries with key_netcdf4 defined will fail at the link stage (see FAQ section below for a typical compiler message in these circumstances)

Compiling with netcdf4 libraries

Generally, compiling with NetCDF4 and with support for the new features should simply involve defining key_netcdf4 and swapping:

-L%NCDF_HOME/lib -lnetcdf

in the FCM arch file, with:

-L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdf -lhdf5_fortran -lhdf5_hl -lhdf5 -lz

If key_netcdf4 is NOT defined whilst linking to NetCDF4 libraries, then access to the new features is disabled and only NetCDF3-compatible files can be produced regardless of namelist parameter settings.

List of changed files

The following files are affected by changes associated with support for NetCDF4:

M      DOC/TexFiles/Chapters/Chap_MISC.tex
A      DOC/TexFiles/Namelist/namnc4

M      NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90
A      NEMOGCM/EXTERNAL/IOIPSL/src/nc4interface.F90

M      NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90
M      NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_interface_ioipsl.f90

M      NEMOGCM/NEMO/OPA_SRC/DOM/domain.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/restart.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90
M      NEMOGCM/NEMO/OPA_SRC/IOM/iom_nf90.F90
M      NEMOGCM/NEMO/OPA_SRC/TRD/trdmld.F90
M      NEMOGCM/NEMO/OPA_SRC/TRD/trdvor.F90
M      NEMOGCM/NEMO/OPA_SRC/DIA/diaptr.F90
M      NEMOGCM/NEMO/OPA_SRC/DIA/diawri.F90

M      NEMOGCM/NEMO/LIM_SRC_2/limwri_2.F90

M      NEMOGCM/NEMO/TOP_SRC/trcdia.F90
M      NEMOGCM/NEMO/TOP_SRC/TRP/trdmld_trc.F90
M      NEMOGCM/NEMO/TOP_SRC/SED/sedwri.F90

M      NEMOGCM/NEMO/LIM_SRC_3/limwri.F90

M      NEMOGCM/NEMO/C1D_SRC/diawri_c1d.F90

M      NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/namelist
M      NEMOGCM/CONFIG/GYRE/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/GYRE/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/1_namelist
M      NEMOGCM/CONFIG/ORCA2_LIM/EXP00/namelist
M      NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/namelist
M      NEMOGCM/CONFIG/POMME/EXP00/xmlio_server.def
M      NEMOGCM/CONFIG/POMME/EXP00/namelist

A      NEMOGCM/ARCH/arch-ALTIX_NAUTILUS4.fcm

FAQ

Q. I get compile time error messages similar to these:

mpif90 -o nemo.exe ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/obj/model.o -L./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib -l__fcm__nemo -L/sw/packages/netcdf/lib -lnetcdf
/sw/Intel/fce/10.0.026/lib/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib/lib__fcm__nemo.a(nc4interface.o): In function `nc4interface_mp_set_nf90_def_var_deflate_':
./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/ppsrc/ioipsl/nc4interface.f90:(.text+0x7c): undefined reference to `nf90_def_var_deflate_'
./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib/lib__fcm__nemo.a(nc4interface.o): In function `nc4interface_mp_set_nf90_def_var_chunking_':
./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/ppsrc/ioipsl/nc4interface.f90:(.text+0x23b): undefined reference to `nf90_def_var_chunking_'
fcm_internal load failed (256)

what is wrong?

A. key_netcdf4 has been defined whilst linking with NetCDF3 libraries. Either undefine (del_key) key_netcdf4 or set paths to NetCDF4 libraries in the FCM arch file.