NEMO version 3.3 [[BR]] '''NEW FEATURE: Support for NetCDF4 chunking and compression'''[[BR]] '''NEW key: key_netcdf4'''[[BR]] '''Authors:''' Andrew Coward and Steven Alderson (NOCS) Page updated to reflect modifications made at revision 2364 to fix compilation problems (#755) NetCDF4 libraries can be used to create "classic" NetCDF3 format data files or data files based on an underlying HDF5 format. This new format is not backwards compatible with NetCDF3 but programs linked with NetCDF4 libraries are able to read both formats through the same user interface. One feature of the new format which makes it an attractive option for use with NEMO is the ability to chunk and compress data. Users unfamiliar with the concepts are referred to the following published paper and the NetCDF4 documentation: http://www.unidata.ucar.edu/software/netcdf/papers/AMS_2008.pdf [[BR]] http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Chunking.html#Chunking Briefly, chunking enables the user to define storage blocks for each array. When an extraction request requires data within a block, the whole block is read into memory cache. Chunking can be used to improve i/o performance when subseting large arrays by leveling out the access times for each axial extraction. For example with the standard (i,j,k,t) arrangement of NEMO volume data, horizontal slices (i,j) can be extracted optimally since data is stored contiguously. However, the extraction of E-W slices and N-S slices will involve greater strides through the volume and performance will suffer. With a judicious choice of chunking, however, E-W and N-S extractions need only address the blocks involved and much smaller strides through memory or across disk are required. Note that performance of the previously preferred direction will suffer so the optimal choice of chunk sizes depends critically on the expected access pattern to the data. The real advantage of chunking, however, is that it enables the compression of data on a chunk by chunk basis. Several compression algorithms are available but good compression ratios can be achieved using standard gzip routines which are often available in optimised forms on supercomputing platforms. When enabled the compression occurs invisibly to the user and with only a small runtime overhead (although this may vary depending on model size and machine architecture). On some MPP platforms the runtime overhead is more than compensated by the reduction in I/O bottlenecks due to the reduced volume of I/O passing through the I/O subsystem. Pages 211 to 213 of the NEMO reference manual have been updated with notes on the key_netcdf4 option. These pages are attached to this page. '''Implementation in NEMO v3.3''' Implementing support for new NetCDF4 features in NEMO has been done in a way which: 1. Continues to support users of NetCDF3 libraries[[BR]] 2. Optionally allows codes linked with NetCDF4 libraries to produce NetCDF3-compatible files[[BR]] 3. Allows control over chunk sizes via namelist parameters[[BR]] 4. Propagates netcdf4 control parameters to IOIPSL routines and I/O server routines (if active)[[BR]] For this first implementation the parameters for the compression algorithm have been fixed to a level 1 gzip compression with the "shuffle" option activated. Tests have shown that increasing the compression level above 1 (it can range from 0 to 9) significantly increases the computational overhead for very little reduction in file size. The NetCDF4 control parameters that are under user control are simply: {{{ nn_nchunks_i = 4 ! number of chunks in i-dimension nn_nchunks_j = 4 ! number of chunks in j-dimension nn_nchunks_k = 31 ! number of chunks in k-dimension ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression ! (F) ignore chunking information and produce netcdf3-compatible files }}} These values are read from the namelist file (namelist ''namnc4'') and passed to ioipsl routines in a simple structure: {{{ !$AGRIF_DO_NOT_TREAT TYPE, PUBLIC :: snc4_ctl !: netcdf4 chunking control structure (always needed for decision making) SEQUENCE INTEGER :: ni INTEGER :: nj INTEGER :: nk LOGICAL :: luse END TYPE snc4_ctl !$AGRIF_END_DO_NOT_TREAT }}} This structure could be easily extended if user control over other aspects of the compression settings are required. The structure is defined in: {{{ ./NEMOGCM/EXTERNAL/IOIPSL/src/nc4interface.F90 }}} Note that, although this module resides in ''IOIPSL/src'', it should be considered as sitting above ''NEMO'', ''IOIPSL'' and ''IOSERVER''. It is used in: {{{ ./NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90 ./NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90 ./NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90 }}} and will need to be retained if ''IOIPSL'' is ever removed from the system. ''in_out_manager.F90'' and ''mod_ioserver_namelist.f90'' define independent variables of type ''snc4_ctl''. The structure is filled in routine ''dom_nam'' (if defined '''key_netcdf4''') and passed as an optional argument to ''histbeg'' and ''histend'' routines. The structure is also available to ''iom_nf90'' for use with restart files. If '''key_netcdf4''' is not defined then the ''luse'' logical within the structure is set to false and the new features will not be used. If '''key_iomput''' is defined then the namelist parameters also need to be set in the ''xmlio_server.def'' namelist. This duplication is unfortunate but appropriate since the IO server domains may have different sizes to the individual processor domains. In this case, the parameters read from the main namelist will apply to the restart files whilst those read from ''xmlio_server.def'' will apply to the mean files. '''Compiling without netcdf4 libraries''' In order to accommodate the new code in ''histcom.f90'' (which will not be used if either '''key_netcdf4''' is not defined or ''ln_nc4zip'' is false) when compiling with netcdf3 libraries, interface routines to the netcdf4-specific routines have been included in ''IOIPSL/src/nc4interface.F90''. The F90 suffix is significant because the file needs to be preprocessed. All existing applications should be unaffected by these changes. Linking to NetCDF3 libraries with '''key_netcdf4''' defined will fail at the link stage (see FAQ section below for a typical compiler message in these circumstances) '''Compiling with netcdf4 libraries''' Generally, compiling with NetCDF4 and with support for the new features should simply involve defining '''key_netcdf4''' and swapping: {{{ -L%NCDF_HOME/lib -lnetcdf }}} in the FCM arch file, with: {{{ -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdf -lhdf5_fortran -lhdf5_hl -lhdf5 -lz }}} If '''key_netcdf4''' is NOT defined whilst linking to NetCDF4 libraries, then access to the new features is disabled and only NetCDF3-compatible files can be produced regardless of namelist parameter settings. '''List of changed files''' The following files are affected by changes associated with support for NetCDF4: {{{ M DOC/TexFiles/Chapters/Chap_MISC.tex A DOC/TexFiles/Namelist/namnc4 M NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90 A NEMOGCM/EXTERNAL/IOIPSL/src/nc4interface.F90 M NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90 M NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_interface_ioipsl.f90 M NEMOGCM/NEMO/OPA_SRC/DOM/domain.F90 M NEMOGCM/NEMO/OPA_SRC/IOM/restart.F90 M NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90 M NEMOGCM/NEMO/OPA_SRC/IOM/iom_nf90.F90 M NEMOGCM/NEMO/OPA_SRC/TRD/trdmld.F90 M NEMOGCM/NEMO/OPA_SRC/TRD/trdvor.F90 M NEMOGCM/NEMO/OPA_SRC/DIA/diaptr.F90 M NEMOGCM/NEMO/OPA_SRC/DIA/diawri.F90 M NEMOGCM/NEMO/LIM_SRC_2/limwri_2.F90 M NEMOGCM/NEMO/TOP_SRC/trcdia.F90 M NEMOGCM/NEMO/TOP_SRC/TRP/trdmld_trc.F90 M NEMOGCM/NEMO/TOP_SRC/SED/sedwri.F90 M NEMOGCM/NEMO/LIM_SRC_3/limwri.F90 M NEMOGCM/NEMO/C1D_SRC/diawri_c1d.F90 M NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/xmlio_server.def M NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/namelist M NEMOGCM/CONFIG/GYRE/EXP00/xmlio_server.def M NEMOGCM/CONFIG/GYRE/EXP00/namelist M NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/xmlio_server.def M NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/namelist M NEMOGCM/CONFIG/ORCA2_LIM/EXP00/xmlio_server.def M NEMOGCM/CONFIG/ORCA2_LIM/EXP00/1_namelist M NEMOGCM/CONFIG/ORCA2_LIM/EXP00/namelist M NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/xmlio_server.def M NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/namelist M NEMOGCM/CONFIG/POMME/EXP00/xmlio_server.def M NEMOGCM/CONFIG/POMME/EXP00/namelist A NEMOGCM/ARCH/arch-ALTIX_NAUTILUS4.fcm }}} '''FAQ'''[[BR]] '''Q.''' I get compile time error messages similar to these: {{{ mpif90 -o nemo.exe ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/obj/model.o -L./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib -l__fcm__nemo -L/sw/packages/netcdf/lib -lnetcdf /sw/Intel/fce/10.0.026/lib/libimf.so: warning: warning: feupdateenv is not implemented and will always fail ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib/lib__fcm__nemo.a(nc4interface.o): In function `nc4interface_mp_set_nf90_def_var_deflate_': ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/ppsrc/ioipsl/nc4interface.f90:(.text+0x7c): undefined reference to `nf90_def_var_deflate_' ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/lib/lib__fcm__nemo.a(nc4interface.o): In function `nc4interface_mp_set_nf90_def_var_chunking_': ./nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/BLD/ppsrc/ioipsl/nc4interface.f90:(.text+0x23b): undefined reference to `nf90_def_var_chunking_' fcm_internal load failed (256) }}} what is wrong? '''A.''' '''key_netcdf4''' has been defined whilst linking with NetCDF3 libraries. Either undefine (del_key) '''key_netcdf4''' or set paths to NetCDF4 libraries in the FCM arch file.