| 19 | | == Substream 1.2 : Configuration manager == |
| | 19 | == Substream 1.2: NetCDF4 IO options == |
| | 20 | [[BR]] |
| | 21 | === Description and participants === |
| | 22 | NetCDF4 offers the opportunity to employ dataset chunking and compression |
| | 23 | algorithms to greatly reduce the volume of data written out by NEMO without any loss of |
| | 24 | precision. NetCDF4.0.1 is the latest, full release of the library and this has been |
| | 25 | successfully tested for use in NEMO by NOCS. Even with severe tests, involving ORCA025 |
| | 26 | configurations with biogeochemical tracers, there are no run-time performance issues |
| | 27 | of any concern. In fact on some I/O limited clusters a speed enhancement is achieved |
| | 28 | because the benefits of writing out fewer data outweigh the computational costs of in-memory |
| | 29 | compression. The great benefit though is in the reduction of filesizes, for example: |
| | 30 | |
| | 31 | |
| | 32 | {{{ |
| | 33 | File name Disk usage for netCDF3.X (MBytes) Disk usage for netCDF4.0 (Mbytes) Reduction factor |
| | 34 | *grid T*.nc 1500 586 2.56 |
| | 35 | *grid U*.nc 677 335 2.02 |
| | 36 | *grid V*.nc 677 338 2.00 |
| | 37 | *grid W*.nc 3300 929 3.55 |
| | 38 | *icemod*.nc 208 145 1.43 |
| | 39 | |
| | 40 | Table 11: Effect of chunking and compression on the size of the NEMO 3.0 output files. |
| | 41 | For each file name the usage is the sum of all 221 individual processor output files. |
| | 42 | }}} |
| | 43 | The code changes required to use netcdf4 within NEMO are relatively straight-forward and involve |
| | 44 | minor changes to: iom_def.F90, iom.F90, iom_nf90.F90 and restart.F90 in the NEMO code and: |
| | 45 | histcom.f90 in IOIPSL. Changes are also required in the relevant makefiles to link in the netcdf4 |
| | 46 | library (-lnetcdff -lnetcdf) and the HDF5 and compression libraries that underlie netcdf4 (-lhdf5_fortran -lhdf5 -lhdf5_hl -lz). The need to make changes to IOIPSL routines makes providing a solution |
| | 47 | which supports both netcdf3 and netcdf4 difficult (preprocessor keys are not used in IOIPSL). This |
| | 48 | requires some discussion. The uptake of netcdf4 in NEMO may also be restricted by slow uptake of |
| | 49 | 3rd party software providers. Any compilable code can simply be relinked to the new libraries. Existing |
| | 50 | utilities will then be able to read netCDF4 files (and write netCDF3 files). Code changes are only |
| | 51 | necessary if it is required to write new netCDF4 files. However, commercial packages, such as IDL, will |
| | 52 | need netCDF4 support provided by the vendor (planned for early 2010). Other packages may fare better; NOCS |
| | 53 | has been able to compile netCDF4-compatible versions of Ferret and a netCDF4-compatible version of the |
| | 54 | mexnc toolkit for use with matlab (versions 2008a onwards). Details are available on request. |
| | 55 | === Advancement === |
| | 56 | |
| | 57 | ---- |
| | 58 | |
| | 59 | == Substream 1.3 : Configuration manager == |
