How to manipulate/compare/examine netcdf files using cdo, nco or ncview?

Author: S. Luyssaert and J. Ryder
Last revision: 2020/02/28, B. Guenet; 2020/05/11, A. Ducharne

ncview: a quick tool for vizualization

The quickest way of all to look at NetCDF files and to quickly plot some outputs is to use ncview:

ncview myJob_20060101_20061231_1M_sechiba_history.nc

Unfortunately the ncview application can be sometimes a little unstable.

nco

nco is a very powerful tool to print data, modify variable names, concatenate, etc on netcdf file. A detailled documentation can be found here ​http://nco.sourceforge.net/

We just summarize here some of the most useful nco commands often used in the orchidee context.

ncdump

This command print all the informations stored in your netcdf file. Some prefix can extract only the information you need.

To look at header information:

ncdump -h myJob_20060101_20061231_1M_sechiba_history.nc

To extract one variable (here for 'fluxlat', the latent heat flux)

ncdump -v fluxlat myJob_20060101_20061231_1M_sechiba_history.nc 

To extract one variable and show them in a tabulated way (here for 'fluxlat', the latent heat flux)

ncdump -v fluxlat -f fortran -n fluxlat myJob_20060101_20061231_1M_sechiba_history.nc 

ncrename

This command change the name of a given variable/dimension depending on the prefix you are using

To rename a variable within the file (here for 'fluxlat', the latent heat flux)

ncrename -v fluxlat,latent_heat_flux myJob_20060101_20061231_1M_sechiba_history.nc 

To rename a variable and create a new file (here for 'fluxlat', the latent heat flux)

ncrename -v fluxlat,latent_heat_flux myJob_20060101_20061231_1M_sechiba_history.nc mynewfile.nc

To rename a dimension within the file (here for 'lon', the longitude)

ncrename -d lon,longitude myJob_20060101_20061231_1M_sechiba_history.nc 

To rename a variable and create a new file (here for 'lon', the longitude)

ncrename -d lon,longitude myJob_20060101_20061231_1M_sechiba_history.nc mynewfile.nc

ncks

The nco ​kitchen sink offers a wealth of possibilities.

A very simple example is to append file f1.nc into f2.nc:

ncks –A f1.nc f2.c 

You can do do the same for only variables var1 and var2 from f1.nc (f1.nc is not modified in doing so):

ncks –v var1,var2 f1.nc var.c 

If you would like to remove a certain time series from a netcdf file (for instance, in this example, all data from 'temp_atmos_pres' in the file jamescdf.nc for which the height dimension is 20) with the output written to jamestest.nc

ncks -H -d height,20 -d temp_atmos_pres,: jamescdf.nc jamestest.nc

ncrcat, ncra

These tools are are very useful if you forgot the post-processing of your sechiba output.

To concatenate several nc files over time, you can use ​ncrcat. The example below can be used on all the sechiba_history.nc files in your Output directory to create a large nc file combining all variables and all years. You get a multi-variable TS file:

ncrcat *nc new.nc 

To restrict the operation to selected variables (var1 and var2 in the example):

ncrcat *nc –v var1,var2 new.nc 

​ncra does the same with a time average, to get climatologies.

For a shell script example to search specific files and combine them together to a single file, look ​here.

ncatted

Sometimes, one just need to add the attribute coordinates (linking to the variables defining the longitude and latitude positions) to enable the recognition of the grid that is used. In that cases the ncatted command might be useful

   ncatted -a coordinates,my_var,o,c,"nav_lon nav_lat" sst_data.nc

or to correct a wrong spelling of an attribute

   ncatted -a units,latitude,m,c,"degrees_north" force2002.nc

ncap2

The most powerful and quickly evolving tool from the nco set of functions

Search what you are looking on this growing list of answers ​forum nco

A simple example showing different capacities (creation of a variable, use of a mask, call to an attribute, count the total and the size of the field (we can restrict this operation to a dimension as shown on the second line)

 ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total())/missing_flag.size();print(missing_count)' stomate_restart.nc  ~/foo.nc
 ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total($x))/missing_flag.size($x);print(missing_count)' stomate_restart.nc  ~/foo.nc

cdo

An alternative to nco is cdo. They are basically doing similar things but dependending on what you want to do the command line might be easier with one or another. Details on cdo are here ​https://code.mpimet.mpg.de/projects/cdo cdo is useful for :

• changing of time axis
• regridding (many grid are already implemented and it is an even more versatile tool)
• complex operations...

List of classical cdo command (non exhaustive)

• print list of available operators
  

     cdo -v 
  

• print the version you are using (if very old, think of downloading the source and install a new version, as it is a tool that is evolving quickly)

     cdo -V
  

• more info on the file

     cdo sinfov my_file.nc
  

• grid description

     cdo griddes my_file.nc 
  

• print some information about the command remapcon

     cdo -h remapcon
  

• an example of a conservative regridding to an other resolution

     cdo remapcon,my_grid my_climatology.nc my_regridded_climatology.nc 
  

my_grid could be a file like :

   gridtype  = lonlat
   gridsize  = 220
   xname     = longitude
   xunits    = degrees_east
   yname     = latitude
   yunits    = degrees_north
   xsize     = 20
   ysize     = 11
   xfirst    = -18.75
   xinc      = 3.75
   yfirst    = 20

Operator chaining

It is one of the main features of CDO. Beware, however, that operators with an arbitrary list of input files cannot be combined with other operators.

Simple combination example:

   cdo sub -dayavg ifile2 -timavg ifile1 ofile

instead of:

  cdo timavg ifile1 tmp1

  cdo dayavg ifile2 tmp2

  cdo sub tmp2 tmp1 ofile

  rm tmp1 tmp2

Regridding

CDO can also be used to regrid NetCDF files. It is very straight and fast. The call has the following structure:

cdo -f nc -regriddingMethod,gridTemplate.nc input.nc output.nc

Note that Sebastiaan Luyssaert did not find how to increase the memory allocation for CDO, contrary to Ferret. Thus, to regrid very large files, Ferret seems to be the software of your choice. See examples ​here

There are three regridding methods: remapbil (bilinear interpolation), remapbic (bicubic interpolation), remapnn (nearest neighbour), remapcon (conservative regridding). Use remapnn and remapcon to avoid weird values if your variable is heterogeneous.

For the template file with your target grid, it can be any netcdf with regular lat/lon. A trick here is to use "full paths" and not relative paths to the files in order to work.

Some examples:

cdo -f nc -remapbil,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapbil.45N.nc
cdo -f nc -remapnn,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapnn.45N.nc

To install a newer version than the default one on asterix/obelix ==

Get files from ​https://code.zmaw.de/projects/cdo/files

./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr

make clean
make
make install

If you installed proj and hdf5 libraries somewhere, their can also be support for these file formats :

./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr --with-proj=/path/to/install --with-hdf5=/path/where/installed