How to manipulate/compare/examine netcdf files using cdo or nco?

Author: S. Luyssaert and J. Ryder
Last revision: 2020/02/28, B. Guenet

The quickest way of all to look at NetCDF files is to use ncview, i.e.:

ncview myJob_20060101_20061231_1M_sechiba_history.nc

Unfortunately the ncview application can be a little unstable (at least on my system anyway).

ncdump is more flexible, and is a great way to look at header information:

ncdump -h myJob_20060101_20061231_1M_sechiba_history.nc

or, for example to extract one column of data in a tabulated text format (here for 'fluxlat', the latent heat flux):

ncdump -v fluxlat -f fortran -n fluxlat myJob_20060101_20061231_1M_sechiba_history.nc > myJob_fluxlat.cdl

But there's a really useful script that can be used in IDL to import all of the data series at once, without having to spend time writing scripts to define input arrays first. First of all, download ​cdf2idl.pro, compile it within IDL and then input the following:

cdf2idl, '/home/rest_of_the_path/myJob_20060101_20061231_1M_sechiba_history.nc'

@myJob_20060101_20061231_1M_sechiba_history.idl

And then you have all of your variables automatically imported, each into arrays of an appropriate size, and they can be quickly plotted, e.g:

plot, tstep, tair(0,0,0,*)

(remember that IDL by default defines arrays as starting from 0, instead of from 1)

ncks: If you would like to remove a certain time series from a netcdf file (for instance, in this example, all data from 'temp_atmos_pres' in the file jamescdf.nc for which the height dimension is 20) with the output written to jamestest.nc

ncks -H -d height,20 -d temp_atmos_pres,: jamescdf.nc jamestest.nc

XXXXXXXXXXXXXX

Similar to Ferret (see also How To), CDO can also be used to regrid NetCDF files. It is very straight and fast. Contrary to Ferret I [SL] did not find how to increase the memory allocation for CDO. Thus for very large files Ferret seems to be the software of your choice.

The call has the following structure:

cdo -f nc -regriddingMethod,gridTemplate.nc input.nc output.nc

There are three regridding methods: remapbil (bilinear interpolation), remapbic (bicubic interpolation), remapnn (nearest neighbour), remapcon (conservative regridding). Use remapnn and remapcon to avoid weird values if your variable is heterogeneous.

For the template file with your target grid, it can be any netcdf with regular lat/lon. A trick here is to use "full paths" and not relative paths to the files in order to work.

Some examples:

cdo -f nc -remapbil,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapbil.45N.nc
cdo -f nc -remapnn,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapnn.45N.nc

ncap2

The most powerful and quickly evolving tool from the nco set of functions

See the file of exemples : {{:reunions_orchidee:ncap2.pdf| ncap2 file}} or search what you are looking on this growing list of answers ​forum nco

A simple exemple showing different capacities (creation of a variable, use of a mask, call to an attribute, count the total and the size of the field (we can restrict this operation to a dimension as shown on the second line)

<code>

ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total())/missing_flag.size();print(missing_count)' stomate_restart.nc ~/foo.nc ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total($x))/missing_flag.size($x);print(missing_count)' stomate_restart.nc ~/foo.nc

</code>

cdo

A command that handle many operations one would like to do when working on outputs of models of the Earth System :

• changing of time axis
• regridding (many grid are already implemented and it is an even more versatile tool)
• complex operations...

See the website of the project {{​https://code.zmaw.de/projects/cdo/wiki}} or the doc : {{:reunions_orchidee:cdo-manual.pdf| CDO Manual}} and the {{:reunions_orchidee:cdo_refcard.pdf | quickref}}

<code>

cdo -v # print list of available operators
cdo -V # version (if very old, think of downloading the source and install a new version, as it is a tool that is evolving quickly
cdo -h remapcon # informations about a command
cdo remapcon,my_grid my_climatology.nc my_regridded_climatology.nc # conservative regridding to an other resolution

</code>

my_grid could be a file like :

<code>

gridtype = lonlat gridsize = 220 xname = longitude xunits = degrees_east yname = latitude yunits = degrees_north xsize = 20 ysize = 11 xfirst = -18.75 xinc = 3.75 yfirst = 20

</code>

<code>

cdo sinfov my_file.nc #most info on the file cdo griddes my_file.nc #grid description (check that the type is not generic (which means unrecognised and thus not useable for mapping

</code>

Sometimes, one just need to add the attribute coordinates (linking to the variables defining the longitude and latitude positions) to enable the recognition of the grid that is used (see ​https://code.zmaw.de/boards/1/topics/55) <code>

ncatted -a coordinates,my_var,o,c,"nav_lon nav_lat" sst_data.nc

</code>

or to correct a wrong spelling of an attribute

<code>

ncatted -a units,latitude,m,c,"degrees_north" force2002.nc

</code>

Operator chaining

It is one of the main features of CDO. Use it as often as possible. But Note: Operatos with a arbitrary list of input files cannot be combined with other operators

# Simple combination: <code>

cdo sub -dayavg ifile2 -timavg ifile1 ofile

</code>

instead of :

<code>

cdo timavg ifile1 tmp1

cdo dayavg ifile2 tmp2

cdo sub tmp2 tmp1 ofile

rm tmp1 tmp2

</code>

To install a newer version than the default one on asterix/obelix

<code>

get files from ​https://code.zmaw.de/projects/cdo/files ./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr

make clean make make install

</code>

If you installed proj and hdf5 libraries somewhere, their can also be support for these file formats :

<code>

./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr --with-proj=/path/to/install --with-hdf5=/path/where/installed

</code>