wiki:Doc/DataAnalyse

Version 11 (modified by servonnat, 4 years ago) (diff)

--

Data and Analyse


1. Data

1.1. CMIP archive

1.1.1. Start with CMIP archives

We highly recommend to read the user's guide of CMIP projects available at: https://pcmdi.llnl.gov/mips/

1.1.2. Access CMIP archives

CMIP archives are available through the ESPRI platform including CICLAD and ClimServ machines. You can create an account on ESPRI mesocentre to transparently log to CICLAD or ClimServ at https://meso-account.ipsl.fr/.

1.1.3. Dive into CMIP archives at IPSL

The IPSL-CM model produces CMIP-compliant files. The CMIP convention include the CF convention with additional rules from the WIP panel (ex. a CMIP file must describe only one variable). Once the IPSL-CM data are quality checked, the CMIP-compliant files are migrated to a dedicated filesystem at TGCC following the CMIP directory structure. This part of the TGCC filesystem is read-only mounted on the ESPRI mesocentre. On CICLAD and ClimServ, the CMIP data archives are then available under :

/bdd/CMIP6

/bdd/CMIP5

The CMIP data from the other GCMs are downloaded at the IDRIS and IPSL. They are organized in the same CMIP directory structure and transparently available in the same /bdd root.

Example: /bdd/CMIP5/output/IPSL provides access to the whole IPSL-CM data production for CMIP5 exercice. /bdd/CMIP5/output/BCC provides access to BCC data requested by the IPSL community.

We adopted an "on-demand" process to add CMIP data from other GCM. Please send your request at glipsl@ipsl.fr with the following template:

cat < EOF > my_template.txt
project=CMIP5
experiments=historical amip
models=IPSL-CM5A-LR CNRM-CM5
ensembles=all
variables[atmos][3hr]=cltc tas
variables[land][fx]=sftgif
variables[seaIce][mon]=sic evap
EOF

1.1.4. What's a dataset?

A "dataset" (as defined by ESGF) is one version of a data set resulting from a single simulation (i.e., characterized by a unique option of each CMIP facet before the version such as the institute, the model, the domain, the experiment, the frequency, the ensemble, etc.).

Examples: CMIP5 dataset: CMIP5/output1/IPSL/IPSL-CM5A-LR/ 1pctCO2/mon/atmos/Amon/r1i1p1/v20110427

A dataset is the finest granularity for ESGF publication.

1.2. THREDDS access (aka DODS service)

The IPSL ESGF node includes a THREDDS data serveur which replaces the deprecated DODS serveur. This service automatically provides free access to parts of filesystems or remotly-mounted filesystems. From TGCC, the shared space /ccc/work/cont003/thredds/$LOGIN is expose at https://vesg.ipsl.upmc.fr/thredds/catalog/work/catalog.html.

Be careful that the THREDDS spaces at TGCC should only includes hard-links of data from your project spaces on WORK. The THREDDS folders were not designed to host deep copies of your files because of high constraints on storage and inodes.

1.3. Observations/Reanalyses?

Many observational datasets and reanalyses are available thanks to Climserv on /bdd (ERAInterim, ERA5, EOBS, and others).

Have a look at it before downloading data to both save your time and make multiple copies of the same dataset on the disks!

2. Analyse

Many toolboxes/softwares are available on IPSL servers to do analyses (like CDO, nco, Ferret, Python, R). Additionnaly, the CliMAF python library and the C-ESM-EP evaluation package are in-house developments available for you to ease your analyses.

2.1. CliMAF: a Climate Model Assessment Framework

CliMAF (https://climaf.readthedocs.io/en/master/) is a python library to help you:

  • browse and find data in your archives (like CMIP, CORDEX, IPSL model outputs, or observation/reanalyses)
  • easily do pretreatments like period or geographical domain selection, regridding, computing climatologies
  • either on one dataset or on an ensemble
  • plot your results
  • gather the plots in a html page
  • all this taking advantage of a smart cache that automatically avoids recomputing an existing result

The CliMAF documentation (https://climaf.readthedocs.io/en/master/) has many examples in the form of html versions of jupyter notebooks. See here:

If you are interested in following the CliMAF activity and ask users questions, subscribe to the mailing list: https://climaf.readthedocs.io/en/master/community.html

2.2. The C-ESM-EP: the CliMAF Earth System Model Evaluation Platform

The C-ESM-EP (https://github.com/jservonnat/C-ESM-EP/wiki) is an evaluation package based on CliMAF and developed between IPSL, CNRM and CERFACS to apply evaluation diagnostics, routinely or on demand, to a list of simulations/models and compare them easily. The result is an html frontpage with links to the atlases (html pages gathering the results of the evaluation diagnostics), covering the scientific demand of the scientists working on the development of the coupled models and on the individual components (atmosphere, ocean, land surfaces, sea ice, biogeochemistry).

The C-ESM-EP documentation is here: https://github.com/jservonnat/C-ESM-EP/wiki Follow the main page step by step and you will find the elements to use it. If not, do not hesitate to contact J. Servonnat or post an issue on github.