wiki:docs/synda

ABOUT Synda

Synda basic usage is to mirror a relatively large subset of ESGF data on a local cluster or local machine and keep datasets versions up-to-date.

PLATFORM

Linux

DESCRIPTION

Synda is a Python program managing discovery/authentication/certificate/download processes from the ESGF archive (CMIP5 archive) in an easy way. It's a command line tool designed to facilitate the download of files hosted by the distributed digital repositories of the ESGF Federation. It has been developed by the Institut Pierre Simon Laplace (IPSL) in Paris, France.

The download of files is achieved through exploration of the ESGF data repositories governed by the configuration files in which user defines the search criteria for the file selection. These criteria pertains to metadata attributes used for discovery of climate data. These attributes are defined by Data Reference Syntax (DRS). Thus, a user can enter lists of values for variables, frequencies, experiments and ensemble numbers into a configuration as directed by already provided templates. Before running the application, the user also needs to enter the ESGF openID and password that can be obtained by registering at any of the ESGF gateways, which are required for the data download.

The user defines one or many templates. Each of them have a list of variables, frequencies, experiments and ensemble members. Using these templates, the program explores the ESGF grid and dowload all the corresponding available files. The program may be run regularly to download the possible new files. Typically each template is associated with an analysis (cfmip template, downscaling template and so on). Create as many template_name.txt as you want in the user_selections folder, following the user_selections/default.txt syntax.

MAIN FEATURES

  • manage ESGF archive download processes
  • simple data selection with model,experiment,realm and variable
  • discovery phase prior to download
  • parallel downloads
  • checksums verification
  • manage datasets version following new drs
  • incremental process (download only what's new)
  • download history stored in a db
  • provide various statistics about download status
  • transfer priority

INSTALLATION & CONFIGURATION

BASIC USAGE

  • search datasets
    • synda search FACET..
  • install a dataset
    • synda install DATASET..
  • start the download
    • synda daemon start
  • check download completion
    • synda queue status
  • stop the download
    • synda daemon stop
  • downloaded files location
    • $HOME/sdt/data

LICENSE

SELECTION FILE SAMPLE

CONTACT

  • <sdipsl AT ipsl DOT jussieu DOT fr>

ACKNOWLEDGMENT

This work has been undertaken by IPSL and has been funded by France-Grilles, European Grid Infrastructure and IPSL.

Last modified 17 months ago Last modified on 04/26/16 10:42:08