Version 62 (modified by snguyen, 2 years ago) (diff) |
---|
IPSL-CM5A2-VLR configuration
Table of contents
Persons in charge: Sébastien Nguyen, Pierre Sepulchre
1. IPSL-CM5A2 model
IPSL-CM5A2 is an IPSL coupled climate model including atmosphere, land, ocean, sea ice and carbon cycle. The goal of this model is to reproduce IPSLCM5A-LR version, with reduced computing times to allow to run long simulations (i.e thousands of years) within reasonable time. Main characteritics of this model are ORCA2-LIM2-PISCES x LMD 96x95x39, old LMDZ physics, ORCHIDEE with Choisnel hydrology (2 layers). This model uses both recent component versions and recent technical developments (hybrid parallelization MPI-OpenMP, XIOS 2.0 as input-output library, OASIS3-MCT parallel coupler) that allow to reach 80 simulated years per day on about 300 computing cores. This model is available on Irene (TGCC) and JeanZay (IDRIS) supercomputers and benefits from last developments of libIGCM running environment.
IPSL-CM5A2 model includes :
- model components :
- LMDZ as atmospheric model ;
- NEMO as ocean model including sea ice (LIM2) and marine biogeochemistry (PISCES) ;
- ORCHIDEE as land model ;
- tools :
- OASIS3-MCT as parallel coupler ;
- XIOS 2.0 as I/O library ;
- libIGCM as running environment (scripts) to run the model and to perform post processing ;
2. Technical details
IPSL-CM5A2 model is available at following resolutions/configurations :
- IPSL-CM5A2-VLR: LMDZ (Old Physics) 96x95x39-ORCHIDEE (Choisnel) - NEMO-LIM2-PISCES ORCA2
The resolution of LMDZ is 96x95 (3.75° in longitude and 1.875° in latitude) with 39 vertical levels. The ocean configuration is ORCA2 : global ocean with a tripolar grid with one South Pole, one North Pole above Siberia and one North Pole above northern America. The resolution is 2°. In the tropical region, the latitudinal resolution decreases to 1/2°. There are 31 vertical levels.
IPSL-CM5A2 is composed of following components and tools and available within IPSLCM5A2.1 release:
#-H- IPSLCM5A2.1 IPSLCM5A2.1 coupled configuration #-H- IPSLCM5A2.1 2nd frozen IPSLCM5A2 version #-H- IPSLCM5A2.1 NEMOGCM branch nemo_v3_6_STABLE revision 6665 #-H- IPSLCM5A2.1 XIOS2 trunk revision 1551 #-H- IPSLCM5A2.1 IOIPSL/src svn tags/v2_2_2 #-H- IPSLCM5A2.1 LMDZ5 branches/IPSLCM5A2.1 rev 3342 #-H- IPSLCM5A2.1 ORCHIDEE branches/ORCHIDEE-IPSLCM5A2.1 rev 5307 #-H- IPSLCM5A2.1 OASIS3-MCT 2.0_branch rev 1818 #-H- IPSLCM5A2.1 IPSLCM5A2 svn #-H- IPSLCM5A2.1 libIGCM trunk rev 1477 #-M- IPSLCM5A2.1 arnaud.caubel@lsce.ipsl.fr #-C- IPSLCM5A2.1 IOIPSL/tags/v2_2_2/src HEAD 8 IOIPSL/src modeles #-C- IPSLCM5A2.1 branches/ORCHIDEE-IPSLCM5A2.1/ORCHIDEE 5307 14 ORCHIDEE modeles #-C- IPSLCM5A2.1 branches/OASIS3-MCT_2.0_branch/oasis3-mct 1818 15 oasis3-mct . #-C- IPSLCM5A2.1 LMDZ5/branches/IPSLCM5A2.1 3342 11 LMDZ modeles #-C- IPSLCM5A2.1 CONFIG/UNIFORM/v6/IPSLCM5A2 4041 8 IPSLCM5A2 config #-C- IPSLCM5A2.1 trunk/libIGCM 1477 10 libIGCM . #-C- IPSLCM5A2.1 branches/2015/nemo_v3_6_STABLE/NEMOGCM 6665 7 . modeles #-C- IPSLCM5A2.1 XIOS/trunk 1551 12 XIOS modeles
Note that previous IPSLCM5A2.0 release (with bug in runoff conservation) is still available.
Newer version IPSLCM5A2.2 has been released. It uses the following components:
#-H- IPSLCM5A2.2 IPSLCM5A2.2 coupled configuration #-H- IPSLCM5A2.2 Working IPSLCM5A2 version #-H- IPSLCM5A2.2 NEMOGCM branch nemo_v3_6_STABLE revision 6665 #-H- IPSLCM5A2.2 XIOS2 branchs/xios-2.5 revision 1903 #-H- IPSLCM5A2.2 IOIPSL/src svn tags/v2_2_2 #-H- IPSLCM5A2.2 LMDZ5 branches/IPSLCM5A2.1 rev 3907 #-H- IPSLCM5A2.2 ORCHIDEE branches/ORCHIDEE-IPSLCM5A2.1 rev 7376 #-H- IPSLCM5A2.2 OASIS3-MCT 2.0_branch (rev 1818 Cerfacs server, rev 4775 IPSL server) #-H- IPSLCM5A2.2 IPSLCM5A2 svn 6056 #-H- IPSLCM5A2.2 libIGCM trunk rev 1534 #-M- IPSLCM5A2.2 platform-extract@listes.ipsl.fr #-C- IPSLCM5A2.2 IOIPSL/tags/v2_2_2/src HEAD 8 IOIPSL/src modeles #-C- IPSLCM5A2.2 branches/ORCHIDEE-IPSLCM5A2.1/ORCHIDEE 7376 14 ORCHIDEE modeles #-C- IPSLCM5A2.2 CPL/oasis3-mct/branches/OASIS3-MCT_2.0_branch 4775 8 oasis3-mct . #-C- IPSLCM5A2.2 LMDZ5/branches/IPSLCM5A2.1 3907 20 LMDZ modeles #-C- IPSLCM5A2.2 CONFIG/UNIFORM/v6/IPSLCM5A2 6056 8 IPSLCM5A2 config #-C- IPSLCM5A2.2 trunk/libIGCM 1534 10 libIGCM . #-C- IPSLCM5A2.2 branches/2015/nemo_v3_6_STABLE/NEMOGCM 6665 7 . modeles #-C- IPSLCM5A2.2 XIOS/branchs/xios-2.5 1903 12 XIOS modeles
2.0.1. How to use it
Here are the commands you need to know if you want to retrieve and compile the IPSL-CM5A2 model and if you want to setup and run a piControl experiment (pdControl is also available):
mkdir YOUR_DIRECTORY ; cd YOUR_DIRECTORY svn_ano # svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl cd modipsl/util ./model IPSLCM5A2.1 cd ../config/IPSLCM5A2 gmake # by default ORCA2xLMD9695-L39 cp EXPERIMENTS/IPSLCM/piControl/config.card . vi config.card # modify JobName (at least) : MYJOBNAME, restarts ../../libIGCM/ins_job # Check and complete job's header cd MYJOBNAME vi Job_MYJOBNAME # modify PeriodNb, adjust the time, headers ... llsubmit Job_MYJOBNAME # IDRIS ccc_msub Job_MYJOBNAME # TGCC
2.0.2. IPSLCM5A2.2 specifics
It retains all the standard functionalities of IPSLCM5A2.1. It's usage is transparent for "non-paleo" simulations.
It restores the possibility to compile the create_etatlimite executable that was broken in preceding versions.
Some new functionalities have been added for "paleo" runs:
- you can compile a paleo version of the model including sources suitable for paleo simulations that are aware of paleorca grids. Use gmake PALEOIPSLCM5A2-VLR.
- the model now accepts configurations with no land ice (a bug has been corrected in Orchidee in rev 7376).
- you can use MOSAIX interpolation weights instead of MOSAIC if you wish. You need to set cpl_old_calving = n in run.def if you use MOSAIX. Also you need to recompute the initial state of your atmosphere with create etat limit. Do not use a restart from MOSAIC with LMDZ.
Note that
- some problem has been found by Delphine Tardif regarding plant phenologies in dynamical vegetation within orchidee. You need to replace 0% by 1% in your PFT map file and deactivate the restart for species to grow properly.
- you can't use the paleo version alongside the standard version of the model within the same installation directory (modipsl/config/IPSLCM5A2).
2.0.3. Computing performances
The basic configuration (default configuration) ran smoothly on Irene TGCC (and very soon on JeanZay IDRIS) machine and uses 302 computing cores : 1 process for XIOS, 45 processes for NEMO, and 32 MPI processes and 8 OpenMP threads for LMDZ. This configuration allows you to run 80 simulated years per day on Irene TGCC.
2.0.3.1. How to run more efficiently (using less resources)
In order to use less computing resources, it is possible to run on 160 cores (56 simulated years per day) as follows : 1 process for XIOS, 31 processes for NEMO, and 32 MPI processes and 4 OpenMP thread for LMDZ. To do that, you have to :
- Modify config.card as follows :
#======================================================================== #D-- Executable - [Executable] #D- For each component, Real name of executable, Name of executable for oasis ATM= (gcm.e, lmdz.x, 32MPI, 4OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) OCE= (opa, opa.xx, 31MPI) ICE= ("" ,"" ) MBG= ("" ,"" ) CPL= ("", "" ) IOS= (xios_server.exe, xios.x, 1MPI)
- create the new job :
../../libIGCM/ins_job
- Modify PARAM/namelist_ORCA2_cfg
jpni = 1 ! jpni number of processors following i (set automatically if < 1) jpnj = 31 ! jpnj number of processors following j (set automatically if < 1) jpnij = 31 ! jpnij number of local domains (set automatically if < 1)
- submit the job : ccc_msub (or sbatch) Job_MYJOBNAME
2.0.3.2. How to run faster (using more resources)
In order to run faster (using more computing resources), it is possible to run on 437 cores (95 simulated years per day) as follows : 1 process for XIOS, 60 processes for NEMO, and 47 MPI processes and 8 OpenMP thread for LMDZ. To do that, you have to :
- Modify config.card as follows :
#======================================================================== #D-- Executable - [Executable] #D- For each component, Real name of executable, Name of executable for oasis ATM= (gcm.e, lmdz.x, 47MPI, 8OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) OCE= (opa, opa.xx, 60MPI) ICE= ("" ,"" ) MBG= ("" ,"" ) CPL= ("", "" ) IOS= (xios_server.exe, xios.x, 1MPI)
- create the new job :
../../libIGCM/ins_job
- Modify PARAM/namelist_ORCA2_cfg
jpni = 4 ! jpni number of processors following i (set automatically if < 1) jpnj = 15 ! jpnj number of processors following j (set automatically if < 1) jpnij = 60 ! jpnij number of local domains (set automatically if < 1)
- submit the job : ccc_msub (or sbatch) Job_MYJOBNAME
2.0.3.3. Load balancing
Because of load-balancing (difference between ocean computing time and atmosphere computing time), not all of the configurations (in terms of number of process/threads) are efficient. If you want to run a configuration with fewer cores, ask Arnaud Caubel what would be the optimum configuration.
2.0.4. Restart files
Default IPSL-CM5A2-VLR configuration starts from CM5A2.1.pi.debug simulation at the date of 5009-12-31. The monitoring of this simulation is available here : https://vesg.ipsl.upmc.fr/thredds/fileServer/work_thredds/p25sepul/IPSLCM5A2/PROD/piControl/CM5A2.1.pi.debug/MONITORING/index.html
2.0.5. Output level
By default, low output levels are activated : monthly and daily outputs for atmospheric component, monthly and/or yearly outputs for other components. Remind that you have to think of the output you need before launching a long simulation.
2.0.6. Lengths, frequencies
2.0.6.1. Period length
Default period length is 1Y, i.e in config.card :
PeriodLength=1Y
Note that clean_PeriodLength.job will remove last period files, i.e last simulated year files.
2.0.6.2. Pack Frequency
Default pack frequency is 10Y, i.e in config.card :
PackFrequency=10Y
Note that since clean_latestPackperiod.job works on the latest pack period, clean_latestPackperiod.job will remove files from latest 10Y pack period. clean_latestPackperiod.job can also be used several time in a row to delete several 10Y pack periods. In order to optimize the use of inodes (beware of quotas !), we advice you to use 100Y as PackFrequency for very long simulations.
2.0.6.3. Rebuild frequency
Since we run with XIOS (server mode) as output library, the rebuild step is not needed anymore.
2.0.7. Computing centres
This configuration runs on TGCC-Irene-SKL and at IDRIS-JeanZay.
2.0.7.1. IDRIS-JeanZay
Because of JeanZay architecture (computing nodes based on sockets of 20 cores), we advice you to use 2, 4, 5, 10 or 20 OpenMP threads for LMDZ-ORCHIDEE component. By default, the number of OpenMP threads defined in config.card is 8. To change that, you have to :
- edit and modify config.card as follows :
#======================================================================== #D-- Executable - [Executable] #D- For each component, Real name of executable, Name of executable for oasis ATM= (gcm.e, lmdz.x, 35MPI, 10OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) OCE= (opa, opa.xx, 48MPI) ICE= ("" ,"" ) MBG= ("" ,"" ) CPL= ("", "" ) IOS= (xios_server.exe, xios.x, 1MPI)
- create the new job :
../../libIGCM/ins_job
- Modify PARAM/namelist_ORCA2_cfg
jpni = 4 ! jpni number of processors following i (set automatically if < 1) jpnj = 12 ! jpnj number of processors following j (set automatically if < 1) jpnij = 48 ! jpnij number of local domains (set automatically if < 1)
- submit the job : sbatch Job_MYJOBNAME
The above configuration allows you to run 93 SYPD.
3. Tuning and evaluation of the model
Tuning and evaluation of the model are described in a paper in open-discussion at Geoscientific Model Development :
https://www.geosci-model-dev-discuss.net/gmd-2019-332/
Most of the scripts used to evaluate the model can be retrieved and run interactively here :
https://zenodo.org/record/3549652
4. Modifications since frozen release
4.1. January, 24th 2018
File bathy.orca.nc changed into bathy.orca_v1.nc (see https://forge.ipsl.jussieu.fr/igcmg/changeset/3545/CONFIG/UNIFORM/v6/IPSLCM5A2). No change in results except for MBG component. The corrupted file (it was not defined on the right grid) was here : $(ccc_home --cccwork -u igcmg)/IGCM/OCE/IPSLCM6/ORCA2.3/bathy.orca_orca1l46.nc.
Attachments (6)
- IPSLCM6.png (108.2 KB) - added by aclsce 8 years ago.
-
CRE_3sim+data.png
(86.9 KB) -
added by pir 8 years ago.
CRF3sim
-
Evo_t2m_preind02.png
(89.6 KB) -
added by pir 8 years ago.
tun02
- Tuning_CM5A2.pdf (855.4 KB) - added by aclsce 8 years ago.
-
Tuning_CM5A2.2.pdf
(892.4 KB) -
added by pir 7 years ago.
CM5A2.1_note_25_01_2017
-
Note_CM5A2.1_27_01_17.pdf
(892.4 KB) -
added by pir 7 years ago.
CM5A2.1_note_25_01_2017