{{{
#!html
<h1>IPSL-CM5A2-VLR configuration </h1>
}}}
----
[[PageOutline(1-3,Table of contents,,numbered)]]

Persons in charge: Arnaud Caubel, Pierre Sepulchre
----

# IPSL-CM5A2 model #

'''IPSL-CM5A2''' is an ''' IPSL coupled climate model''' including atmosphere, land, ocean, sea ice and carbon cycle. The goal of this model is to reproduce IPSLCM5A-LR version, with reduced computing times to allow to run long simulations (i.e thousands of years) within reasonable time. Main characteritics of this model are '''ORCA2-LIM2-PISCES x LMD 96x95x39, old LMDZ physics, ORCHIDEE with Choisnel hydrology (2 layers)'''. This model uses both recent component versions and recent technical developments (hybrid parallelization MPI-OpenMP, XIOS 2.0 as input-output library, OASIS3-MCT parallel coupler) that allow to reach '''56 simulated years per day''' on about 300 computing cores. This model is available on Ada (IDRIS) and Curie (TGCC) supercomputers and benefits from last developments of libIGCM running environment.

'''IPSL-CM5A2''' model includes :
 * model components :
  * '''LMDZ''' as atmospheric model ; 
  * '''NEMO''' as ocean model including sea ice (LIM2) and marine biogeochemistry (PISCES) ; 
  * '''ORCHIDEE''' as land model ;  
 * tools :
  * '''OASIS3-MCT''' as parallel coupler ;
  * '''XIOS 2.0''' as I/O library ; 
  * '''libIGCM''' as running environment (scripts) to run the model and to perform post processing ;

[[Image(IPSLCM6.png, 500px)]]

# Technical details #

'''IPSL-CM5A2 model''' is available at following resolutions/configurations :

   * '''IPSL-CM5A2-VLR''' : LMDZ (Old Physics) 96x95x39-ORCHIDEE (Choisnel) - NEMO-LIM2-PISCES ORCA2

The resolution of LMDZ is 96x95 (3.75° in longitude and 1.875° in latitude) with 39 vertical levels. The ocean configuration is ORCA2 : global ocean with a tripolar grid with one South Pole, one North Pole above Siberia and one North Pole above northern America. The resolution is 2°. In the tropical region, the latitudinal resolution decreases to 1/2°. There are 31 vertical levels. 

'''IPSL-CM5A2''' is composed of following components and tools and available within '''IPSLCM5A2.1''' release :
{{{
#-H- IPSLCM5A2.1  IPSLCM5A2.1 coupled configuration
#-H- IPSLCM5A2.1  2nd frozen IPSLCM5A2 version
#-H- IPSLCM5A2.1  NEMOGCM branch nemo_v3_6_STABLE revision 6665
#-H- IPSLCM5A2.1  XIOS2 trunk revision 965
#-H- IPSLCM5A2.1  IOIPSL/src svn tags/v2_2_2
#-H- IPSLCM5A2.1  LMDZ5 trunk rev 2621
#-H- IPSLCM5A2.1  ORCHIDEE version trunk rev 3930
#-H- IPSLCM5A2.1  OASIS3-MCT 2.0_branch rev 1818
#-H- IPSLCM5A2.1  IPSLCM5A2 svn
#-H- IPSLCM5A2.1  libIGCM trunk rev 1373
#-M- IPSLCM5A2.1  arnaud.caubel@lsce.ipsl.fr
#-C- IPSLCM5A2.1  IOIPSL/tags/v2_2_2/src                    HEAD         8  IOIPSL/src     modeles
#-C- IPSLCM5A2.1  trunk/ORCHIDEE                            3930        14  ORCHIDEE       modeles
#-C- IPSLCM5A2.1  branches/OASIS3-MCT_2.0_branch/oasis3-mct 1818        15  oasis3-mct     .
#-C- IPSLCM5A2.1  LMDZ5/trunk                               2621        11  LMDZ           modeles
#-C- IPSLCM5A2.1  CONFIG/UNIFORM/v6/IPSLCM5A2               HEAD         8  IPSLCM5A2        config
#-C- IPSLCM5A2.1  trunk/libIGCM                             1373        10  libIGCM        .
#-C- IPSLCM5A2.1  branches/2015/nemo_v3_6_STABLE/NEMOGCM    6665         7  .              modeles
#-C- IPSLCM5A2.1  XIOS/trunk                                 965        12  XIOS           modeles
}}}

Note that previous IPSLCM5A2.0 release (with bug in runoff conservation) is still available.

### How to use it ###
Here are the commands you need to know if you want to retrieve and compile the IPSL-CM5A2 model and if you want to setup and run a piControl experiment (pdControl is also available):
{{{
#!sh
mkdir YOUR_DIRECTORY ; cd YOUR_DIRECTORY
svn_ano # svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
cd modipsl/util
./model IPSLCM5A2.1
cd ../config/IPSLCM5A2
gmake # by default ORCA2xLMD9695-L39
cp EXPERIMENTS/IPSLCM/piControl/config.card .
vi config.card # modify JobName (at least) : MYJOBNAME, restarts
../../libIGCM/ins_job # Check and complete job's header
cd MYJOBNAME
vi Job_MYJOBNAME # modify PeriodNb, adjust the time, headers ...
llsubmit Job_MYJOBNAME # IDRIS
ccc_msub  Job_MYJOBNAME # TGCC
}}}

### Computing performances ###

The basic configuration (default configuration) ran smoothly on Curie TGCC and Ada IDRIS machine and uses '''302 computing cores''' : 1 process for XIOS, 45 processes for NEMO, and 32 MPI processes and 8 OpenMP thread for LMDZ. This configuration allows you to run '''56 simulated years per day'''. 
Test on Irène TGCC and Jean-Zay IDRIS still to be done.

#### How to run more efficiently (using less resources) ####

In order to use less computing resources, it is possible to run on 160 cores (38 simulated years per day) as follows : 1 process for XIOS, 31 processes for NEMO, and 32 MPI processes and 4 OpenMP thread for LMDZ. To do that, you have to :
 * Modify config.card as follows :
{{{
#========================================================================
#D-- Executable -
[Executable]
#D- For each component, Real name of executable, Name of executable for oasis
ATM= (gcm.e, lmdz.x, 32MPI, 4OMP)
SRF= ("" ,"" )
SBG= ("" ,"" )
OCE= (opa, opa.xx, 31MPI)
ICE= ("" ,"" )
MBG= ("" ,"" )
CPL= ("", "" )
IOS= (xios_server.exe, xios.x, 1MPI)
}}}

 * create the new job : 
{{{
../../libIGCM/ins_job
}}}
 * Modify PARAM/namelist_ORCA2_cfg
{{{
   jpni        =   1       !  jpni   number of processors following i (set automatically if < 1)
   jpnj        =  31       !  jpnj   number of processors following j (set automatically if < 1)
   jpnij       =  31       !  jpnij  number of local domains (set automatically if < 1)
}}} 
 * submit the job : ccc_msub (or llsubmit) Job_MYJOBNAME 

#### How to run faster (using more resources) ####

In order to run faster (using more computing resources), it is possible to run on 437 cores (66 simulated years per day) as follows : 1 process for XIOS, 60 processes for NEMO, and 47 MPI processes and 8 OpenMP thread for LMDZ. To do that, you have to :
 * Modify `config.card` as follows :
{{{
#========================================================================
#D-- Executable -
[Executable]
#D- For each component, Real name of executable, Name of executable for oasis
ATM= (gcm.e, lmdz.x, 47MPI, 8OMP)
SRF= ("" ,"" )
SBG= ("" ,"" )
OCE= (opa, opa.xx, 60MPI)
ICE= ("" ,"" )
MBG= ("" ,"" )
CPL= ("", "" )
IOS= (xios_server.exe, xios.x, 1MPI)
}}}

 * create the new job : 
{{{
../../libIGCM/ins_job
}}}
 * Modify `PARAM/namelist_ORCA2_cfg`
{{{
   jpni        =   4       !  jpni   number of processors following i (set automatically if < 1)
   jpnj        =  15       !  jpnj   number of processors following j (set automatically if < 1)
   jpnij       =  60       !  jpnij  number of local domains (set automatically if < 1)
}}} 
 * submit the job : ccc_msub (or llsubmit) Job_MYJOBNAME

#### Load balancing ####

Because of load-balancing (difference between ocean computing time and atmosphere computing time), not all of the configurations (in terms of number of process/threads) are efficient. 
If you want to run a configuration with fewer cores, ask Arnaud Caubel what would be the optimum configuration. 

### Restart files ###
Default IPSL-CM5A2-VLR configuration starts from '''CM5A2.1.pi.debug''' simulation at the date of '''5009-12-31'''. The monitoring of this simulation is available here :  http://esgf.extra.cea.fr/thredds/fileServer/work/p25sepul/IPSLCM5A2/PROD/piControl/CM5A2.1.pi.debug/MONITORING/index.html

### Output level ###
By default, '''low output levels''' are activated : monthly and daily outputs for atmospheric component, monthly and/or yearly outputs for other components. Remind that you have to think of the output you need before launching a long simulation.

### Lengths, frequencies  ###
#### Period length ####
Default period length is 1Y, i.e in config.card :
{{{
PeriodLength=1Y
}}}
Note that clean_PeriodLength.job will remove last period files, i.e last simulated year files.

#### Pack Frequency ####
Default pack frequency is 10Y, i.e in config.card :
{{{
PackFrequency=10Y
}}}
Note that since clean_latestPackperiod.job works on the latest pack period, clean_latestPackperiod.job will remove files from latest 10Y pack period. clean_latestPackperiod.job can also be used several time in a row to delete several 10Y pack
periods.

#### Rebuild frequency ####
Since we run with XIOS (server mode) as output library, '''the rebuild step is not needed anymore'''.

### Computing centres ###

#### IDRIS Ada features ####

The basic configuration (default configuration) runs on Ada (IDRIS). In order to use efficiently computing resources, you have to use Intel environment. To do that you have to :
 * ask IDRIS assistance (mail to assist@idris.fr) to be authorized to use Ada full node and (# @ node = N) option. 
 * specify the environment you want to use when you do the `../../libIGCM/ins_job` command as follows :
{{{
Hit Enter or give MPI Environment (default is IBM), possible MPI environments are IBM (MPI IBM) and Intel (MPI Intel) :
Intel
MPIEnvironment is Intel
}}}

# Tuning and evaluation of the model #

'''A note''' is being written on the tuning strategy of IPSL-CM5A2. The following pdf document is updated very frequently:
[raw-attachment:Note_CM5A2.1_27_01_17.pdf​ IPSL-CM5A2 : Set-up, tuning strategy and evaluation]

'''CliMAF atlases''' comparing IPSL-CM5A2 and IPSL-CM5A pre-industrial climatologies will soon be available here :
[http://esgf.extra.cea.fr/thredds/fileServer/work/p25sepul/C-ESM-EP_CM5A2.1_vs_CM5A_vs_data.html
][[BR]]

Total run length after tuning:  '''2500 years'''.[[BR]]

Total run length from climatologies : '''3100 years'''.

# Modifications since frozen release #

## January, 24th 2018 ##

File bathy.orca.nc changed into bathy.orca_v1.nc (see https://forge.ipsl.jussieu.fr/igcmg/changeset/3545/CONFIG/UNIFORM/v6/IPSLCM5A2). No change in results except for MBG component. The corrupted file (it was not defined on the right grid) was here : `$(ccc_home --cccwork -u igcmg)/IGCM/OCE/IPSLCM6/ORCA2.3/bathy.orca_orca1l46.nc`.