Guides

Migration to Irene (from Curie)

Environment

Irene requires new environmental modules. For that purpose, make sure each time you connnect to the machine all the modules below are loaded.

To avoid any potential conflict with previous versions, run "module purge".

• module purge
• module load intel/17.0.4.196
• module load mpi/openmpi/2.0.2
• module load flavor/hdf5/parallel
• module load netcdf-fortran/4.4.4
• module load mkl
• module load hdf5/1.8.20
• module load feature/bridge/heterogenous_mpmd
• module load nco
• module load cdo
• module load ncview

Compilation

This guide is meant for those people who wants to keep using its current modipsl and/or Orchidee folder.

All those steps are already done if you decide to use the latest version of modipsl and Orchidee. So no step is required.

So,

Follow the steps below to properly migrate your current Orchidee version from Curie HPC to Irene. In big lines what you are going to do is:

1. Update modipsl: new libIGCM version (2.8.4)
2. Update Orchidee: include the arch/X64_IRENE.* files

Before start:

• Backup the folder you want to migrate

Let's start:

1. Update modipsl
   • It is required to upgrade your libIGCM version to 2.8.4: ​Source

cd modipsl
svn up
mv libIGCM libIGCM_old
svn checkout http://forge.ipsl.jussieu.fr/libigcm/svn/tags/libIGCM_v2.8.4 libIGCM
libIGCM/ins_job

2. Update Orchidee

• Copy LATEST_MICT/arch/*X64_IRENE* files to your current arch/ folder. ​Orchidee-MICT commit

3. Compile

• Reinstall the Makefiles for the new architecture:
  • cd yourmodipsl/util
  • ./ins_make

• Compile Orchidee using IRENE configuration
  • Go to yourmodipsl/config/ORCHIDEE_OL
  • Run: gmake clean && gmake

Job files

Make sure your Job header files include all neccessary fields. Below you can those specific options that must be change/add:

• #MSUB -q skylake # Select Architecture - required (modify)
• #MSUB -m store,work,scratch # Enable Disks - required (new)

Example:

#!/bin/bash
#MSUB -r mict_48 # Request name
#MSUB -n 24 # Number of tasks to use 
#MSUB -T 5000 # Elapsed time limit in seconds
#MSUB -o orchid_%I.o # Standard output. %I is the job id
#MSUB -e orchid_%I.e # Error output. %I is the job id
#MSUB -Q normal      # priority 
#MSUB -A gen6328     # Project ID 
#MSUB -X             # Enable GUI 
#MSUB -q skylake     # Select Architecture
#MSUB -m store,work,scratch # Enable Disks

More info

You will find more info typing in the Irene command line:

irene.info

Spinup Carbon MICT simulations (libIGCM configuration)

This guide is written by Matthieu Guimberteau:

​https://forge.ipsl.jussieu.fr/orchidee/attachment/wiki/DevelopmentActivities/ORCHIDEE-MICT-IMBALANCE-P/Guides/20180409_guimberteau_spinup_MICTv8.6.pdf

Spinup run in MICT-V6 (r3266)

This guide is written by Ana Bastos. It works for revision 3266.

In the folder /modipsl/config/ORCHIDEE_OL:

cp -r SPINUP MY_SPINUP_DIR

In MY_SPINUP_DIR you should find the following files/folders:

1. config.card
2. COMP
3. PARAM
4. SUBJOB

1. CONFIG.CARD

In config.card change:

JobName=My.Spinup.experiment
PeriodLength= 1Y
JobNumProcTot=NPROC # number of processors for simulation
ARCHIVE = /some/path/to/

Make sure to verify: SpaceName?

The spin-up (as far as I understand) does not work if it does not restart from a transient simulation. So first one needs to have restart files from the transient.

#D-- SRF options passed to SuRFace (SECHIBA) subjobs
[SRF]
WriteFrequency="1Y"
Restart=y
#-- Last day of the experience used as restart	
RestartDate=2011-12-31
# Define restart simulation name
RestartJobName=transient.my.experiment
RestartPath=${ARCHIVE}/IGCM_OUT/OL2/...[DEVT/PROD]/secsto
# Old component name for restart (if empty, use new name)
OldName=

#D-- Surface BioGeochemistry ( STOMATE ) subjobs
[SBG]
WriteFrequency= 1Y
Restart= y
#-- Last day of the experience used as restart
RestartDate= 2011-12-31
# Define restart simulation name
RestartJobName= transient.my.experiment
RestartPath=${ARCHIVE}/IGCM_OUT/OL2/...[DEVT/PROD]/secsto

Run the command:

../../../libIGCM/ins_job

Verify number of processors in Job_My.Spinup.experiment. It does not seem to update automatically.

• In CURIE:

  #MSUB -n 16  # Number of MPI tasks (SPMD case) or cores (MPMD case)
  

• In OBELIX:

  #PBS -v BATCH_NUM_PROC_TOT=16 # example
  #PBS -l nodes=4:ppn=4 # example
  

Add, if desired, specify a run dir path folder:

RUN_DIR_PATH = /to/some/folder/

Check, if desired/necessary:

PeriodNb = SOMEVALUE

2. COMP/

2.1. In spinup.card change/verify:

# If you want to use the same forcing file
DRIVER_NORESTART=y
## set to no because of in config.card periodlength=1yr
# If you want use config.card PeriodLength for TIME_LENGTH  
DRIVER_TIMELENGTH=n
## NO for MICT!
# Should we start the analytic resolution of the spinup? 
SPINUP_ANALYTIC=n
#(number of years to put C in soil - default is 100)
duree_carbonsol=X 

n_iter=X

# Boundary Files for ORCHIDEE subjobs :
List= (/.../.../.../.../my.forcing.file.nc, forcing_file.nc)

Verifiy other flags, eg:

• impose_veg
• land_use
• new_hydrol
• ...

2.2 - In spinup.driver

Below line 320:

# Nb years for forcing FORCESOIL
ORCHIDEE_def FORCESOIL_STEP_PER_YEAR 12
ORCHIDEE_def WRITELEVEL_FORCESOIL 3

Modify line 840:

IGCM_card_WriteOption ${New_SUBMIT_DIR}/COMP/stomate.card BoundaryFiles ListNonDel \
           "(${config_SBG_RestartPath}/${LastJobName}/SBG/Restart/${LastJobName}_${output_PreviousStage_LastRestartDate}_stomate_Cforcing.nc, stomate_Cforcing.nc), \\"

to:

IGCM_card_WriteOption ${New_SUBMIT_DIR}/COMP/stomate.card BoundaryFiles ListNonDel \
           "(${config_SBG_RestartPath}/${LastJobName}/SBG/Restart/${LastJobName}_${output_PreviousStage_LastRestartDate}_stomate_Cforcing_permafrost.nc, stomate_Cforcing_permafrost.nc), \\"

Add below:

# Nb years for forcing FORCESOIL
ORCHIDEE_def FORCESOIL_STEP_PER_YEAR 12
ORCHIDEE_def TIME_LENGTH X 			### equal to duree carbon sols #### 

3 - PARAM/

Use the same run.def as used for the whole experiment (see MICT run.def example)

4 - SUBJOB/

4.1 SUBJOB/FORCESOIL

4.1.1 - in config.card:

PeriodLength=X # (same as duree_carbonsol in spinup.card)
		
ARCHIVE = /.../.../.../ - archive folder
		
SBG= (forcesoil, forcesoil, NPROCMPI) # verify that number of processors matches between Spinup and subjobs
		
#D-- SRF - STOMATE
[SBG]
WriteFrequency="XY" # same as PeriodLength

4.1.2 - in Job_AUTO

Verify number of processors (as for the main job)

4.1.3 - in COMP/

4.1.3.1 - stomate.card update boundary file for Cforcing_permafrost

[BoundaryFiles]
List=   ()
ListNonDel= (${config_SBG_RestartPath}/${config_SBG_RestartJobName}/SBG/Restart/${config_SBG_RestartJobName}_${Date_Restarts}_stomate_Cforcing_permafrost.nc, stomate_Cforcing_permafrost.nc)\
            (${R_BC}/SRF/${config_UserChoices_TagName}/reftemp.nc, .)

4.1.3.2 - stomate.driver

ORCHIDEE_def STOMATE_RESTART_FILEIN stomate_rest_in.nc
ORCHIDEE_def STOMATE_CFORCING_NAME stomate_Cforcing.nc
ORCHIDEE_def STOMATE_CFORCING_PF_NM stomate_Cforcing_permafrost.nc # ADD THIS LINE

4.1.2 - in PARAM:

Add TIME_LENGTH=XY to run.def #(is empty)

4.2 SUBJOB/OOL_SEC_STO

4.2.1 - in SUBJOB/OOL_SEC_STO/config.card

#D- For each component, Real name of executable, Name of executable for oasis
OOL= (orchidee_ol, orchidee_ol, NPROCMPI) ##!!! Verify the it matches the main Job
	
#D-- SRF - SECHIBA
[SRF]
WriteFrequency="1Y"
Restart=n
		
#D-- SRF - STOMATE
[SBG]
WriteFrequency="1Y"
Restart=n

4.2.2 - in SUBJOB/OOL_SEC_STO/Job_AUTO

Verify number of processors (as for the main job)

4.2.3 - in SUBJOB/OOL_SEC_STO/COMP/

4.2.3.1 - orchidee_ol.card

[BoundaryFiles]
List= (/.../.../.../.../my.forcing.file.nc, forcing_file.nc)

4.2.3.2 - sechiba.card

Verify flags and add vegetation map (example):

LAIMAP=n
IMPOSE_VEG=n
LAND_USE=y	
VEGET_UPDATE=0Y

OLD_VEGET=n
ROUTING=n
NEWHYDROL=y
OKCO2=y
sechiba_LEVEL=11

[InitialStateFiles]
List=(${R_INIT}/SRF/${config_UserChoices_TagName}/routing.nc, .),     \
     (${R_INIT}/SRF/${config_UserChoices_TagName}/soils_param.nc, .), \
     (${R_INIT}/SRF/${config_UserChoices_TagName}/cartepente2d_15min.nc, .), \
     (${R_INIT}/SRF/${config_UserChoices_TagName}/floodplains.nc, .), \
     (/.../.../.../.../my.PFT.file.nc, PFTmap.nc)

4.2.3.3 - in stomate.driver

ORCHIDEE_def STOMATE_FORCING_NAME stomate_forcing.nc
ORCHIDEE_def STOMATE_CFORCING_NAME stomate_Cforcing.nc
ORCHIDEE_def STOMATE_CFORCING_PF_NM stomate_Cforcing_permafrost.nc  ## ADD THIS LINE
    	
    	
if [ $( IGCM_date_DaysBetweenGregorianDate ${PeriodDateEnd} ${DateEnd} ) -ge 0 ] ; then
        IGCM_sys_Put_Out stomate_Cforcing.nc ${R_OUT_SBG_R}/${config_UserChoices_JobName}_${PeriodDateEnd}_stomate_Cforcing.nc
           IGCM_sys_Put_Out stomate_forcing.nc  ${R_OUT_SBG_R}/${config_UserChoices_JobName}_${PeriodDateEnd}_stomate_forcing.nc
     	   IGCM_sys_Put_Out stomate_Cforcing_permafrost.nc ${R_OUT_SBG_R}/${config_UserChoices_JobName}_${PeriodDateEnd}_stomate_Cforcing_permafrost.nc ### ADD THIS LINE
   rm -f stomate_Cforcing.nc
   rm -f stomate_forcing.nc
   rm -f stomate_Cforcing_permafrost.nc      #### ADD THIS LINE
fi

4.2.4 - in SUBJOB/OOL_SEC_STO/PARAM/run.def

# Use same run.def as the experiment, but add:

FORCESOIL_STEP_PER_YEAR=12
STOMATE_CFORCING_PF_NM = stomate_Cforcing_permafrost.nc