HOWTO install/compile/run ORCHIDEE-CN-P for offline use with libIGCM.

This howto is based on the information which can be found in the ORCHIDEE wiki regarding the trunk version of ORCHIDEE. Please check the wiki for information regarding the respective commands listed here.

---

---

1. install libIGCM & ORCHIDEE-CN-P

1.1 where to install

You should install the model NOT in the home directory; it is even too small to handle diagnostics like "out_execution" or "out_orchidee". You can use , for example, your work directory: curie:

/ccc/work/cont003/dsm/<userID>

or project storage; here example for the IMBALANCE-P project on obelix:

/home/surface3/dgoll

1.2 install libIGCM & trunk ORCHIDEE

svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
cd modipsl/util
./model ORCHIDEE_trunk

1.3 Exchange the trunk ORCHIDEE with ORCHIDEE-CN-P

cd ../modeles
rm -fr ORCHIDEE
svn co svn://forge.ipsl.jussieu.fr/orchidee/branches/ORCHIDEE-CN-P ORCHIDEE
cd ../util ; ./ins_make

---

---

2. Compile ORCHIDEE

2.1 modify IOIPSL

adjust IOIPSL to be able to write more variables into the output by increase the value of max_var by 50 in ../modeles/IOIPSL/src/restcom.f90

 &  max_var=550, max_file=50, max_dim=NF90_MAX_VAR_DIMS

2.2 compile orchidee

A: on obelix

cd ../modeles/ORCHIDEE  

First do a "normal" make to compile XIOS & IPOSL components which are needed; don't worry if the compilation of ORCHIDEE crashes

make

then compile (do compile always with "-parallel mpi" as this executable can be run in sequential mode, too. But not vice versa)

./makeorchidee_fcm -driver -arch ifort_LSCE -noxios -parallel mpi

B: on curie

WARNING: before you do anything on curie; make sure you have copied the p86ipsl login environment to your home according to ​https://forge.ipsl.jussieu.fr/igcmg_doc/wiki/DocBenvBtgcc:

cp ~p86ipsl/.bashrc_curie  ~/.
cp ~p86ipsl/.bash_profile ~/.

Then compile the standard way:

cd ../config/ORCHIDEE_OL
gmake without_xios

Optional: I usually re-compile ORCHIDEE again with makeorchidee_fcm to set options (don't forget to disable xios):

cd ../../modeles/ORCHIDEE
./makeorchidee_fcm -driver -parallel mpi -arch X64_CURIE -driver -noxios

---

---

3. Run ORCHIDEE

There are different setups possible with libIGCM. Please check the documentation of libIGCM for more information, here I just tell you what you need to do.

3.1 SPINUP_ANALYTIC

I did not yet resolved how to best spin up the biogeochemical cycles; more information here: ​https://forge.ipsl.jussieu.fr/orchidee/wiki/DevelopmentActivities/ORCHIDEE-CNP/TechnicalMods

3.1.1 create new experiment folder

cd ../../config/ORCHIDEE_OL/
cp -fr SPINUP_ANALYTIC/ MY_SPINUP_ANALYTIC/
cd MY_SPINUP_ANALYTIC/

3.1.2 tell libIGCM about the new input files (they are stored on the IMBALANCE-P disk on obelix)

modify the variable "ListNonDel?" in COMP/stomate.card:

ListNonDel= (${R_IN}/SRF/reftemp.nc, .) \	 
(/home/surface3/common/CNP_files/rev00/USDA_SoilSuborder.nc, .), \
(/home/surface3/common/CNP_files/rev00/lithology.nc, .), \
(/home/surface3/common/CNP_files/rev00/deposition.nc, .)

WARNING: the deposition file is a static field at the moment; it does not include N deposition either. At the moment N deposition is set to P deposition in the ORCHIDEE code. Anyway N deposition is not accounted for in the calculations for the N & P cycle for now.

tell that we use the new “soil_param.nc”; in COMP/sechiba.card:

(/home/surface3/common/CNP_files/rev00/soils_param.nc, .), \

3.1.3 adjust the libIGCM

disable XIOS in COMP/orchidee_ol.card

# Use XIOS as output library instead of IOIPSL
XIOS=n

disable XIOS in PARAM/run.def

# Use XIOS for writing diagnostics file
# defulat = n
XIOS_ORCHIDEE_OK = n

set the IO frequency to avoid crashes in PARAM/run.def

# Writefrequency in seconds in sechiba_history.nc
# default = 86400.0
WRITE_STEP = 1800.0

# Writefrequency in days in stomate_history.nc
# default = 10.
STOMATE_HIST_DT = 1

add the new flags for the nutrient cycles in PARAM/run.def:

# to activate P cycle set =y
STOMATE_OK_PCYCLE=y
# to activate N cycle set =y
STOMATE_OK_NCYCLE=y
# make sure we dont fake
LD_FAKE_HEIGHT=n
# you can enable mass conservation and stoichiometry checks by =y
STOMATE_MASS_CONSERVATION=y
STOMATE_DSG_DEBUG=y

in addition you should add all the PFT and soil parameter values as set in /home/users/dgoll/ORCHIDEE/DEFS_n_CARDS/run.def_NV2

disable river routing if you run on non-global scale in run.def

# Activate river routing
RIVER_ROUTING = n

3.1.4 adjust the job as usual

modify config.card as usual (see wiki)

modify run.def as usual (see wiki) plus if you want to archive, for example in the IMBALANCE-P storage; add to config.card:

ARCHIVE=/home/surface3/dgoll 

3.1.5 create job

../../../libIGCM/ins_job

3.1.6 adjust job_file (optional)

Header to run a site scale simulation on obelix

######################
	## OBELIX      LSCE ##
	######################
	#PBS -N test
	#PBS -m a
	#PBS -j oe
	#PBS -q short
	#PBS -o Script_Output_test.000001
	#PBS -S /bin/ksh
	#PBS -v BATCH_NUM_PROC_TOT=1
	#PBS -l nodes=1:ppn=1

to automatically have existing files deleted, set JobType?=DEV

 	#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
	JobType=DEV 

to avoid queuing take full advantage of your job time; the value depends on #PBS -q short and computation time of orchidee

#D- Number of execution in one job
	PeriodNb=60

to have a temp directory you can access on obelix; exchange dgoll with your ID

#D- Define running directory
#D- Default=${TMPDIR} ie temporary batch directory
#D-
#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
RUN_DIR_PATH=/home/scratch01/dgoll/RUN_DIR

3.1.7 submit the job

on obelix:

qsub Job_ID

on curie

ccc_msub Job_ID

--- ---

3.2 ENSEMBLE: fluxnet sites ​http://forge.ipsl.jussieu.fr/orchidee/wiki/Scripts/FluxnetValidation

Figure shows the sequence of simulations: the labels(TOP), the time length(MIDDLE) and the spinup_analytic(BOTTOM). Adopted from NV scheme.

3.2.1 create new experiment folder

cd ../../config/ORCHIDEE_OL/
cp -fr ENSEMBLE/ ENSEMBLE_CNP/
cd ENSEMBLE_CNP/

3.2.2 tell libIGCM about the new input files

For now, I store all files needed on the IMBALANCE-P disk on obelix (you find the path below). Please ensure you are using the newest revision (rev00 is not fully tested).

The configuration which are used are stored in ../SPINUP/SUBJOB/OOL_SEC_STO/COMP/ so, do as in 4.1.2: change the path of soil_param.nc and add the new files

sechiba.card:

(/home/surface3/common/CNP_files/rev00/soils_param.nc, .), \

stomate.card:

ListNonDel= (${R_IN}/SRF/reftemp.nc, .) \	 
(/home/surface3/common/CNP_files/rev00/USDA_SoilSuborder.nc, .), \
(/home/surface3/common/CNP_files/rev00/lithology.nc, .), \
(/home/surface3/common/CNP_files/rev00/deposition.nc, .)

WARNING: the deposition file is a static field at the moment; it does not include N deposition either. At the moment N deposition is set to P deposition in the ORCHIDEE code. Anyway N deposition is not accounted for in the calculations for the N & P cycle for now.

exchange the spinup.driver file in ../SPINUP/SUBJOB/COMP/ with the new modified driver from NV.

cp /home/surface3/common/CNP_files/rev00/spinup.driver ../SPINUP/SUBJOB/OOL_SEC_STO/COMP/

copy a run.def with all parameters needed for the ORCHIDEE-CAN parts of the model to PARAM, for example this one:

cp /home/surface3/common/CNP_files/rev00/run.def_NV2 PARAM/run.def

3.2.3 adjust the libIGCM

modify the configuration of the SPINUP. These variables were originally used for the outdated spinup method using FORCESOIL. Thus the variable names are misleading.

modify them in ensemble.card and fluxnet.card:

[SPINUP]
# Initialisation for spin-up :
# orchidee with sechiba alone (!!! if ok_stomate == n !!!)
duree_nostomate=0
#  normal SEC_STO simulation to fill biomass pools (equilibrate litter fluxes) of duree_inistomate years
duree_inistomate=1000
# teststomate (only if duree_inistomate > 0); DSG: don't touch this one
duree_offlineini=0

# Loop over ORCHIDEE runs (used for spin-up)
# The analytical spinup is restarted n_iter times ...
n_iter=3
# ... for duree_sechiba years
duree_sechiba=50
# teststomate; DSG: don't touch this one
duree_stomate=0
# forcesoil; DSG: don't touch this one
duree_carbonsol=1

# Final run after the analytical spinup
# This last parameter must be non-zero.
duree_final=100

In case you want to check if the spinup works well, increase output level of the spinup simulations & prevent libIGCM from removing these files. The files can be found in the respective site folders in the experiment folder.

modify in fluxnet.card and ensemble.card:

# level of the history files for each iteration of the SPINUP
level_hist=10
# level of the history files for the final iteration of the SPINUP
level_hist_final=10

# DEBUG mode for SPINUP 
#
# This mode keep all SPINUP directory in ARCHIVE
# If disable, all ARCHIVE is automaticly cleaned.
DEBUG_SPIN=y
# If you don't want to keep old spinup steps, but last one
CONSERVE=y

add the variables to turn on/off the nutrients cycles to PARAM/run.def:

# switch for N cycle
STOMATE_OK_NCYCLE=y
# switch for P cycle
STOMATE_OK_PCYCLE=y
# switch to either impose N content of vegetation (to surpass N limitation):
STOMATE_IMPOSE_CN=n
# switch to either impose N content of vegetation (to surpass P limitation):
STOMATE_IMPOSE_NP=n
# do not fake the initial height (DOFOCO parameter)
LD_FAKE_HEIGHT=n

Nammonium_FILE =NONE
Nnitrate_FILE =NONE
Nfert_FILE=NONE
Nbnf_FILE=NONE
NINPUT_UPDATE=-1

if you want to debug the model, set printlev to 4 for full debug information in PARAM/run.def:

PRINTLEV=4

you can add defaults for the nutrient switches into PARAM/orchidee.default

# STOMATE_OK_NCYLE ([FLAG]) :  Activate the nitrogen cycle {OK_STOMATE}
STOMATE_OK_NCYCLE = n

# STOMATE_OK_PCYLE ([FLAG]) :  Activate the phosphorus cycle {OK_STOMATE}
STOMATE_OK_PCYCLE = n

# STOMATE_MASS_CONSERVATION ([FLAG]) : activate mass conservation checks which force stop if violated {OK_STOMATE}
STOMATE_MASS_CONSERVATION = n

# STOMATE_DSG_DEBUG ([FLAG]) : activate checks for stoichiometry and negative pools {OK_STOMATE}
STOMATE_DSG_DEBUG = n

# LD_FAKE_HEIGHT ([FLAG]) : fakes height of vegetation (ORCHIDEE-CAN related) {OK_STOMATE}
LD_FAKE_HEIGHT = n

REMARKS: Do not enable XIOS, it is per default disabled in "PARAM/orchidee.default". Do not enable river routing in site scale simulation, is per default disabled in "fluxnet.card"

3.2.4 adjust the job as usual to your specific needs

1. modify config.card following ​http://forge.ipsl.jussieu.fr/orchidee/wiki/Scripts/FluxnetValidation

give information on Job Name, Tag Name, Dates and Restart.

2. modify run.def following ​http://forge.ipsl.jussieu.fr/orchidee/wiki/Scripts/FluxnetValidation

if you want to archive, for example in the IMBALANCE-P storage; add to config.card:

ARCHIVE=/home/surface3/dgoll 

modify fluxnet.card following ​http://forge.ipsl.jussieu.fr/orchidee/wiki/Scripts/FluxnetValidation

• if you want to keep the output files of the spinup you need to set the following variables

  # DEBUG mode for SPINUP 
  #
  # This mode keep all SPINUP directory in ARCHIVE
  # If disable, all ARCHIVE is automaticly cleaned.
  DEBUG_SPIN=y
  # If you don't want to keep old spinup steps, but last one
  CONSERVE=y
  

3.2.5 create job

../../../libIGCM/ins_job

3.2.6 adjust job

on obelix change the queue to longp to increase the allocated time (the default queue mediump is too short and you will exceed wall clock time).

#PBS -q longp

If you want to debug on obelix: set the variable RUN_DIR_PATH to disk with enough space (the default path is not accessible for the user)

# for example on scratch01
RUN_DIR_PATH=/home/scratch01/dgoll/RUN_DIR

3.2.7 start simulations

my_prompt>> ksh
my_prompt>> nohup ./Job_ENSEMBLE fluxnet > out_Job 2>&1 &

3.2.8 check simulations

to see the decomposition of the spatial domain among the processors check in the run dir "Load_balance_orchidee.dat" on curie The RUN_DIR is - if not specified explicitly in the Job - in

/ccc/scratch/cont003/dsm/<userid>/RUN_DIR

to check the status of the job; type

ccc_mstat -u <userid>

to delete a job; type

ccc_mdel <jobid>

3.2.9 post processing

ensure the Jobname and the path to the reference simulation is set in fluxnet.card; in case there is no reference set it to NONE

# History file of former ORCHIDEE runs (Reference) to compare with the current simulations
reference_base_path=/home/scratch01/dgoll/IGCM_OUT/OL2/DEVT/ensemble
reference_config=DSG04EnsCN