Version 306 (modified by flavoni, 12 years ago) (diff)

Last edited Timestamp?

Using libIGCM library with ORCA2_LIM


  • Document the using of libIGCM for NEMO, in configuration ORCA2_LIM

Quick start guide:

  • Define the following alias to use SVN :
     alias svn_ano='svn co modipsl'
  • Create and go into working directory :
     mkdir TRY ; cd TRY
  • Extract modipsl :
  • Extract NEMO :
     cd modipsl/util 
     ./model NEMO_IGCM 
  • Choosing ORCA2_LIM configuration :
     cd ../modeles/UTIL
     ./fait_config ORCA2_LIM
  • Install the makefiles :
     cd ../../util 
  • Compile :
     cd ../config/ORCA2_LIM 
  • Submit Job :
     ( Editing at this point modipsl/config/ORCA2_LIM/IGCM00/config.card )
     change RESOL file in Job_Name:  RESOL=ORCA2
     cd IGCM00 
     qsub Job_NL1  

Set up a simulation:

  • First of all see official documentation

  • All parameters in config.card file

In modipsl/config/ORCA2_LIM/IGCM00/config.card file you can set all parameters to manage your experience:

How to manage restart files ( in the same simulation )

in config.card you have to set duration of your experience and setting PeriodLenght parameter, so you create restart files.

( for example : if you have and experience of 10 years and a PeriodLenght of 1 year, you will have 10 restart files )

#— Début et fin de Job
Date Begin?=2001-01-01
Date End?=2010-12-31
#— 1Y, 1M, 5D, 1D
Period Length?=1Y

Note : If you want to use restart data from another simulation ( already run not in this moment ) you have to manage "Restarts" part of config.card

#D— Restarts -

How to manage write output frequency

in config.card you have to set WriteFrequency of your experience

( for example : if you want output files every 1 month you have to put for Ocean

#D— OCE -
Write Frequency?="1M"

and if you want output files every 1 month you have to put for ICE

#D— ICE -
Write Frequency?="1M"


How to manage calendar type

in config.card you have to set CalendarType like 360d or leap or noleap

( for example : if you want 360 days you have to be careful of

Date End?=2010-12-30

because every month has 30 days

If you put for example Date End?=2005-12-31, you will have another run, of duration like your PeriodLenght )

How to manage rdt ( i.e. timestep )

in COMP/nemo.driver you have to change parameter : ORCA_NPDT_JOUR=15

( 15 corresponds to 86400s/5760s; it does mean that in one day you have 15 iterations; and it does mean that timestep is 5760)

How control all parameters in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist

libIGCM change namelist's values but not in namelist's file, so it is possible to control all changes of namelist's parameters looking at :


How to activate Monitoring

in config.card file

#D- If you want to produce time series, this flag determines
#D- frequency of post-processing submission
Time Series Frequency?=NONE

you have to put your experience lenght. If you do an experience during 10 years you have to put:

Time Series Frequency?=10Y

How to activate dynamic Monitoring (25/02/2009)

  • Copy original monitoring files in SUBMIT_DIR :
     cp ~/KEEPED_FILES/monitoring01_nemo_ORCA2.cfg $SUBMIT_DIR/.
     cp ~/KEEPED_FILES/monitoring01_lim2_ORCA2.cfg $SUBMIT_DIR/.
  • Copy monitoring.job in libIGCM :
     cp ~/KEEPED_FILES/monitoring.job modipsl/libIGCM/.
  • Control Time Series? in COMP/nemo.card and COMP/lim2.card:
     and if you add or remove TimeSeries in nemo.card or lim2.card => 
     it is necessary to add or remove correspondig variable in $SUBMIT_DIR/monitoring01_nemo[lim2]_ORCA2.cfg
  • How control monitoring on idris ( rhodes ) :
     Qstat -a

How to activate Atlas

in config.card file

#D- If you want to produce seasonal average, this flag determines
#D- the period of this average
Seasonal Frequency?=NONE

you have to put you experience lenght. If you do an experience during 10 years you have to put:

Seasonal Frequency?=10Y

How to manage frequency of post processing

( Note: in this moment is not possible to have atlas and monitoring of frequency different from 1M, if we want this we've to see it with Patrick Brockman!! )

in COMP/nemo.card you have to change in [Output Files?] chapter:

Post_1M_grid_T (U,V,W)

and you have to change [Post_1M_grid_T(U,V,W)] chapter.

and in in COMP/lim2.card you have to change in [Output Files?] chapter:

List= (${PREFIX_NWRITE}_${DATE_OPA}, ${R_OUT_ICE_NWRITE}/${PREFIX}_${WF1}, Post_1M_icemod)

and you have to change [Post_1M_icemod] chapter.

  • How to make a parallel run

first of all add a cpp key:

and then see:

  • How to have mail of end of Job, sur ccrt's machines

on mercure: in Job_Name: #PBS -M name@mail

on platine: In Job_Name: #BSUB -u name@mail

Run only post processing:

  • If you have already run a job and you need only to run post processing

you can do it in 3 ways:


in particular:


Example of post processing. a Job already run for 10 years form 2001 to 2010 :

in run.card

  • modify OldPrefix
    where (aa-1)mmdd is last day of last month of one year before you simulation's start
  • modify CumulPeriod

is 10 years that you've already run

  • modify PeriodState
  • modify TimeSeriesRunnging
    no because Time series are depending for each other, so if you want to start a time series it starts ONLY when a previous one is finished, so if you put TimeSeriesRunnging=y (i.e. yes) it can't never start
  • modify TimeSeriesCompleted
    nothing because we want to do time series for all 10 years ( If not you have to put last time series already realised )

in Job_Name:

  • modify cputim_job
    #PBS -l cputim_job=4:00:00  # Limite temps 
  • modify DRYRUN
  • modify RUN_DIR_PATH
     Create, if necessary, a test directory (for example in you workdir)
     Decomment RUN_DIR_PATH=/$workdir/TESTPOST         
  • modify SUBMIT_DIR
     Decomment and put right submit directory (the one in which you are changed you run.card and you Job) 

List of possible errors in modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName :

  • Error : TERM = (vt100) ${HOME:-.}/.profile: .bashrc: not found
  • Error : ksh: D—————————————————-==: not found

it can depends on .profile file.

To solve it
.profile file it can be like:

«# .profile (idris)
# Fichier d environnement a charger sur la frontale TX7 unix LINUX
# Vous pouvez mettre a jour le fichier .profile_tx
if [ `hostname!` = "brodie" -a -f $HOME/.profile_tx ]; then
. $HOME/.profile_tx
# Fichier d environnement a charger sur les noeuds SX8 unix SUPER-UX
# Vous pouvez mettre a jour le fichier .profile_sx
elif [ `hostname!` != "brodie" -a -f $HOME/.profile_sx ]; then
. $HOME/.profile_sx
fi »

  • Error : cat: cannot open ………/modipsl/config/ORCA2_LIM/IGCM00/../.resol :

it is due to modipsl/libIGCM/AA_job file.

To solve it
change in this way:

#— Resolution FLAG for oasis input
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)

NOTE: Be careful because this is ok for ORCA2, but if we use ORCA0.5 ore ORCA0.25 is NOT TRUE!!!!!

go in modipsl/config/ORCA2_LIM/IGCM00 directory

cancel run.card.init and Job_Name files

and then do an ins_job again.

(does not still fixed in libIGCM_v1_2)

  • Error : Input files and BC for NEMO DO NOT EXIST (i.e.,,,, etc…)

IGCM_sys_Get, ERROR : regular file …../OCE/ORCA2_LIM/ DOES NOT EXIST .

To solve it
In this moment we've still to download on!reee512/NEMO/ORCA2_LIM_nemo_v3.tar file, and to untar it into the directory:


and where ${R_BC} is a variable that can be changed in modipsl/config/ORCA2_LIM/IGCM00/config.card file.

  • Error : IGCM_debug_Exit : IGCM_config_Initialize Error PeriodState : Fatal
To solve it
change line in modipsl/config/ORCA2_LIM/IGCM00/run.card


or rm modipsl/config/ORCA2_LIM/IGCM00/run.card

  • ls: ERROR: Cannot access stack: No such file or directory

in Script_Outputxxx:

( is it a problem with filemane? )

IGCM_sys_Put_Out : out_nemo /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out/N365d_5d_20030101_20031231_out_nemo
IGCM_sys_MkdirArchive : /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out
IGCM_sys_Chmod : 444 out_nemo
—Debug1—> OCE
—Debug1—> ICE
Files that will be deleted before next period-run :
ls: ERROR: Cannot access stack: No such file or directory

(no ok still with libIGCM_v1_2)

To solve it
you have to activate corresponding keys, if not in Script_Output you will have:


List of possible remarks and suggestions :

  • be careful at group (for example for a gensi one)

if we use a common account on CCRT we have to be sure that everygroup can have access to it, this is important for chmod 444, do another test!!!!!!

  • Every time that we change modipsl/config/ORCA2_LIM/IGCM00/config.card

we have to cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init

cancel modipsl/config/ORCA2_LIM/IGCM00/Job_Name

and run ./modipsl/util/ns_job script

  • To perform a second simulation after a run with PeriodState=Fatal

cancel modipsl/config/ORCA2_LIM/IGCM00/stack_error, modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName

but do not cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init file, if not run.card file is not re-created.

( File run.card.init is created with ins_job script )

  • Does it exists a caption file for run.card.init options?

Or Is there a reference file?
In to run.card.init file who can we undesrtand syntax of? :



RebuildFrequency : how much has it? = PeriodLenght???

  • test tags/libIGCM_v1_2

Why defaul of config.card is Write Frequency?="5D" instead of 1M?

We've seen that with 1M monitoring and atlas are not possible.

  • test tags/libIGCM_v1_2

Is it still true?: To use libIGCM trunk version it is necessary to add in config.card in POST Session:


  • Change in nemo.card or in nemo.driver variable: ORCA_NPDT_JOUR and OPA_NPDT_JOURS

Done :

  • run on CCRT machines ( mercure ) and at IDRIS ( brodie )

Tests for Calendar Type, Output frequency and Restarts:

Simulation environment

Simulation duration = 10 years

PeriodLength = 1Y

timestep: rdt = 5760 equivalent to ORCA_NPDT_JOUR = 15

Output Frequency Output Frequency Output Frequency
5d 1M 1Y
Calendar Type 360d ok ok ok
Calendar Type 365d ok ko ok
Calendar Type leap ok ko to do

for noleap year we can do now ( mars 2009 ) PeriodLenght=1M and WriteFrequency=1M ko = there are only 11 output ( point how NEMO do averadge for every month )

NOTE: control in:

  • why in /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N360d_1m/OCE/Debug/*_ocean.output

Namelist namrun

job number no = 0

is EVER = 0????

Control if file name is correct!!!! (my job is N365d_1m)

Job Name versions calendar type
ONLY nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
option only(x) in diawri.F90
ONLYio4 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 360d
dynamic nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic
io7nemo3 nemo_v3, ioipsl_v2_1_7, libIGCM trunk 360d
v3360d nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2 360d
v31bok nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2 360d

  • monitoring : done on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

'ok': monitoring dynamic ( reading nemo.card ⇒ doing monitoring )

NOTE: control mesure unity for monitoring_nemo_ORCA2.cfg for soicecov ( [0,1]: is it ok? )

  • atlas : ok on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

NOTE: changed COMP/nemo.card and set Output Files? and Post_1M_grid_T[U,V,W]
NOTE: changed COMP/lim2.card and set Output Files? and Post_1M_grid_T[U,V,W]

To do list :

  • simulation=100Y ( ongoing with tags/libIGCM_v1_2 )
    and sortiee=1Y
  • test mpi: ongoing on mercure

point out: all parameters in config.card :
is it necessary to change

#Job Run Options?='"-np "${NUM_PROC}"'

Job Run Options?='"-np "${BATCH_NUM_PROC_TOT}"' ?

if yes commit it

NOTE files to change (or changed) for NEMO :

and modipsl/modeles/NEMO/WORK/domain.F90

  • modipsl/modeles/UTIL/fait_AA_make

difference with the old one:

in #- Compilation options —-

Committed: shade 10000 (for mercure)

  • modipsl/libIGCM/AA_job

difference with the old one:
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)

  • modipsl/modeles/NEMO/WORK/domain.F90

difference with the old one:

in line 171

WRITE(numout,*) ' leap year calendar (0/1/30) nleapy = ', nleapy

  • modipsl/config/ORCA2_LIM/IGCM00/COMP/nemo.driver

difference with the old one:

added nleapy variable that corresponds to nbisex on opa.driver

# nleapy leap year calendar (0/1) (30 for 360d)

PAT_NLEAPY=$( supergrep nleapy ${SUBMIT_DIR}/PARAM/namelist )

in lines 141-151

case ${config_UserChoices_CalendarType} in

leap | | Gregorian)


in line 158

-e "s%${PAT_NLEAPY}% nleapy=${ORCA_NLEAPY}%" \

Appendix A - Dods directory:

  • to remove a directory from dods :

On Mercure to CCRT DOds directory is :


cd ${SAVE_DIR}/dods
dods_rm public/${LOGIN}/${TAG_NAME}/directory

Simona Flavoni last update: Timestamp?

Attachments (1)

Download all attachments as: .zip