Version 337 (modified by flavoni, 12 years ago) (diff)

Last edited Timestamp?

Using libIGCM library with ORCA2_LIM


Purposes:


  • Document the using of libIGCM for NEMO, in configuration ORCA2_LIM



Quick start guide:


  • Define the following alias to use SVN :
     alias svn_ano='svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl'
    
  • Create and go into working directory :
     mkdir TRY ; cd TRY
    
  • Extract modipsl :
     svn_ano 
    
  • Extract NEMO :
     cd modipsl/util 
     ./model NEMO_IGCM 
    
  • Modify AA_job to avoid bug for ORCA2_LIM ( mars 2009 ) :
     cd ../libIGCM 
     vi AA_job : change RESOL file in Job_Name:  RESOL=ORCA2
    
  • Choosing ORCA2_LIM configuration :
     cd ../modeles/UTIL
     ./fait_config ORCA2_LIM
    
  • Install the makefiles :
     cd ../../util 
     ./ins_make
    
  • Compile :
     cd ../config/ORCA2_LIM 
     gmake
    
  • Submit Job :
     ( Editing at this point modipsl/config/ORCA2_LIM/IGCM00/config.card )
    
     ../../util/ins_job
     cd IGCM00 
     qsub Job_Name
    



Set up a simulation:


  • First of all see official documentation

http://forge.ipsl.jussieu.fr/libigcm/wiki/

  • All parameters in config.card file

In modipsl/config/ORCA2_LIM/IGCM00/config.card file you can set all parameters to manage your experience:

How to manage restart files ( in the same simulation )

in config.card you have to set duration of your experience and setting PeriodLenght parameter, so you create restart files.

( for example : if you have and experience of 10 years and a PeriodLenght of 1 year, you will have 10 restart files )

#— Début et fin de Job
#— "YYYY-MM-DD"
Date Begin?=2001-01-01
Date End?=2010-12-31
#============================
#— 1Y, 1M, 5D, 1D
Period Length?=1Y

Note : If you want to use restart data from another simulation ( already run not in this moment ) you have to manage "Restarts" part of config.card

#================
#D— Restarts -
[Restarts]

How to manage write output frequency

in config.card you have to set WriteFrequency of your experience

( for example : if you want output files every 1 month you have to put for Ocean

#===================
#D— OCE -
[OCE]
Write Frequency?="1M"

and if you want output files every 1 month you have to put for ICE

#===================
#D— ICE -
[ICE]
Write Frequency?="1M"

)

How to manage calendar type

in config.card you have to set CalendarType like 360d or leap or noleap

# for 360d :

you have to be careful at

Date End?=2010-12-30

because every month has 30 days

If you put for example Date End?=2005-12-31, you will have another run, of duration like your PeriodLenght

# for noleap :

you have to be careful at

WriteFrequency="1M"

and PeriodLength=1M

because in this moment simulation of 365 days it is possible only with a write output frequency of one month, and restart every month.

( If you put a different value for WriteFrequency you will not find restart files )

How to manage rdt ( i.e. timestep )

in COMP/nemo.driver you have to change parameter : ORCA_NPDT_JOUR=15

( 15 corresponds to 86400s/5760s; it does mean that in one day you have 15 iterations; and it does mean that timestep is 5760)

How control all parameters in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist

libIGCM change namelist's values but not in namelist's file, so it is possible to control all changes of namelist's parameters looking at :

{OUTPUT_DIR}/IGCM_OUT/ORCA2_LIM/N360d_5d/OCE/Debug/!JobName_date_date_ocean.output

How to activate Monitoring

in config.card file

#D- If you want to produce time series, this flag determines
#D- frequency of post-processing submission
Time Series Frequency?=NONE

you have to put your experience lenght. If you do an experience during 10 years you have to put:

Time Series Frequency?=10Y

How to activate dynamic Monitoring (25/02/2009)

  • Copy original monitoring files in SUBMIT_DIR :
     cp ~/KEEPED_FILES/monitoring01_nemo_ORCA2.cfg $SUBMIT_DIR/.
     cp ~/KEEPED_FILES/monitoring01_lim2_ORCA2.cfg $SUBMIT_DIR/.
    
  • Copy monitoring.job in libIGCM :
     cp ~/KEEPED_FILES/monitoring.job modipsl/libIGCM/.
    
  • Control Time Series? in COMP/nemo.card and COMP/lim2.card:
     and if you add or remove TimeSeries in nemo.card or lim2.card => 
     it is necessary to add or remove correspondig variable in $SUBMIT_DIR/monitoring01_nemo[lim2]_ORCA2.cfg
    
  • How control monitoring on idris ( rhodes ) :
     Qstat -a
    

How to activate Atlas

in config.card file

#D- If you want to produce seasonal average, this flag determines
#D- the period of this average
Seasonal Frequency?=NONE

you have to put you experience lenght. If you do an experience during 10 years you have to put:

Seasonal Frequency?=10Y

How to manage frequency of post processing

( Note: in this moment is not possible to have atlas and monitoring of frequency different from 1M, if we want this we've to see it with Patrick Brockman!! )

in COMP/nemo.card you have to change in [Output Files?] chapter:

Post_1M_grid_T (U,V,W)

and you have to change [Post_1M_grid_T(U,V,W)] chapter.

and in in COMP/lim2.card you have to change in [Output Files?] chapter:

List= (${PREFIX_NWRITE}_${DATE_OPA}_icemod.nc, ${R_OUT_ICE_NWRITE}/${PREFIX}_${WF1}_icemod.nc, Post_1M_icemod)

and you have to change [Post_1M_icemod] chapter.

  • How to make a parallel run

first of all add a cpp key:

from Arnaud Caubel: "Il y a un pb MPI sur mercure avec la lib par defaut. J'avais mis sur le wiki un : Attention : sur mercure l'utilisation de la librairie MPI 7.2.4 est indispensable. Pour l'utiliser : export
SX_BASE_MPI=/SX/opt/mpisx/new avant de faire le gmake"

http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-add-modify-new-modules-or-new-cpp-keys

and then see: http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-set-up-one-simulation#eztoc1210_5

  • How to have mail of end of Job, on ccrt's machines

on mercure: in Job_Name: #PBS -M name@mail

on platine: In Job_Name: #BSUB -u name@mail



Run only post processing:


  • If you have already run a job and you need only to run post processing

you can do it in 3 ways:

see: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ

in particular: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelafrontale

and http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelamachinedecalcul


Example of post processing. a Job already run for 10 years form 2001 to 2010 :

in run.card

  • modify OldPrefix
     OldPrefix=''Job_!Name''_(aa-1)mmdd
    
    where (aa-1)mmdd is last day of last month of one year before you simulation's start
  • modify CumulPeriod
    !CumulPeriod=10
    

is 10 years that you've already run

  • modify PeriodState
     !PeriodState=!OnQueue
    
  • modify TimeSeriesRunnging
     !TimeSeriesRunnging=n
    
    no because Time series are depending for each other, so if you want to start a time series it starts ONLY when a previous one is finished, so if you put TimeSeriesRunnging=y (i.e. yes) it can't never start
  • modify TimeSeriesCompleted
     !TimeSeriesCompleted=
    
    nothing because we want to do time series for all 10 years ( If not you have to put last time series already realised )

in Job_Name:

  • modify cputim_job
    #PBS -l cputim_job=4:00:00  # Limite temps 
    
  • modify DRYRUN
     DRYRUN=3
    
  • modify RUN_DIR_PATH
     Create, if necessary, a test directory (for example in you workdir)
     Decomment RUN_DIR_PATH=/$workdir/TESTPOST         
    
  • modify SUBMIT_DIR
     Decomment and put right submit directory (the one in which you are changed you run.card and you Job) 
     SUBMIT_DIR=/$workdir/modipsl/config/$CONFIG_NAME/EXP00 
    



List of possible errors in modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName :


  • Error : TERM = (vt100) ${HOME:-.}/.profile: .bashrc: not found
  • Error : ksh: D—————————————————-==: not found

it can depends on .profile file.

To solve it
.profile file it can be like:

«# .profile (idris)
# Fichier d environnement a charger sur la frontale TX7 unix LINUX
# Vous pouvez mettre a jour le fichier .profile_tx
if [ `hostname!` = "brodie" -a -f $HOME/.profile_tx ]; then
. $HOME/.profile_tx
# Fichier d environnement a charger sur les noeuds SX8 unix SUPER-UX
# Vous pouvez mettre a jour le fichier .profile_sx
elif [ `hostname!` != "brodie" -a -f $HOME/.profile_sx ]; then
. $HOME/.profile_sx
else
echo
fi »

  • Error : cat: cannot open ………/modipsl/config/ORCA2_LIM/IGCM00/../.resol :

it is due to modipsl/libIGCM/AA_job file.

To solve it
change in this way:

#— Resolution FLAG for oasis input
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)
RESOL=ORCA2

NOTE: Be careful because this is ok for ORCA2, but if we use ORCA0.5 ore ORCA0.25 is NOT TRUE!!!!!

go in modipsl/config/ORCA2_LIM/IGCM00 directory

cancel run.card.init and Job_Name files

and then do an ins_job again.

(does not still fixed in libIGCM_v1_2)

  • Error : Input files and BC for NEMO DO NOT EXIST (i.e. sst_data.nc, sss_data.nc, geothermal_heating.nc, bathy_level.nc, etc…)

IGCM_sys_Get, ERROR : regular file …../OCE/ORCA2_LIM/bathy_level.nc DOES NOT EXIST .

To solve it
In this moment we've still to download on gaya.idris.fr:~/!reee512/NEMO/ORCA2_LIM_nemo_v3.tar file, and to untar it into the directory:

${R_BC}/OCE/ORCA2_LIM

and where ${R_BC} is a variable that can be changed in modipsl/config/ORCA2_LIM/IGCM00/config.card file.

  • Error : IGCM_debug_Exit : IGCM_config_Initialize Error PeriodState : Fatal
To solve it
change line in modipsl/config/ORCA2_LIM/IGCM00/run.card

PeriodState=Fatal
in
PeriodState=

or rm modipsl/config/ORCA2_LIM/IGCM00/run.card

  • ls: ERROR: Cannot access stack: No such file or directory

in Script_Outputxxx:

( is it a problem with filemane? )

IGCM_sys_Put_Out : out_nemo /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out/N365d_5d_20030101_20031231_out_nemo
IGCM_sys_MkdirArchive : /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out
IGCM_sys_Chmod : 444 out_nemo
—Debug1—> OCE
—Debug1—> ICE
Files that will be deleted before next period-run :
ls: ERROR: Cannot access stack: No such file or directory
-r—r—r— N365d_5d_00016425_restart.nc
-r—r—r— N365d_5d_00016425_restart_ice.nc

(no ok still with libIGCM_v1_2)

  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_trends.nc DOES NOT EXIST
  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_diagap.nc DOES NOT EXIST
  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_output.abort.nc DOES NOT EXIST
To solve it
you have to activate corresponding keys, if not in Script_Output you will have:

IGCM_sys_Put_Out dynamic_1m_20010101_20011230_trends.nc DOES NOT EXIST .



List of possible remarks and suggestions :


  • add grep AUTO for namelist
  • with new trunk (1340) :

ln_mskland= (.true. ) new parameter to activate only(x)) in modeles/NEMO/OPA_SRC/diawri.F90 and in modeles/NEMO/LIM_SRC_2/limwri_2.F90

  • be careful at group (for example for a gensi one)

if we use a common account on CCRT we have to be sure that everygroup can have access to it, this is important for chmod 444, do another test!!!!!!

  • Every time that we change modipsl/config/ORCA2_LIM/IGCM00/config.card

we have to cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init

cancel modipsl/config/ORCA2_LIM/IGCM00/Job_Name

and run ./modipsl/util/ns_job script

  • To perform a second simulation after a run with PeriodState=Fatal

cancel modipsl/config/ORCA2_LIM/IGCM00/stack_error, modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName

but do not cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init file, if not run.card file is not re-created.

( File run.card.init is created with ins_job script )

  • Does it exists a caption file for run.card.init options?

Or Is there a reference file?
In to run.card.init file who can we undesrtand syntax of? :

TimeSeriesRunning=n
TimeSeriesCompleted=

SeasonalRunning=n

RebuildFrequency : how much has it? = PeriodLenght???

  • test tags/libIGCM_v1_2

Is it still true?: To use libIGCM trunk version it is necessary to add in config.card in POST Session:

RebuildFromArchive=NONE

  • Change in nemo.card or in nemo.driver variable: ORCA_NPDT_JOUR and OPA_NPDT_JOURS
  • How to do a long job without exit from queue

You can play with Duration Experience, Period Lenght? and Period Nb?, elapstim_req in this way :
in config.card Duration Experience is = DateBegin - DateEnd
in config.card PeriodLenght is how many time you do restart files
in Job_Name PeriodNb is in Job_Name elapstim_req is elapsed time limit

How can you see from this :
http://forge.ipsl.jussieu.fr/libigcm/attachment/wiki/DocUtilisateur/job.png

Your job is out from queue every time that is finished a Period Nb?, so you can do :

PeriodNb = ( Duration Experience ) / PeriodLenght

but be careful with elapstim_req, because this is a limit, you've to be sure that it exists a queue that can hold you.



Done :


  • run on CCRT machines ( mercure ) and at IDRIS ( brodie )

Tests for Calendar Type, Output frequency and Restarts:

Simulation environment

Simulation duration = 10 years

PeriodLength = 1Y

timestep: rdt = 5760 equivalent to ORCA_NPDT_JOUR = 15

Output Frequency Output Frequency Output Frequency
5d 1M 1Y
Calendar Type 360d ok ok ok
Calendar Type 365d ok ko ok
Calendar Type leap ok ko to do

for noleap year we can do now ( mars 2009 ) PeriodLenght=1M and WriteFrequency=1M ko = there are only 11 output ( point how NEMO do averadge for every month )

NOTE: control in:

  • why in /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N360d_1m/OCE/Debug/*_ocean.output

Namelist namrun

job number no = 0

is EVER = 0????

N365d_1m_20011231_restart.nc

Control if file name is correct!!!! (my job is N365d_1m)

Job Name versions calendar type
ONLY nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
option only(x) in diawri.F90
360d
ONLYio4 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 360d
dynamic nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic
360d
io7nemo3 nemo_v3, ioipsl_v2_1_7, libIGCM trunk 360d
v3360d nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2 360d
v31bok nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2 360d

  • monitoring : done on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

'ok': monitoring dynamic ( reading nemo.card ⇒ doing monitoring )

  • atlas : ok on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

NOTE: changed COMP/nemo.card and set Output Files? and Post_1M_grid_T[U,V,W]
NOTE: changed COMP/lim2.card and set Output Files? and Post_1M_grid_T[U,V,W]



To do list :


  • insert a control for processors' number in nemo.driver
    ongoing on mercure: MPIcontrol
  • simulation=100Y ( ongoing with tags/libIGCM_v1_2 )
    periodlenght=10y
    and sortiee=1Y
  • test mpi: ongoing on mercure

point out: all parameters in config.card :
is it necessary to change

#Job Run Options?='"-np "${NUM_PROC}"'
to

Job Run Options?='"-np "${BATCH_NUM_PROC_TOT}"' ?

if yes commit it



NOTE files to change (or changed) for NEMO :


  • modipsl/modeles/UTIL/fait_AA_make : for compilation's problems

difference with the old one:

in #- Compilation options —-

Committed: shade 10000 (for mercure)

  • modipsl/libIGCM/AA_job : for missing resolution's file

difference with the old one:
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)
RESOL=ORCA2

  • modipsl/modeles/NEMO/WORK/domain.F90 : for 360d Calendar Type?'s problems

difference with the old one:

WRITE(numout,*) ' leap year calendar (0/1/30) nleapy = ', nleapy

  • modipsl/config/ORCA2_LIM/IGCM00/COMP/nemo.driver : for 360d Calendar Type?'s problems

difference with the old one:

added nleapy variable that corresponds to nbisex on opa.driver

# nleapy leap year calendar (0/1) (30 for 360d) : for 360d Calendar Type?'s problems

PAT_NLEAPY=$( supergrep nleapy ${SUBMIT_DIR}/PARAM/namelist ) ……….. case ${config_UserChoices_CalendarType} in

leap | gregorian)
ORCA_NLEAPY=1;;
noleap)
ORCA_NLEAPY=0;;
360d)
ORCA_NLEAPY=30;;

esac

………….. -e "s%${PAT_NLEAPY}% nleapy=${ORCA_NLEAPY}%" \



Appendix A - Dods directory:


  • to put a directory on dods :

On idris is necessary to add on gaya a .rhosts file, to be able to do rsh:

on gaya you have to add in your home directory a .rhosts file ( with permissions 600 ) like:

rhodes.idris.fr
brodie.idris.fr
  • to remove a directory from dods :

On Mercure to CCRT DOds directory is :

${SAVE_DIR}/dods/public/${LOGIN}/${TAG_NAME}

cd ${SAVE_DIR}/dods
dods_rm public/${LOGIN}/${TAG_NAME}/directory



Simona Flavoni last update: Timestamp?

Attachments (1)

Download all attachments as: .zip