New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
user/flavoni/libIGCM – NEMO
wiki:user/flavoni/libIGCM

Version 371 (modified by flavoni, 15 years ago) (diff)

--

Last edited Timestamp?

Using libIGCM library with ORCA2_LIM


Purposes:


  • Document the using of libIGCM for NEMO, in configuration ORCA2_LIM



Quick start guide:


  • Define the following alias to use SVN :
     alias svn_ano='svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl'
    
  • Create and go into working directory :
     mkdir TRY ; cd TRY
    
  • Extract modipsl :
     svn_ano 
    
  • Extract NEMO :
     cd modipsl/util 
     ./model NEMO_IGCM 
    
  • Modify AA_job to avoid bug for ORCA2_LIM ( mars 2009 ) :
     cd ../libIGCM 
     vi AA_job : change RESOL file in Job_Name:  RESOL=ORCA2
    
  • Choosing ORCA2_LIM configuration :
     cd ../modeles/UTIL
     ./fait_config ORCA2_LIM
    
  • Install the makefiles :
     cd ../../util 
     ./ins_make
    
  • Compile :
     cd ../config/ORCA2_LIM 
     gmake
    
  • Submit Job :
     ( Editing at this point modipsl/config/ORCA2_LIM/IGCM00/config.card )
    
     ../../util/ins_job
     cd IGCM00 
     qsub Job_Name
    



Set up a simulation:


  • First of all see official documentation

http://forge.ipsl.jussieu.fr/libigcm/wiki/

  • All parameters in config.card file

In modipsl/config/ORCA2_LIM/IGCM00/config.card file you can set all parameters to manage your experience:

How to manage restart files ( in the same simulation )

in config.card you have to set duration of your experience and setting PeriodLenght parameter, so you create restart files.

( for example : if you have and experience of 10 years and a PeriodLenght of 1 year, you will have 10 restart files )

#-- Début et fin de Job
#-- "YYYY-MM-DD"
DateBegin?=2001-01-01
DateEnd?=2010-12-31
#============================
#-- 1Y, 1M, 5D, 1D
PeriodLength?=1Y

Note : If you want to use restart data from another simulation ( already run not in this moment ) you have to manage "Restarts" part of config.card

#================
#D-- Restarts -
[Restarts]

How to have mail of end of Job, on ccrt's machines

on mercure: in Job_Name: #PBS -M name@mail
on platine: In Job_Name: #BSUB -u name@mail

How to manage write output frequency

in config.card you have to set WriteFrequency of your experience

( for example : if you want output files every 1 month you have to put for Ocean

#===================
#D-- OCE -
[OCE]
WriteFrequency?="1M"

and if you want output files every 1 month you have to put for ICE

#===================
#D-- ICE -
[ICE]
WriteFrequency?="1M"

)

How to manage calendar type

in config.card you have to set CalendarType like 360d or leap or noleap

# for 360d :

you have to be careful at

DateEnd?=2010-12-30

because every month has 30 days

If you put for example DateEnd?=2005-12-31, you will have another run, of duration like your PeriodLenght

# for noleap :

you have to be careful at

WriteFrequency="1M"

and PeriodLength=1M

because in this moment simulation of 365 days it is possible only with a write output frequency of one month, and restart every month.

( If you put a different value for WriteFrequency you will not find restart files )

How to manage rdt ( i.e. timestep )

in COMP/nemo.driver you have to change parameter : ORCA_NPDT_JOUR=15

( 15 corresponds to 86400s/5760s; it does mean that in one day you have 15 iterations; and it does mean that timestep is 5760)

How control all parameters in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist

libIGCM change namelist's values but not in namelist's file, so it is possible to control all changes of namelist's parameters looking at :

{OUTPUT_DIR}/IGCM_OUT/ORCA2_LIM/N360d_5d/OCE/Debug/!JobName_date_date_ocean.output

Post Processing :


  • How to activate Monitoring

in config.card file

#D- If you want to produce time series, this flag determines
#D- frequency of post-processing submission
TimeSeriesFrequency?=NONE

you have to put your experience lenght. If you do an experience during 10 years you have to put:

TimeSeriesFrequency?=10Y

  • How to activate dynamic Monitoring (25/02/2009)

  • Copy original monitoring files in SUBMIT_DIR :
     cp ~/KEEPED_FILES/monitoring01_opa9_ORCA2.cfg $SUBMIT_DIR/.
     cp ~/KEEPED_FILES/monitoring01_lim2_ORCA2.cfg $SUBMIT_DIR/.
    
  • Copy monitoring.job in libIGCM :
     cp ~/KEEPED_FILES/monitoring.job modipsl/libIGCM/.
    
  • Control TimeSeries? in COMP/opa9.card and COMP/lim2.card:
     and if you add or remove TimeSeries in opa9.card or lim2.card => 
     it is necessary to add or remove correspondig variable in $SUBMIT_DIR/monitoring01_opa9[lim2]_ORCA2.cfg
    
  • How control monitoring on idris ( rhodes ) :
     Qstat -a
    
  • How to activate Atlas

in config.card file

#D- If you want to produce seasonal average, this flag determines
#D- the period of this average
SeasonalFrequency?=NONE

you have to put you experience lenght. If you do an experience during 10 years you have to put:

SeasonalFrequency?=10Y

  • How to manage frequency of post processing

( Note: in this moment is not possible to have atlas and monitoring of frequency different from 1M )

in COMP/nemo.card you have to change in [OutputFiles?] chapter:

Post_1M_grid_T (U,V,W)

and you have to change [Post_1M_grid_T(U,V,W)] chapter.

and in in COMP/lim2.card you have to change in [OutputFiles?] chapter:

List= (${PREFIX_NWRITE}_${DATE_OPA}_icemod.nc, ${R_OUT_ICE_NWRITE}/${PREFIX}_${WF1}_icemod.nc, Post_1M_icemod)

and you have to change [Post_1M_icemod] chapter.



Re-Run only post processing, for an already run job :


  • Run post processing

see: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ

in particular: (1) http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelafrontale

and (2) http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelamachinedecalcul


Example of post processing (1): a Job already run for 10 years form 2001 to 2010

in run.card

go to $scratchdir 
mkdir TESTPOST 
mkdir IGCM00 
cp -r $SUBMIT_DIR/.../COMP . 
cp -r $SUBMIT_DIR/.../config.card . 
cp -r $SUBMIT_DIR/.../modipsl/libIGCM/create_ts.job . 
vi create_ts.job . 
uncomment set -vx
substitue correct path for libIGCM=${libIGCM:=xxxxxxxxxxxxxx/modipsl/libIGCM}
qsub create_ts.job . 


Example of post processing (2): a Job already run for 10 years form 2001 to 2010

in run.card

  • modify OldPrefix
     OldPrefix=''Job_!Name''_(aa-1)mmdd
    
    where (aa-1)mmdd is last day of last month of one year before you simulation's start
  • modify CumulPeriod
    !CumulPeriod=10
    

is 10 years that you've already run

  • modify PeriodState
     !PeriodState=!OnQueue
    
  • modify TimeSeriesRunnging
     !TimeSeriesRunnging=n
    
    no because Time series are depending for each other, so if you want to start a time series it starts ONLY when a previous one is finished, so if you put TimeSeriesRunnging=y (i.e. yes) it can't never start
  • modify TimeSeriesCompleted
     !TimeSeriesCompleted=
    
    nothing because we want to do time series for all 10 years ( If not you have to put last time series already realised )

in Job_Name:

  • modify cputim_job
    #PBS -l cputim_job=4:00:00  # Limite temps 
    
  • modify DRYRUN
     DRYRUN=3
    
  • modify RUN_DIR_PATH
     Create, if necessary, a test directory (for example in you workdir)
     Decomment RUN_DIR_PATH=/$workdir/TESTPOST         
    
  • modify SUBMIT_DIR
     Decomment and put right submit directory (the one in which you are changed you run.card and you Job) 
     SUBMIT_DIR=/$workdir/modipsl/config/$CONFIG_NAME/EXP00 
    



How to make a parallel run:


  • First of all add a cpp key:

from Arnaud Caubel: "Il y a un pb MPI sur mercure avec la lib par defaut. J'avais mis sur le wiki un : Attention : sur mercure l'utilisation de la librairie MPI 7.2.4 est indispensable. Pour l'utiliser : export
SX_BASE_MPI=/SX/opt/mpisx/new avant de faire le gmake" (is only for coupled model?)

http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-add-modify-new-modules-or-new-cpp-keys

and then see:

http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-set-up-one-simulation#eztoc1210_5

  • Add key in modipsl/config/ORCA2_LIM/scripts/BB_make.ldef :
    key_mpp_mpi
    
  • Set number of processor in modipsl/modeles/NEMO/OPA_SRC/par_oce.F90 :
    set jpni, jpnj and jpnij    
    

  • Do an ins_make :
    ./modipsl/util/ins_make
    

  • Go in ORCA2_LIM directory :
    cd modipsl/config/ORCA2_LIM
    
  • Compile :
    gmake
    
  • Set same number of processors at compiling time in modipsl/config/ORCA2_LIM/IGCM00/config.card: :
    #-- Total Number of Processors 
    JobNumProcTot=#proc at compiling time
    
  • Enable run multi processors option in modipsl/config/ORCA2_LIM/IGCM00/config.card: :
    #-- Run Options :
    JobRunOptions='"-np "${BATCH_NUM_PROC_TOT}"'
    



List of possible errors


no monitoring


problem with common account and exit in monitoring.job

To solve it
you can do dynamic monitoring, so you can chose all your monitoring, or to exit same variables of IPSLCM5 and use same monitoring01_opa9_ORCA2.cfg file

in modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName :


  • Error : TERM = (vt100) ${HOME:-.}/.profile: .bashrc: not found
  • Error : ksh: D-----------------------------------==: not found

it can depends on .profile file.

To solve it
.profile file it can be like:

«# .profile (idris)
# Fichier d environnement a charger sur la frontale TX7 unix LINUX
# Vous pouvez mettre a jour le fichier .profile_tx
if [ `hostname!` = "brodie" -a -f $HOME/.profile_tx ]; then
. $HOME/.profile_tx
# Fichier d environnement a charger sur les noeuds SX8 unix SUPER-UX
# Vous pouvez mettre a jour le fichier .profile_sx
elif [ `hostname!` != "brodie" -a -f $HOME/.profile_sx ]; then
. $HOME/.profile_sx
else
echo
fi »

  • Error : cat: cannot open ........./modipsl/config/ORCA2_LIM/IGCM00/../.resol :

it is due to modipsl/libIGCM/AA_job file.

To solve it
change in this way:

#-- Resolution FLAG for oasis input
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)
RESOL=ORCA2

NOTE: Be careful it works by default with ORCA2 because in modipsl/libIGCM/AA_create_se but if we use ORCA0.5 ore ORCA0.25 is NOT TRUE!!!!!

file modipsl/libIGCM/AA_create_se there is by default

#D- TEMPORARY Flag to determine ocean resolution #D- Default : value from ocean driver if any RESOL_OCE=${RESOL_OCE:=ORCA2}

go in modipsl/config/ORCA2_LIM/IGCM00 directory

cancel run.card.init and Job_Name files

and then do an ins_job again.

(does not still fixed in libIGCM_v1_2)

  • Error : Input files and BC for NEMO DO NOT EXIST (i.e. sst_data.nc, sss_data.nc, geothermal_heating.nc, bathy_level.nc, etc...)

IGCM_sys_Get, ERROR : regular file ...../OCE/ORCA2_LIM/bathy_level.nc DOES NOT EXIST .

To solve it
In this moment we've still to download on gaya.idris.fr:~/!reee512/NEMO/ORCA2_LIM_nemo_v3.tar file, and to untar it into the directory:

${R_BC}/OCE/ORCA2_LIM

and where ${R_BC} is a variable that can be changed in modipsl/config/ORCA2_LIM/IGCM00/config.card file.

  • Error : IGCM_debug_Exit : IGCM_config_Initialize Error PeriodState : Fatal
To solve it
change line in modipsl/config/ORCA2_LIM/IGCM00/run.card

PeriodState=Fatal
in
PeriodState=

or rm modipsl/config/ORCA2_LIM/IGCM00/run.card

  • ls: ERROR: Cannot access stack: No such file or directory

in Script_Outputxxx:

( is it a problem with filemane? )

IGCM_sys_Put_Out : out_nemo /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out/N365d_5d_20030101_20031231_out_nemo
IGCM_sys_MkdirArchive : /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out
IGCM_sys_Chmod : 444 out_nemo
--Debug1--> OCE
--Debug1--> ICE
Files that will be deleted before next period-run :
ls: ERROR: Cannot access stack: No such file or directory
-r--r--r-- N365d_5d_00016425_restart.nc
-r--r--r-- N365d_5d_00016425_restart_ice.nc

(no ok still with libIGCM_v1_2)

  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_trends.nc DOES NOT EXIST
  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_diagap.nc DOES NOT EXIST
  • WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_output.abort.nc DOES NOT EXIST
To solve it
you have to activate corresponding keys, if not in Script_Output you will have:

IGCM_sys_Put_Out dynamic_1m_20010101_20011230_trends.nc DOES NOT EXIST .

No ATLAS :


  • There is no ATLAS directory

it can depends by your config_UserChoices_TagName.

To solve it
run only post processing of ATLAS: you've to do like in

http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelafrontale :

and then qsub create_se.job

and if in your SE.out you see that your config_UserChoices_TagName is not in the list you have to add your config_UserChoices_TagName in create_se.job



List of possible remarks and suggestions :


  • add grep AUTO for namelist
  • with new trunk (1340) :

Color( green, ln_mskland= (.true. ))? new parameter to activate only(x)) in modeles/NEMO/OPA_SRC/diawri.F90 and in modeles/NEMO/LIM_SRC_2/limwri_2.F90

  • be careful at group (for example for a gensi one)

if we use a common account on CCRT we have to be sure that everygroup can have access to it, this is important for chmod 444, do another test!!!!!!

  • Every time that we change modipsl/config/ORCA2_LIM/IGCM00/config.card

we have to cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init

cancel modipsl/config/ORCA2_LIM/IGCM00/Job_Name

and run ./modipsl/util/ns_job script

  • To perform a second simulation after a run with PeriodState=Fatal

cancel modipsl/config/ORCA2_LIM/IGCM00/stack_error, modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName

but do not cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init file, if not run.card file is not re-created.

( File run.card.init is created with ins_job script )

  • Does it exists a caption file for run.card.init options?

Or Is there a reference file?
In to run.card.init file who can we undesrtand syntax of? :

TimeSeriesRunning=n
TimeSeriesCompleted=

SeasonalRunning=n

RebuildFrequency : how much has it? = PeriodLenght???

  • test tags/libIGCM_v1_2

Is it still true?: To use libIGCM trunk version it is necessary to add in config.card in POST Session:

RebuildFromArchive=NONE

  • How to do a long job without exit from queue

You can play with Duration Experience, PeriodLenght? and PeriodNb?, elapstim_req in this way :
in config.card Duration Experience is = DateBegin - DateEnd
in config.card PeriodLenght is how many time you do restart files
in Job_Name PeriodNb is in Job_Name elapstim_req is elapsed time limit

How can you see from this :
http://forge.ipsl.jussieu.fr/libigcm/attachment/wiki/DocUtilisateur/job.png

Your job is out from queue every time that is finished a PeriodNb?, so you can do :

PeriodNb = ( Duration Experience ) / PeriodLenght

but be careful with elapstim_req, because this is a limit, you've to be sure that it exists a queue that can hold you.



Done :


  • run on CCRT machines ( mercure ) and at IDRIS ( brodie )

Tests for Calendar Type, Output frequency and Restarts:

Simulation environment

Simulation duration = 10 years

PeriodLength = 1Y

timestep: rdt = 5760 equivalent to ORCA_NPDT_JOUR = 15

Output Frequency Output Frequency Output Frequency
5d 1M 1Y
Calendar Type 360d ok ok ok
Calendar Type 365d ok ko ok
Calendar Type leap ok ko to do

for noleap year we can do now ( mars 2009 ) PeriodLenght=1M and WriteFrequency=1M ko = there are only 11 output ( point how NEMO do averadge for every month )

Job Name versions calendar type
ONLY nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
option only(x) in diawri.F90
360d
ONLYio4 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 360d
dynamic nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic
360d
io7nemo3 nemo_v3, ioipsl_v2_1_7, libIGCM trunk 360d
v3360d nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2 360d
v31bok nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2 360d
.... trunk nemo, trunk libIGCM 360d
.... trunk nemo, trunk libIGCM noleap
.... PARALLEL : trunk nemo, trunk libIGCM 360d
.... PARALLEL : trunk nemo, trunk libIGCM noleap
.... trunk nemo, trunk libIGCM 360d
.... trunk nemo, trunk libIGCM 360d

  • monitoring : done on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

'ok': monitoring dynamic ( reading nemo.card => doing monitoring )

  • atlas : ok on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M

NOTE: changed COMP/nemo.card and set OutputFiles? and Post_1M_grid_T[U,V,W]
NOTE: changed COMP/lim2.card and set OutputFiles? and Post_1M_grid_T[U,V,W]

  • simulation=100Y ( done with tags/libIGCM_v1_2 )
    periodlenght=10y
    and sortiee=1Y
  • test mpi: ok on mercure



To do list :


  • To have ZNL ATLAS, to avoid additional (but this is a not definitively solution! Be careful at it!, like IPSLCM5) :
  1. in modipsl/modeles/NEMO/WORK/diaptr.F90 put:
LOGICAL, PUBLIC
ln_ptrcomp = ;TRUE.

ln_diaptr = .true.
ln_diaznl = .true.
ln_subbas = .true.

  1. copy subbasins.nc in R_BC directory (see config.card)
  1. add in modipsl/config/ORCA2_LIM/IGCM00/COMP/opa9.card TimeSeries:
    [Post_1M_diaptr]
    Patches = ()
    GatherWithInternal? = (lat, deptht, depthw, time_counter)
    TimeSeriesVars? = (zotemglo, zosalglo, zomsfglo, zotematl, zosalatl, zomsfatl, zotempac, zosalpac, zomsfpac, zotemind, zosalind, zomsfind, zotemipc, zosalipc, zomsfipc, sohtatl, sostatl, sohtpac, sostpac, sohtind, sostind, sohtipc, sostipc, sophtadv, sophtove, sophtldf, sopstadv, sopstove, zomsfeiv, sophteiv, sopsteiv)
  1. change in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist :
    nf_ptr_wri : write frequency (default is 15 that is 1d)

if you want 1m with CalendarType=360d is nf_ptr_wri=450

if you want 1m with CalendarType=noleap is nf_ptr_wri=465

  1. change AA_atlas_ORCA_LIM to add (like IPSLCM5)

# WHILE WAITING FOR A MORE GENERIC SOLUTION
if [ ${config_UserChoices_TagName} = "ORCA2_LIM2"] ; then

IGCM_sys_Get ${R_OUT_OCE}/Analyse/SE/${config_UserChoices_JobName}_SE_${YEARS}_1M_diaptr.nc ${RUN_DIR} ;
atlas -o ${RUN_DIR}/${R_ATLAS}/ZNL -d -l ${FER_ATLAS}/atlas_l_${RESOL_OCE}_znl.cfg ${RUN_DIR}/${config_UserChoices_JobName}_SE_${YEARS}_1M_diaptr.nc ;
for file in $( ls ${R_ATLAS}/ZNL/*/*jnl ) ; do

echo -e "\n\n################################" ;
echo -e "$file\n" ; cat $file ;

done

fi

# WHILE WAITING FOR A MORE GENERIC SOLUTION
if [ ${config_UserChoices_TagName} = "ORCA2_LIM2" ] ; then

IGCM_sys_Rm ${config_UserChoices_JobName}_SE_${YEARS}_1M_diaptr.nc ;

fi

  • Put subbasins.nc in common account/v3_1/. but this is a not definitively solution! Be careful at it!
  • document how to have withe continents ( in namelist ln_mskland )
  • insert a control for processors' number in nemo.driver
    ongoing on mercure: MPIcontrol : IS NOT POSSIBLE AT NEMO LEVEL, IT IS POSSIBLE AT LIBIGCM LEVEL.



NOTE files to change (or changed) for NEMO :


  • modipsl/modeles/UTIL/fait_AA_make : for compilation's problems

difference with the old one:

in #- Compilation options ---

Committed: shade 10000 (for mercure)

  • modipsl/modeles/NEMO/WORK/domain.F90 : for 360d CalendarType?'s problems

difference with the old one:

WRITE(numout,*) ' leap year calendar (0/1/30) nleapy = ', nleapy



Appendix A - Dods directory:


  • to put a directory on dods :

On idris is necessary to add on gaya a .rhosts file, to be able to do rsh:

on gaya you have to add in your home directory a .rhosts file ( with permissions 600 ) like:

rhodes.idris.fr
brodie.idris.fr
ulam.idris.fr
  • to remove a directory from dods :

On Mercure to CCRT DOds directory is :

${SAVE_DIR}/dods/public/${LOGIN}/${TAG_NAME}

cd ${SAVE_DIR}/dods
dods_rm public/${LOGIN}/${TAG_NAME}/directory

Appendix B - Ulam:

  • ulam:

To submit a job : llsubmit
To cancel a job : llcancel
To follow a job : llq



Simona Flavoni last update: Timestamp?

Attachments (1)

Download all attachments as: .zip