Version 406 (modified by flavoni, 15 years ago) (diff) |
---|
Last edited Timestamp?
-
Using libIGCM library with ORCA2_LIM
- Purposes:
- Quick start guide:
- Set up a simulation:
- Post Processing :
- Re-Run only post processing, for an already run job :
- Example of post processing (1): a Job already run for 10 years form 2001 …
- Example of post processing (2): a Job already run for 10 years form 2001 …
- How to make a parallel run:
- How to have ZNL ATLAS ( using diaptr.F90 ):
- Mini start guide to use diaptr in MPI mode
- List of possible errors
- List of possible remarks and suggestions :
- Done :
- To do list :
- NOTE files to change (or changed) for NEMO :
- Appendix A - Dods directory:
- Appendix B - Ulam:
Using libIGCM library with ORCA2_LIM
Purposes:
- Document the using of libIGCM for NEMO, in configuration ORCA2_LIM
Quick start guide:
- Define the following alias to use SVN :
alias svn_ano='svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl'
- Create and go into working directory :
mkdir TRY ; cd TRY
- Extract modipsl :
svn_ano
- Extract NEMO :
cd modipsl/util ./model NEMO_IGCM
- Modify AA_job to avoid bug for ORCA2_LIM ( mars 2009 ) :
cd ../libIGCM vi AA_job : change RESOL file in Job_Name: RESOL=ORCA2
- Choosing ORCA2_LIM configuration :
cd ../modeles/UTIL ./fait_config ORCA2_LIM
- Install the makefiles :
cd ../../util ./ins_make
- Compile :
cd ../config/ORCA2_LIM gmake
- Submit Job :
( Editing at this point modipsl/config/ORCA2_LIM/IGCM00/config.card ) ../../util/ins_job cd IGCM00 qsub Job_Name
Set up a simulation:
- First of all see official documentation
- All parameters in config.card file
In modipsl/config/ORCA2_LIM/IGCM00/config.card file you can set all parameters to manage your experience:
How to manage restart files ( in the same simulation )
in config.card you have to set duration of your experience and setting PeriodLenght parameter, so you create restart files.
( for example : if you have and experience of 10 years and a PeriodLenght of 1 year, you will have 10 restart files )
#-- Début et fin de Job
#-- "YYYY-MM-DD"
DateBegin?=2001-01-01
DateEnd?=2010-12-31
#============================
#-- 1Y, 1M, 5D, 1D
PeriodLength?=1Y
Note : If you want to use restart data from another simulation ( already run not in this moment ) you have to manage "Restarts" part of config.card
#================
#D-- Restarts -
[Restarts]
How to have mail of end of Job, on ccrt's machines
on mercure: in Job_Name: #PBS -M name@mail
on platine: In Job_Name: #BSUB -u name@mail
How to manage write output frequency
in config.card you have to set WriteFrequency of your experience
( for example : if you want output files every 1 month you have to put for Ocean
#===================
#D-- OCE -
[OCE]
WriteFrequency?="1M"
and if you want output files every 1 month you have to put for ICE
#===================
#D-- ICE -
[ICE]
WriteFrequency?="1M"
)
How to manage calendar type
in config.card you have to set CalendarType like 360d or leap or noleap
# for 360d :
you have to be careful at
DateEnd?=2010-12-30
because every month has 30 days
If you put for example DateEnd?=2005-12-31, you will have another run, of duration like your PeriodLenght
# for noleap :
you have to be careful at
WriteFrequency="1M"
and PeriodLength=1M
because in this moment simulation of 365 days it is possible only with a write output frequency of one month, and restart every month.
( If you put a different value for WriteFrequency you will not find restart files )
How to manage rdt ( i.e. timestep )
in COMP/nemo.driver you have to change parameter : ORCA_NPDT_JOUR=15
( 15 corresponds to 86400s/5760s; it does mean that in one day you have 15 iterations; and it does mean that timestep is 5760)
How control all parameters in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist
libIGCM change namelist's values but not in namelist's file, so it is possible to control all changes of namelist's parameters looking at :
{OUTPUT_DIR}/IGCM_OUT/ORCA2_LIM/N360d_5d/OCE/Debug/!JobName_date_date_ocean.output
Post Processing :
- How to activate Monitoring
in config.card file
#D- If you want to produce time series, this flag determines
#D- frequency of post-processing submission
TimeSeriesFrequency?=NONE
you have to put your experience lenght. If you do an experience during 10 years you have to put:
- How to activate dynamic Monitoring (25/02/2009)
- Copy original monitoring files in SUBMIT_DIR :
cp ~/KEEPED_FILES/monitoring01_opa9_ORCA2.cfg $SUBMIT_DIR/. cp ~/KEEPED_FILES/monitoring01_lim2_ORCA2.cfg $SUBMIT_DIR/.
- Copy monitoring.job in libIGCM :
cp ~/KEEPED_FILES/monitoring.job modipsl/libIGCM/.
- Control TimeSeries? in COMP/opa9.card and COMP/lim2.card:
and if you add or remove TimeSeries in opa9.card or lim2.card => it is necessary to add or remove correspondig variable in $SUBMIT_DIR/monitoring01_opa9[lim2]_ORCA2.cfg
- How control monitoring on idris ( rhodes ) :
Qstat -a
- How to activate Atlas
in config.card file
#D- If you want to produce seasonal average, this flag determines
#D- the period of this average
SeasonalFrequency?=NONE
you have to put you experience lenght. If you do an experience during 10 years you have to put:
- How to manage frequency of post processing
( Note: in this moment is not possible to have atlas and monitoring of frequency different from 1M )
in COMP/nemo.card you have to change in [OutputFiles?] chapter:
Post_1M_grid_T (U,V,W)
and you have to change [Post_1M_grid_T(U,V,W)] chapter.
and in in COMP/lim2.card you have to change in [OutputFiles?] chapter:
List= (${PREFIX_NWRITE}_${DATE_OPA}_icemod.nc, ${R_OUT_ICE_NWRITE}/${PREFIX}_${WF1}_icemod.nc, Post_1M_icemod)
and you have to change [Post_1M_icemod] chapter.
Re-Run only post processing, for an already run job :
- Run post processing
see: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ
in particular: (1) http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelafrontale
Example of post processing (1): a Job already run for 10 years form 2001 to 2010
in run.card
go to $scratchdir
mkdir TESTPOST
mkdir IGCM00
cp -r $SUBMIT_DIR/.../COMP .
cp -r $SUBMIT_DIR/.../config.card .
cp -r $SUBMIT_DIR/.../modipsl/libIGCM/create_ts.job .
vi create_ts.job . uncomment set -vx substitue correct path for libIGCM=${libIGCM:=xxxxxxxxxxxxxx/modipsl/libIGCM}
qsub create_ts.job .
Example of post processing (2): a Job already run for 10 years form 2001 to 2010
in run.card
- modify OldPrefix
OldPrefix=''Job_!Name''_(aa-1)mmdd
where (aa-1)mmdd is last day of last month of one year before you simulation's start
- modify CumulPeriod
!CumulPeriod=10
is 10 years that you've already run
- modify PeriodState
!PeriodState=!OnQueue
- modify TimeSeriesRunnging
!TimeSeriesRunnging=n
no because Time series are depending for each other, so if you want to start a time series it starts ONLY when a previous one is finished, so if you put TimeSeriesRunnging=y (i.e. yes) it can't never start
- modify TimeSeriesCompleted
!TimeSeriesCompleted=
nothing because we want to do time series for all 10 years ( If not you have to put last time series already realised )
in Job_Name:
- modify cputim_job
#PBS -l cputim_job=4:00:00 # Limite temps
- modify DRYRUN
DRYRUN=3
- modify RUN_DIR_PATH
Create, if necessary, a test directory (for example in you workdir) Decomment RUN_DIR_PATH=/$workdir/TESTPOST
- modify SUBMIT_DIR
Decomment and put right submit directory (the one in which you are changed you run.card and you Job) SUBMIT_DIR=/$workdir/modipsl/config/$CONFIG_NAME/EXP00
How to make a parallel run:
- First of all add a cpp key:
and then see:
- Add key in modipsl/config/ORCA2_LIM/scripts/BB_make.ldef :
key_mpp_mpi
- Set number of processor in modipsl/modeles/NEMO/OPA_SRC/par_oce.F90 :
set jpni, jpnj and jpnij
- Do an ins_make :
./modipsl/util/ins_make
- Go in ORCA2_LIM directory :
cd modipsl/config/ORCA2_LIM
- Compile :
gmake
- Set same number of processors at compiling time in modipsl/config/ORCA2_LIM/IGCM00/config.card: :
#-- Total Number of Processors JobNumProcTot=#proc at compiling time
- Enable run multi processors option in modipsl/config/ORCA2_LIM/IGCM00/config.card: :
#-- Run Options : JobRunOptions='"-np "${BATCH_NUM_PROC_TOT}"'
How to have ZNL ATLAS ( using diaptr.F90 ):
To have ZNL ATLAS, to avoid additional (but this is a not definitively solution! Be careful at it!, like IPSLCM5) :
- in modipsl/config/ORCA2_LIM/scripts/BB_make.ldef add key_diahth to have sohtc300 variable in MONITORING
- in modipsl/modeles/NEMO/WORK/diaptr.F90 put :
LOGICAL, PUBLIC : : ln_ptrcomp = .TRUE.
- copy subbasins.nc in R_BC directory (see config.card)
- add in modipsl/config/ORCA2_LIM/IGCM00/COMP/opa9.card TimeSeries:
[Post_1M_diaptr]
Patches = ()
GatherWithInternal? = (lat, deptht, depthw, time_counter)
TimeSeriesVars? = (zotemglo, zosalglo, zomsfglo, zotematl, zosalatl, zomsfatl, zotempac, zosalpac, zomsfpac, zotemind, zosalind, zomsfind, zotemipc, zosalipc, zomsfipc, sohtatl, sostatl, sohtpac, sostpac, sohtind, sostind, sohtipc, sostipc, sophtadv, sophtove, sophtldf, sopstadv, sopstove, zomsfeiv, sophteiv, sopsteiv)
- change in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist :
ln_diaptr = .true.
ln_diaznl = .true.
ln_subbas = .true.
nf_ptr_wri : write frequency (default is 15 that is 1d)
Mini start guide to use diaptr in MPI mode
alias svn_ano='svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl' cd modipsl/util ./model NEMO_IGCM add key_diahth in ../../modipsl/config/ORCA2_LIM/scripts/BB_make.ldef add key_mpp_mpi in ../../modipsl/config/ORCA2_LIM/scripts/BB_make.ldef chose jpni, jpnj, jpnij in ../NEMO/OPA_SRC/par_oce.F90 cd ../modeles/UTIL ./fait_config ORCA2_LIM put in modipsl/modeles/NEMO/WORK/diaptr.F90 "LOGICAL, PUBLIC : : ln_ptrcomp = .TRUE." cd ../../util ./ins_make cd ../config/ORCA2_LIM gmake put in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist ln_diaptr = .true. ln_diaznl = .true. ln_subbas = .true. and control value of "nf_ptr_wri" : write frequency (default is 15 that is 1d) ../../util/ins_job cd IGCM00 qsub Job_Name
List of possible errors
no monitoring
problem with common account and exit in monitoring.job
- To solve it
- you can do dynamic monitoring, so you can chose all your monitoring, or to exit same variables of IPSLCM5 and use same monitoring01_opa9_ORCA2.cfg file
in modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName :
- Error : TERM = (vt100) ${HOME:-.}/.profile: .bashrc: not found
- Error : ksh: D-----------------------------------==: not found
it can depends on .profile file.
- To solve it
- .profile file it can be like:
«# .profile (idris)
# Fichier d environnement a charger sur la frontale TX7 unix LINUX
# Vous pouvez mettre a jour le fichier .profile_tx
if [ `hostname!` = "brodie" -a -f $HOME/.profile_tx ]; then
. $HOME/.profile_tx
# Fichier d environnement a charger sur les noeuds SX8 unix SUPER-UX
# Vous pouvez mettre a jour le fichier .profile_sx
elif [ `hostname!` != "brodie" -a -f $HOME/.profile_sx ]; then
. $HOME/.profile_sx
else
echo
fi »
- Error : cat: cannot open ........./modipsl/config/ORCA2_LIM/IGCM00/../.resol :
it is due to modipsl/libIGCM/AA_job file.
- To solve it
- change in this way:
#-- Resolution FLAG for oasis input
#RESOL=$(cat ${SUBMIT_DIR}/../.resol)
RESOL=ORCA2
NOTE: Be careful it works by default with ORCA2 because in modipsl/libIGCM/AA_create_se but if we use ORCA0.5 ore ORCA0.25 is NOT TRUE!!!!!
file modipsl/libIGCM/AA_create_se there is by default
#D- TEMPORARY Flag to determine ocean resolution #D- Default : value from ocean driver if any RESOL_OCE=${RESOL_OCE:=ORCA2}
go in modipsl/config/ORCA2_LIM/IGCM00 directory
cancel run.card.init and Job_Name files
and then do an ins_job again.
(does not still fixed in libIGCM_v1_2)
- Error : Input files and BC for NEMO DO NOT EXIST (i.e. sst_data.nc, sss_data.nc, geothermal_heating.nc, bathy_level.nc, etc...)
IGCM_sys_Get, ERROR : regular file ...../OCE/ORCA2_LIM/bathy_level.nc DOES NOT EXIST .
- To solve it
- In this moment we've still to download on gaya.idris.fr:~/!reee512/NEMO/ORCA2_LIM_nemo_v3.tar file, and to untar it into the directory:
${R_BC}/OCE/ORCA2_LIM
and where ${R_BC} is a variable that can be changed in modipsl/config/ORCA2_LIM/IGCM00/config.card file.
- Error : IGCM_debug_Exit : IGCM_config_Initialize Error PeriodState : Fatal
- To solve it
- change line in modipsl/config/ORCA2_LIM/IGCM00/run.card
PeriodState=Fatal
in
PeriodState=
or rm modipsl/config/ORCA2_LIM/IGCM00/run.card
- ls: ERROR: Cannot access stack: No such file or directory
in Script_Outputxxx:
( is it a problem with filemane? )
IGCM_sys_Put_Out : out_nemo /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out/N365d_5d_20030101_20031231_out_nemo
IGCM_sys_MkdirArchive : /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out
IGCM_sys_Chmod : 444 out_nemo
--Debug1--> OCE
--Debug1--> ICE
Files that will be deleted before next period-run :
ls: ERROR: Cannot access stack: No such file or directory
-r--r--r-- N365d_5d_00016425_restart.nc
-r--r--r-- N365d_5d_00016425_restart_ice.nc
(no ok still with libIGCM_v1_2)
- WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_trends.nc DOES NOT EXIST
- WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_diagap.nc DOES NOT EXIST
- WARNING : IGCM_sys_Put_Out JobName_xfrequency_datebegin_dateend_output.abort.nc DOES NOT EXIST
- To solve it
- you have to activate corresponding keys, if not in Script_Output you will have:
IGCM_sys_Put_Out dynamic_1m_20010101_20011230_trends.nc DOES NOT EXIST .
no ATLAS :
- There is no ATLAS directory
it can depends by your config_UserChoices_TagName.
- To solve it
- run only post processing of ATLAS: you've to do like in
and then qsub create_se.job
and if in your SE.out you see that your config_UserChoices_TagName is not in the list you have to add your config_UserChoices_TagName in create_se.job
List of possible remarks and suggestions :
- add grep AUTO for namelist
- with new trunk (1340) :
Color( green, ln_mskland= (.true. ))? new parameter to activate only(x)) in modeles/NEMO/OPA_SRC/diawri.F90 and in modeles/NEMO/LIM_SRC_2/limwri_2.F90
- be careful at group (for example for a gensi one)
if we use a common account on CCRT we have to be sure that everygroup can have access to it, this is important for chmod 444, do another test!!!!!!
- Every time that we change modipsl/config/ORCA2_LIM/IGCM00/config.card
we have to cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init
cancel modipsl/config/ORCA2_LIM/IGCM00/Job_Name
and run ./modipsl/util/ns_job script
- To perform a second simulation after a run with PeriodState=Fatal
cancel modipsl/config/ORCA2_LIM/IGCM00/stack_error, modipsl/config/ORCA2_LIM/IGCM00/Script_Output_JobName
but do not cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init file, if not run.card file is not re-created.
( File run.card.init is created with ins_job script )
- Does it exists a caption file for run.card.init options?
Or Is there a reference file?
In to run.card.init file who can we undesrtand syntax of? :
TimeSeriesRunning=n
TimeSeriesCompleted=
SeasonalRunning=n
RebuildFrequency=1Y
- RebuildFromArchive=NONE
In MPI it is good to use Rebuild in "archive" machine, to not ovrerload calcul machine.
You have to put in config.card in POST Session:
RebuildFrequency=1Y
RebuildFromArchive=true
NOTA BENE: If you put RebuildFrequency=1Y and you forget to set RebuildFromArchive=true, it is very dangerous. NEVER DO IT!
- How to do a long job without exit from queue
You can play with Duration Experience, PeriodLenght? and PeriodNb?, elapstim_req in this way :
in config.card Duration Experience is = DateBegin - DateEnd
in config.card PeriodLenght is how many time you do restart files
in Job_Name PeriodNb is in Job_Name elapstim_req is elapsed time limit
How can you see from this :
http://forge.ipsl.jussieu.fr/libigcm/attachment/wiki/DocUtilisateur/job.png
Your job is out from queue every time that is finished a PeriodNb?, so you can do :
PeriodNb = ( Duration Experience ) / PeriodLenght
but be careful with elapstim_req, because this is a limit, you've to be sure that it exists a queue that can hold you.
Done :
- run on CCRT machines ( mercure ) and at IDRIS ( brodie )
Tests for Calendar Type, Output frequency and Restarts:
Simulation environment
Simulation duration = 10 years
PeriodLength = 1Y ( if CalendarType=360d )
PeriodLength = 1M ( if CalendarType=noleap )
timestep: rdt = 5760 equivalent to ORCA_NPDT_JOUR = 15
Output Frequency | Output Frequency | Output Frequency | ||
5d | 1M | 1Y | ||
Calendar Type | 360d | ok | ok | ok |
Calendar Type | noleap | ok | ok | ko |
Calendar Type | leap | ok | ko | to do |
for noleap year we can do now ( mars 2009 ) PeriodLenght=1M and WriteFrequency=1M
ko = there are only 11 output ( point how NEMO do averadge for every month )
- Test result :
Job Name versions calendar type ONLY nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
option only(x) in diawri.F90360d ONLYio4 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 360d dynamic nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic360d io7nemo3 nemo_v3, ioipsl_v2_1_7, libIGCM trunk 360d v3360d nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2
with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic360d v31bok nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2 360d MPImulti PARALLEL : nemo_v3_1, libIGCM_v1_2 360d MPImultiok PARALLEL : nemo_v3_1, libIGCM_v1_2, and PeriodNb?=10 360d MPIv31 PARALLEL : nemo_v3_1, libIGCM_v1_2 360d noleapdy trunk nemo, libIGCM_v1_2 noleap ON1340st trunk nemo, trunk libIGCM
option only(x) in diawri.F90
ORCA_NSTOCK=ORCA_NITEND in nemo.drivernoleap trunk trunk nemo, trunk libIGCM, and PeriodNb?=120
ORCA_NSTOCK=ORCA_NITEND in nemo.drivernoleap
Job Name versions calendar type orca9 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2
option only(x) in diawri.F90360d opa9 nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 360d OPA1Y nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 ith in SUBMIT_DIR: monitoring dynamic 360d OPAparam nemo_v3, ioipsl_v2_1_7, libIGCM trunk 360d opaOK nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2 360d opaMPI nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2 360d opaMPInl trunk nemo, trunk libIGCM 360d newMonit trunk nemo, trunk libIGCM noleap ZNLmpi PARALLEL : trunk nemo, trunk libIGCM
use diaptr.F90 to have ZNL ATLAS360d znlmpi300 PARALLEL : trunk nemo, trunk libIGCM revision 95
use diaptr.F90 to have ZNL ATLAS360d libIGCM105 trunk nemo, trunk libIGCM revision 105
use diaptr.F90 to have ZNL ATLAS360d
- monitoring : done on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M
'ok': monitoring dynamic ( reading nemo.card => doing monitoring )
- atlas : ok on mercure/brodie for experience=10Y, PeriodLenght=1Y, WriteFrequency=1M
NOTE: changed COMP/nemo.card and set OutputFiles? and Post_1M_grid_T[U,V,W]
NOTE: changed COMP/lim2.card and set OutputFiles? and Post_1M_grid_T[U,V,W]
- simulation=100Y ( done with tags/libIGCM_v1_2 )
periodlenght=10y
and sortiee=1Y
- test mpi: ok on mercure
To do list :
- Put subbasins.nc in common account/v3_1/. but this is a not definitively solution! Be careful at it!
- document how to have withe continents ( in namelist ln_mskland )
- insert a control for processors' number in nemo.driver
ongoing on mercure: MPIcontrol : IS NOT POSSIBLE AT NEMO LEVEL, IT IS POSSIBLE AT LIBIGCM LEVEL.
NOTE files to change (or changed) for NEMO :
- modipsl/modeles/UTIL/fait_AA_make : for compilation's problems
difference with the old one:
in #- Compilation options ---
Committed: shade 10000 (for mercure)
- modipsl/modeles/NEMO/WORK/domain.F90 : for 360d CalendarType?'s problems
difference with the old one:
WRITE(numout,*) ' leap year calendar (0/1/30) nleapy = ', nleapy
Appendix A - Dods directory:
- to put a directory on dods :
On idris is necessary to add on gaya a .rhosts file, to be able to do rsh:
on gaya you have to add in your home directory a .rhosts file ( with permissions 600 ) like:
rhodes.idris.fr brodie.idris.fr ulam.idris.fr
- to remove a directory from dods :
On Mercure to CCRT DOds directory is :
${SAVE_DIR}/dods/public/${LOGIN}/${TAG_NAME}
cd ${SAVE_DIR}/dods dods_rm public/${LOGIN}/${TAG_NAME}/directory
Appendix B - Ulam:
- ulam:
To submit a job : llsubmit
To cancel a job : llcancel
To follow a job : llq
Simona Flavoni last update: Timestamp?
Attachments (1)
- figura-libigcm.jpg (1.3 MB) - added by trac 6 years ago.
Download all attachments as: .zip