Last edited [[Timestamp]] [[BR]] [[PageOutline]] = '''Using libIGCM library with ORCA2_LIM''' = [[BR]] == Purposes: == [[BR]] * Document the using of libIGCM for NEMO, in configuration ORCA2_LIM [[BR]] [[BR]] == Quick start guide: == [[BR]] * Define the following alias to use SVN : [[BR]] {{{ alias svn_ano='svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl' }}} * Create and go into working directory : [[BR]] {{{ mkdir TRY ; cd TRY }}} * Extract modipsl : [[BR]] {{{ svn_ano }}} * Extract NEMO : [[BR]] {{{ cd modipsl/util ./model NEMO_IGCM }}} * Choosing ORCA2_LIM configuration : [[BR]] {{{ cd ../modeles/UTIL ./fait_config ORCA2_LIM }}} * Install the makefiles : [[BR]] {{{ cd ../../util ./ins_make }}} * Compile : [[BR]] {{{ cd ../config/ORCA2_LIM gmake }}} * Submit Job : [[BR]] {{{ ( Editing at this point modipsl/config/ORCA2_LIM/IGCM00/config.card ) ../../util/ins_job change RESOL file in Job_Name: RESOL=ORCA2 cd IGCM00 qsub Job_NL1 }}} [[BR]] [[BR]] == Set up a simulation: == [[BR]] * '''First of all see official documentation''' http://forge.ipsl.jussieu.fr/libigcm/wiki/ * '''All parameters in config.card file''' In modipsl/config/ORCA2_LIM/IGCM00/'''config.card''' file you can set all parameters to manage your experience: '''How to manage restart files ( in the same simulation )''' in config.card you have to set duration of your experience and setting !PeriodLenght parameter, so you create restart files. ( for example : if you have and experience of 10 years and a !PeriodLenght of 1 year, you will have 10 restart files ) #-- Début et fin de Job [[BR]] #-- "YYYY-MM-DD" [[BR]] DateBegin=2001-01-01 [[BR]] DateEnd=2010-12-31 [[BR]] #============================ [[BR]] #-- 1Y, 1M, 5D, 1D [[BR]] PeriodLength=1Y [[BR]] '''Note : ''' If you want to use restart data from another simulation ( already run not in this moment ) you have to manage "Restarts" part of config.card #================ [[BR]] #D-- Restarts -[[BR]] [Restarts] '''How to manage write output frequency''' in config.card you have to set !WriteFrequency of your experience ( for example : if you want output files every 1 month you have to put for Ocean #=================== [[BR]] #D-- OCE - [[BR]] [OCE] [[BR]] WriteFrequency="1M" and if you want output files every 1 month you have to put for ICE #=================== [[BR]] #D-- ICE - [[BR]] [ICE] [[BR]] WriteFrequency="1M" ) '''How to manage calendar type''' in config.card you have to set !CalendarType like 360d or leap or noleap ( for example : if you want 360 days you have to be careful of DateEnd=2010-12-3'''0''' [[BR]] because every month has 30 days If you put for example DateEnd=2005-12-31, you will have another run, of duration like your !PeriodLenght ) '''How to manage rdt ( i.e. timestep )''' in COMP/nemo.driver you have to change parameter : ORCA_NPDT_JOUR=15 ( 15 corresponds to 86400s/5760s; it does mean that in one day you have 15 iterations; and it does mean that timestep is 5760) '''How control all parameters in modipsl/config/ORCA2_LIM/IGCM00/PARAM/namelist''' libIGCM change namelist's values but not in namelist's file, so it is possible to control all changes of namelist's parameters looking at : {OUTPUT_DIR}/IGCM_OUT/ORCA2_LIM/N360d_5d/OCE/Debug/!JobName_date_date_ocean.output '''How to activate Monitoring''' in config.card file #D- If you want to produce time series, this flag determines [[BR]] #D- frequency of post-processing submission[[BR]] TimeSeriesFrequency=NONE you have to put your experience lenght. If you do an experience during 10 years you have to put:[[BR]] TimeSeriesFrequency=10Y '''How to activate dynamic Monitoring''' (25/02/2009) * Copy original monitoring files in SUBMIT_DIR : [[BR]] {{{ cp ~/KEEPED_FILES/monitoring01_nemo_ORCA2.cfg $SUBMIT_DIR/. cp ~/KEEPED_FILES/monitoring01_lim2_ORCA2.cfg $SUBMIT_DIR/. }}} * Copy monitoring.job in libIGCM : [[BR]] {{{ cp ~/KEEPED_FILES/monitoring.job modipsl/libIGCM/. }}} * Control TimeSeries in COMP/nemo.card and COMP/lim2.card: [[BR]] {{{ and if you add or remove TimeSeries in nemo.card or lim2.card => it is necessary to add or remove correspondig variable in $SUBMIT_DIR/monitoring01_nemo[lim2]_ORCA2.cfg }}} * How control monitoring on idris ( rhodes ) : [[BR]] {{{ Qstat -a }}} '''How to activate Atlas''' in config.card file #D- If you want to produce seasonal average, this flag determines [[BR]] #D- the period of this average [[BR]] SeasonalFrequency=NONE you have to put you experience lenght. If you do an experience during 10 years you have to put:[[BR]] SeasonalFrequency=10Y '''How to manage frequency of post processing''' ( Note: in this moment is not possible to have atlas and monitoring of frequency different from '''1M''', if we want this we've to see it with Patrick Brockman!! ) in COMP/nemo.card you have to change in [OutputFiles] chapter: Post_'''1M'''_grid_T (U,V,W) and you have to change [Post_'''1M'''_grid_T(U,V,W)] chapter. and in in COMP/lim2.card you have to change in [OutputFiles] chapiter: List= (${PREFIX_NWRITE}_${DATE_OPA}_icemod.nc, ${R_OUT_ICE_NWRITE}/${PREFIX}_${WF1}_icemod.nc, Post_'''1M'''_icemod) and you have to change [Post_'''1M'''_icemod] chapter. * '''How to make a parallel run ''' first of all add a cpp key: http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-add-modify-new-modules-or-new-cpp-keys and then see: http://www.nemo-ocean.eu/index.php//Using-NEMO/User-Guides/Advanced/How-to-set-up-one-simulation#eztoc1210_5 * '''How to have mail of end of Job, sur ccrt's machines''' on mercure: in Job_Name: #PBS -M name@mail on platine: In Job_Name: #BSUB -u name@mail * '''How to activate particular key cpp ''' (if not for example we have this error: WARNING : IGCM_sys_Put_Out xxx_trends.nc DOES NOT EXIST [[BR]] WARNING : IGCM_sys_Put_Out xxx_diagap.nc DOES NOT EXIST [[BR]] WARNING : IGCM_sys_Put_Out output.abort.nc DOES NOT EXIST ) [[BR]] [[BR]] == Run only post processing: == [[BR]] * '''If you have already run a job and you need only to run post processing''' you can do it in 3 ways: see: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ in particular: http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelafrontale and http://forge.ipsl.jussieu.fr/libigcm/wiki/libIGCM/DocUtilisateur/FAQ#Commentrelancerlespost-traitements%C3%A0partirdelamachinedecalcul == Example of post processing. a Job already run for 10 years form 2001 to 2010 : == === in '''run.card''' === * modify '''!OldPrefix''' {{{ OldPrefix=''Job_!Name''_(aa-1)mmdd }}} where (aa-1)mmdd is last day of last month of one year before you simulation's start * modify '''!CumulPeriod''' {{{ !CumulPeriod=10 }}} is 10 years that you've already run * modify '''!PeriodState''' {{{ !PeriodState=!OnQueue }}} * modify '''!TimeSeriesRunnging''' {{{ !TimeSeriesRunnging=n }}} no because Time series are depending for each other, so if you want to start a time series it starts ONLY when a previous one is finished, so if you put !TimeSeriesRunnging=y (i.e. yes) it can't never start * modify '''!TimeSeriesCompleted''' {{{ !TimeSeriesCompleted= }}} nothing because we want to do time series for all 10 years ( If not you have to put last time series already realised ) === in '''Job_Name''': === * modify '''cputim_job''' {{{ #PBS -l cputim_job=4:00:00 # Limite temps }}} * modify '''DRYRUN''' {{{ DRYRUN=3 }}} * modify '''RUN_DIR_PATH''' {{{ Create, if necessary, a test directory (for example in you workdir) Decomment RUN_DIR_PATH=/$workdir/TESTPOST }}} * modify '''SUBMIT_DIR''' {{{ Decomment and put right submit directory (the one in which you are changed you run.card and you Job) SUBMIT_DIR=/$workdir/modipsl/config/$CONFIG_NAME/EXP00 }}} [[BR]] [[BR]] == List of possible errors in modipsl/config/ORCA2_LIM/IGCM00/Script_Output_''!JobName'' : == [[BR]] * '''Error : TERM = (vt100) ${HOME:-.}/.profile: .bashrc: not found''' * '''Error : ksh: D-----------------------------------==: not found''' it can depends on '''.profile''' file. To solve it:: .profile file it can be like: «# .profile (idris) [[BR]] # Fichier d environnement a charger sur la frontale TX7 unix LINUX [[BR]] # Vous pouvez mettre a jour le fichier .profile_tx [[BR]] if [ !`hostname!` = "brodie" -a -f $HOME/.profile_tx ]; then [[BR]] . $HOME/.profile_tx [[BR]] # Fichier d environnement a charger sur les noeuds SX8 unix SUPER-UX [[BR]] # Vous pouvez mettre a jour le fichier .profile_sx [[BR]] elif [ !`hostname!` != "brodie" -a -f $HOME/.profile_sx ]; then [[BR]] . $HOME/.profile_sx [[BR]] else [[BR]] echo [[BR]] fi » * '''Error : cat: cannot open ........./modipsl/config/ORCA2_LIM/IGCM00/../.resol ''' : it is due to modipsl/libIGCM/AA_job file. To solve it:: change in this way: #-- Resolution FLAG for oasis input [[BR]] #RESOL=$(cat ${SUBMIT_DIR}/../.resol) [[BR]] RESOL=ORCA2 '''NOTE: Be careful because this is ok for ORCA2, but if we use ORCA0.5 ore ORCA0.25 is NOT TRUE!!!!! ''' go in modipsl/config/ORCA2_LIM/IGCM00 directory cancel '''run.card.init''' and '''Job_''Name''' files and then do an '''ins_job''' again. (does not still fixed in libIGCM_v1_'''2''') * '''Error : Input files and BC for NEMO DO NOT EXIST (i.e. sst_data.nc, sss_data.nc, geothermal_heating.nc, bathy_level.nc, etc...)''' IGCM_sys_Get, ERROR : regular file ...../OCE/ORCA2_LIM/bathy_level.nc DOES NOT EXIST . To solve it:: In this moment we've still to download on gaya.idris.fr:~/!reee512/NEMO/ORCA2_LIM_nemo_v3.tar file, and to untar it into the directory: ${R_BC}/OCE/ORCA2_LIM and where ${R_BC} is a variable that can be changed in modipsl/config/ORCA2_LIM/IGCM00/config.card file. * '''Error : IGCM_debug_Exit : IGCM_config_Initialize Error !PeriodState : Fatal ''' To solve it:: change line in modipsl/config/ORCA2_LIM/IGCM00/run.card !PeriodState=Fatal [[BR]] in [[BR]] !PeriodState= or rm modipsl/config/ORCA2_LIM/IGCM00/run.card * '''ls: ERROR: Cannot access stack: No such file or directory''' in Script_Outputxxx: ( '''is it a problem with filemane'''? ) IGCM_sys_Put_Out : out_nemo /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out/N365d_5d_20030101_20031231_out_nemo [[BR]] IGCM_sys_MkdirArchive : /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N365d_5d/Out [[BR]] IGCM_sys_Chmod : 444 out_nemo [[BR]] --Debug1--> OCE [[BR]] --Debug1--> ICE [[BR]] Files that will be deleted before next period-run : [[BR]] ls: ERROR: Cannot access stack: No such file or directory [[BR]] -r--r--r-- N365d_5d_00016425_restart.nc [[BR]] -r--r--r-- N365d_5d_00016425_restart_ice.nc [[BR]] (no ok still with libIGCM_v1_'''2''') [[BR]] [[BR]] == List of possible remarks and suggestions : == [[BR]] * '''be careful at group (for example for a gensi one)''' if we use a common account on CCRT we have to be sure that everygroup can have access to it, this is important for chmod 444, do another test!!!!!! * '''Every time that we change modipsl/config/ORCA2_LIM/IGCM00/config.card''' we have to cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init cancel modipsl/config/ORCA2_LIM/IGCM00/Job_''Name'' and run ./modipsl/util/ns_job script * '''To perform a second simulation after a run with !PeriodState=Fatal''' cancel modipsl/config/ORCA2_LIM/IGCM00/stack_error, modipsl/config/ORCA2_LIM/IGCM00/Script_Output_''!JobName'' but do not cancel modipsl/config/ORCA2_LIM/IGCM00/run.card.init file, if not run.card file is not re-created. ( File run.card.init is created with ins_job script ) * '''Does it exists a caption file for run.card.init options?''' Or Is there a reference file?[[BR]] In to run.card.init file who can we undesrtand syntax of? : !TimeSeriesRunning=n [[BR]] !TimeSeriesCompleted= [[BR]] !SeasonalRunning=n [[BR]] '''!RebuildFrequency : how much has it? = !PeriodLenght???''' * '''test tags/libIGCM_v1_2''' Why defaul of config.card is '''WriteFrequency="5D"''' instead of 1M? We've seen that with 1M monitoring and atlas are not possible. * '''test tags/libIGCM_v1_2''' Is it still true?: To use libIGCM trunk version it is necessary to add in config.card in POST Session: !RebuildFromArchive=NONE * '''Change in nemo.card or in nemo.driver variable: ORCA_NPDT_JOUR and OPA_NPDT_JOURS''' [[BR]] [[BR]] == Done : == [[BR]] * '''run on CCRT machines ( mercure ) and at IDRIS ( brodie )''' Tests for Calendar Type, Output frequency and Restarts: [[BR]] '''Simulation environment''' Simulation duration = '''10 years''' !PeriodLength = '''1Y''' timestep: rdt = '''5760''' equivalent to ORCA_NPDT_JOUR = '''15''' || || || '''Output Frequency''' || '''Output Frequency''' || '''Output Frequency''' || || || || 5d || 1M || 1Y || || '''Calendar Type '''|| 360d || '''ok''' || '''ok''' || '''ok''' || || '''Calendar Type '''|| 365d || '''ok''' || '''ko''' || '''ok''' || || '''Calendar Type '''|| leap || '''ok''' || '''ko''' || to do || for noleap year we can do now ( mars 2009 ) !PeriodLenght=1M and !WriteFrequency=1M ko = there are only 11 output ( point how NEMO do averadge for every month ) '''NOTE: control in:''' - why in /dmnfs13/cont003/flavoni/IGCM_OUT/ORCA2_LIM/N360d_1m/OCE/Debug/*_ocean.output Namelist namrun job number no = '''0''' is EVER = 0???? '''N365d_1m_20011231_restart.nc''' Control if file name is correct!!!! (my job is N365d_1m) * Test result :[[BR]] http://dods.extra.cea.fr/data/flavoni/ORCA2_LIM/ [[BR]] ||'''Job Name''' || '''versions''' || '''calendar type''' || || ONLY || nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 [[BR]] option only(x) in diawri.F90 || 360d || || ONLYio4 || nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 || 360d || || dynamic || nemo_v3, ioipsl_v2_1_4, libIGCM_v1_2 [[BR]] with monitorin01_nemo_ORCA2.cfg and monitornig01_lim2_ORCA2.cfg in SUBMIT_DIR: monitoring dynamic || 360d || || io7nemo3|| nemo_v3, ioipsl_v2_1_7, libIGCM trunk|| 360d || || v3360d || nemo_v3, ioipsl_v2_1_8, libIGCM_v1_2|| 360d || || v31bok || nemo_v3_1_beta, ioipsl_v2_1_8, libIGCM_v1_2|| 360d || * '''monitoring''' : done on mercure/brodie for experience=10Y, !PeriodLenght=1Y, !WriteFrequency=1M 'ok': monitoring dynamic ( reading nemo.card => doing monitoring ) '''NOTE: control mesure unity for monitoring_nemo_ORCA2.cfg for soicecov ( ![0,1]: is it ok? )''' * '''atlas''' : ok on mercure/brodie for experience=10Y, !PeriodLenght=1Y, !WriteFrequency=1M NOTE: changed COMP/nemo.card and set OutputFiles and Post_''1M''_grid_T[U,V,W] [[BR]] NOTE: changed COMP/lim2.card and set OutputFiles and Post_''1M''_grid_T[U,V,W] [[BR]] [[BR]] [[BR]] == To do list : == [[BR]] * simulation=100Y ( ongoing with tags/libIGCM_v1_2 ) [[BR]] periodlenght=10y[[BR]] and sortiee=1Y[[BR]] * test mpi: ongoing on mercure point out: all parameters in config.card : [[BR]] is it necessary to change [[BR]] #JobRunOptions='"-np "${NUM_PROC}"' [[BR]] to JobRunOptions='"-np "${BATCH_NUM_PROC_TOT}"' ? '''if yes commit it''' [[BR]] [[BR]] == NOTE files to change (or changed) for NEMO : == [[BR]] * for 360d CalendarType changed: config/ORCA2_LIM/IGCM00/COMP/'''nemo.driver''' [[BR]] and modipsl/modeles/NEMO/WORK/'''domain.F90''' * modipsl/modeles/UTIL/'''fait_AA_make''' difference with the old one: in #- Compilation options --- Committed: shade 10000 (for mercure) * modipsl/libIGCM/'''AA_job''' difference with the old one: [[BR]] #RESOL=$(cat ${SUBMIT_DIR}/../.resol) [[BR]] '''RESOL=ORCA2''' * modipsl/modeles/NEMO/WORK/'''domain.F90''' difference with the old one: in line 171 WRITE(numout,*) ' leap year calendar (0/1/'''30''') nleapy = ', nleapy * modipsl/config/ORCA2_LIM/IGCM00/COMP/'''nemo.driver''' difference with the old one: added nleapy variable that corresponds to nbisex on opa.driver # nleapy leap year calendar (0/1) (30 for 360d) PAT_NLEAPY=$( supergrep nleapy ${SUBMIT_DIR}/PARAM/namelist ) in lines 141-151 case ${config_UserChoices_CalendarType} in [[BR]] leap | | Gregorian)[[BR]] ORCA_NLEAPY=1;;[[BR]] noleap)[[BR]] ORCA_NLEAPY=0;;[[BR]] 360d)[[BR]] ORCA_NLEAPY=30;;[[BR]] esac[[BR]] in line 158 -e "s%${PAT_NLEAPY}% nleapy=${ORCA_NLEAPY}%" \ [[BR]] [[BR]] == Appendix A - Dods directory: == [[BR]] * '''to remove a directory from dods''' : On Mercure to CCRT DOds directory is : ${SAVE_DIR}/dods/public/${LOGIN}/${TAG_NAME} {{{ cd ${SAVE_DIR}/dods dods_rm public/${LOGIN}/${TAG_NAME}/directory }}} [[BR]] [[BR]] Simona Flavoni last update: [[Timestamp]]