source: trunk/libIGCM/AA_job @ 1126

Last change on this file since 1126 was 1126, checked in by jgipsl, 10 years ago

Changed default queue for main job at obelix. Now use mediump which is better for parallel jobs (note both medium and mediump are possible options, both for parallel and sequential jobs).
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 20.2 KB
Line 
1#-Q- curie #!/bin/ksh
2#-Q- curie ######################
3#-Q- curie ## CURIE   TGCC/CEA ##
4#-Q- curie ######################
5#-Q- curie #MSUB -r ::Jobname::        # Job Name
6#-Q- curie #MSUB -o Script_Output_::Jobname::.000001    # standard output
7#-Q- curie #MSUB -e Script_Output_::Jobname::.000001    # error output
8#-Q- curie #MSUB -eo
9#-Q- curie #MSUB -n ::JobNumProcTot::  # Number of cores
10#-Q- curie #MSUB -T 86400              # Wall clock limit (seconds)
11#-Q- curie #MSUB -q standard           # thin nodes
12#-Q- curie #MSUB -A ::default_project::
13#-Q- curie
14#-Q- curie # Below specific options that can be activated
15#-Q- curie ##MSUB -q ivybridge         # Option for Airain
16#-Q- curie ##MSUB -N 8                 # Number of nodes (16 cores per node)
17#-Q- curie ##MSUB -x                   # exclusive node
18#-Q- curie ##MSUB -E '--cpu_bind=none'
19#-Q- curie
20#-Q- curie BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
21#-Q- curie set +x
22#-Q- ada #!/bin/ksh
23#-Q- ada # ######################
24#-Q- ada # ##  ADA       IDRIS ##
25#-Q- ada # ######################
26#-Q- ada # Job name
27#-Q- ada # @ job_name = ::Jobname::
28#-Q- ada # Job type
29#-Q- ada # @ job_type = parallel
30#-Q- ada # Standard output file name
31#-Q- ada # @ output = Script_Output_::Jobname::.000001
32#-Q- ada # Error output file name
33#-Q- ada # @ error = Script_Output_::Jobname::.000001
34#-Q- ada # Total number of tasks
35#-Q- ada # @ total_tasks = ::JobNumProcTot::
36#-Q- ada # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
37#-Q- ada # Maximum CPU time per task hh:mm:ss
38#-Q- ada # @ wall_clock_limit = 1:00:00
39#-Q- ada # Specific option for OpenMP parallelization: Number of OpenMP threads per MPI task
40#-Q- ada ### @ parallel_threads = 4
41#-Q- ada # End of the header options
42#-Q- ada # @ queue
43#-Q- lxiv8 ######################
44#-Q- lxiv8 ## OBELIX      LSCE ##
45#-Q- lxiv8 ######################
46#-Q- lxiv8 #PBS -N ::Jobname::
47#-Q- lxiv8 #PBS -m a
48#-Q- lxiv8 #PBS -j oe
49#-Q- lxiv8 #PBS -q mediump
50#-Q- lxiv8 #PBS -o Script_Output_::Jobname::.000001
51#-Q- lxiv8 #PBS -S /bin/ksh
52#-Q- lxiv8 #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
53#-Q- lxiv8 #PBS -l nodes=1:ppn=::JobNumProcTot::
54#-Q- default #!/bin/ksh
55#-Q- default ##################
56#-Q- default ## DEFAULT HOST ##
57#-Q- default ##################
58#-Q- default #For MPI use, uncomment next line :
59#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
60
61#**************************************************************
62# Author: Sebastien Denvil
63# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
64# $Revision::                                          $ Revision of last commit
65# $Author::                                            $ Author of last commit
66# $Date::                                              $ Date of last commit
67# IPSL (2006)
68#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
69#
70#**************************************************************
71
72#set -eu
73#set -vx
74
75date
76echo
77echo "#######################################"
78echo "#       ANOTHER GREAT SIMULATION      #"
79echo "#######################################"
80echo
81
82#D--------------------------------------------------------------------==
83#D-
84#D-                      Job to launch IGCM models
85#D-
86#D--------------------------------------------------------------------==
87#D-
88
89MODIPSL=::modipsl::
90libIGCM=${MODIPSL}/libIGCM
91
92#D--------------------------------------------------------------------==
93#D- -1. PLACE FOR USER MODIFICATION
94#D-     - Job Verbosity
95#D-     - PeriodNb
96#D-     - Experience type : DEB(ug), DEV(elopment), RUN
97#D--------------------------------------------------------------------==
98
99#D- Task type (computing or post-processing)
100TaskType=computing
101#D- Increased verbosity (1, 2, 3)
102Verbosity=3
103#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
104JobType=RUN
105#D- Number of execution in one job
106PeriodNb=1
107#-Q- sx9mercure PeriodNb=60
108
109#D-
110#D- --------------------------------------------------------------------------------------------------#
111#D-                          ! OPTIONNAL FEATURES /!\ USE WITH CARE !                                 #
112#D- --------------------------------------------------------------------------------------------------#
113
114
115#D- Low level debug : to perform lib test checks, stack construction and IO functions instrumentation
116#D- Default=true
117#D-
118DEBUG_debug=true
119
120#D- Messaging : all activities and call stacks will be sent to ipsl servers
121#D- If true will imply DEBUG_debug=true
122#D- Default=false
123#D-
124BigBrother=true
125
126#D- Set DEBUG_sys to false to disable output calls of function
127#D- true when JobType=DEB # means DEB(ug)
128#D-
129#DEBUG_sys=false
130
131#D- Define running directory
132#D- Default=${TMPDIR} ie temporary batch directory
133#D-
134#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
135
136#D- Define submit directory
137#D- Default= where you launch qsub (variable from scheduler)
138#D-
139#SUBMIT_DIR=$( pwd )
140
141#D- Define input file root directory
142#D- Default=/IGCMG/common/account/of/this/machine
143#D-
144#R_IN=/u/rech/por/rpor111/DATA
145
146#D- Turn in dry run mode ? (0,1,2,3)
147#D- Default=0
148#D-
149#DRYRUN=3
150#D-# YOU HAVE TO COMPILE YOUR EXE FILES to USE DRYRUN MODE !
151#D-# -------------------------------------------------------------------------------------
152#D-# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
153#D-# |          |  Cp/Exe/param/files |            |         |                           |
154#D-# |          |  Chmod Qsub         |            |         |                           |
155#D-# -------------------------------------------------------------------------------------
156#D-# |    0     |       yes           |    yes     |  yes    |      yes                  |
157#D-# -------------------------------------------------------------------------------------
158#D-# |    1     |       yes           |    yes     |  yes    |      no                   |
159#D-# -------------------------------------------------------------------------------------
160#D-# |    2     |       yes           |    yes     |  no     |      no                   |
161#D-# -------------------------------------------------------------------------------------
162#D-# |    3     |       yes           |    no      |  no     |      no                   |
163#D-# -------------------------------------------------------------------------------------
164
165#D-
166# --------------------------------------------------------------------------------------------------#
167#D- ! END OF OPTIONNAL FEATURES /!\ DO NOT MODIFY ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING #
168# --------------------------------------------------------------------------------------------------#
169
170#D-
171#D--------------------------------------------------------------------==
172#D- 0.0 SYSTEM ENVIRONMENT
173#D-     - Define variables for Number of processors
174#D-     - Define MPI variables
175#D-     - Define batch scheduler variables
176#D-     - Source IGCM Library
177#D-     - Get RESOLution in .resol file (temporary)
178#D--------------------------------------------------------------------==
179
180#---------------------------------------------------------------------==
181
182. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
183. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
184. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
185#-------
186. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
187. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
188. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
189. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
190#-------
191( ${DEBUG_debug} ) && IGCM_debug_Check
192( ${DEBUG_debug} ) && IGCM_card_Check
193( ${DEBUG_debug} ) && IGCM_date_Check
194
195#--------------------------------------------------------------------==
196
197#-- Resolution FLAG for oasis input
198#RESOL="ORCA2xLMD9671"
199[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
200#-- Chemistrie FLAG to ease chemistries configurations management
201#CHEM="AER"
202[ -f ${SUBMIT_DIR}/../.chimie ] && eval $(grep CHEM ${SUBMIT_DIR}/../.chimie)
203
204#D-
205#D--------------------------------------------------------------------==
206#D- 0.1 COMMON ENVIRONMENT
207#D-     - Read libIGCM compatibility version in config.card
208#D-     - Read UserChoices section
209#D-     - Read Ensemble section
210#D-     - Read Post section
211#D-     - Define all netcdf output directories
212#D--------------------------------------------------------------------==
213IGCM_config_CommonConfiguration ${SUBMIT_DIR}/config.card
214
215if [ ! -r ${SUBMIT_DIR}/run.card ] ; then
216  #================================================#
217  #         The file run.card doesn't exist        #
218  #================================================#
219  FirstInitialize=true
220  #copy initial run.card
221  IGCM_sys_Cp ${SUBMIT_DIR}/run.card.init ${SUBMIT_DIR}/run.card
222  IGCM_debug_Print 2 "run.card copied from run.card.init"
223else
224  FirstInitialize=false
225  IGCM_debug_Print 2 "run.card exists"
226fi
227
228# ------------------------------------------------------------------
229# Activate BigBrother so as to supervise this simulation
230# ------------------------------------------------------------------
231IGCM_debug_ActivateBigBro
232
233#D-
234#D--------------------------------------------------------------------==
235#D- 1. INITIALIZE CONFIGURATION
236#D-    - Simulation configuration
237#D-    - Simulation parameters
238#D-    - Execution parameter
239#D-    - Define input files directory
240#D-    - Read or initialize CumulPeriod
241#D-    - run.card
242#D--------------------------------------------------------------------==
243IGCM_config_Initialize
244
245# --------------------------------------------------------------------==
246# Define, create and cd RUN_DIR
247# --------------------------------------------------------------------==
248RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
249IGCM_sys_MkdirWork ${RUN_DIR}
250IGCM_sys_Cd ${RUN_DIR}
251
252# ------------------------------------------------------------------
253# Test if all was right before proceeding further
254# ------------------------------------------------------------------
255IGCM_debug_Verif_Exit
256
257#D-
258#D---------------------------------------------------------------------==
259#D- 2. INITIALIZE ALL COMPONENT OF THE CONFIGURATION
260#D-    - Define component executable
261#D-    - Define Write Frequency by component "1M" "1D" ...
262#D-    - Source ${comp}.driver
263#D-    - Define Storage Places by component
264#D-    - ${comp}_Initialize :
265#D-    - Patterns to sed
266#D-    - Variables from namelist
267#D---------------------------------------------------------------------==
268IGCM_comp_Initialize
269
270#D-
271#D---------------------------------------------------------------------==
272#D- 3. PERFORM GENERAL VERIFICATION
273#D-    - Verify compatibility of period length, write frequencies, ...
274#D-    - Verify compatibility of rebuild choices and post-process choices
275#D-    - Will stop here if something is wrong
276#D---------------------------------------------------------------------==
277IGCM_config_Check
278
279# ------------------------------------------------------------------
280# Test if all was right before entering the period loop
281# ------------------------------------------------------------------
282IGCM_debug_Verif_Exit
283
284#D-
285#D---------------------------------------------------------------------==
286#D- 4. ENTER THE DEEP EXECUTION LOOP
287#D-    ! FASTEN YOUR SEAT BELTS PLEASE. !
288#D-    !   WE WILL CROSS TURBULENCES.   !
289#D---------------------------------------------------------------------==
290
291Period=1
292
293PeriodContinue=false
294if [ ${Period} -le ${PeriodNb} ]; then
295  PeriodContinue=true
296fi
297while ( ${PeriodContinue} ) ; do
298
299  echo
300  echo "Starting iteration ${Period} / ${PeriodNb}"
301
302  #D-
303  # ------------------------------------------------------------------
304  #D-   4.1 COMPUTE AND DEFINE DATE INFORMATION.
305  #D-       - ${PeriodDateBegin}
306  #D-       - ${PeriodDateEnd}
307  #D-       - ${CumulPeriod}
308  #D-       - ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
309  #D-       - ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
310  #D-       - update run.card value
311  # ------------------------------------------------------------------
312  IGCM_config_PeriodStart
313
314  #D-
315  # ------------------------------------------------------------------
316  #D-   4.2 BEGIN COMPUTATION ONLY IF NEEDED
317  # ------------------------------------------------------------------
318  if [ ${SimulationLengthInDays} -gt ${ExperienceLengthInDays} ] ; then
319    IGCM_debug_Print 1 "break because '${SimulationLengthInDays} -gt ${ExperienceLengthInDays}' : ${SimulationLengthInDays} -gt ${ExperienceLengthInDays}"
320    break ;
321  fi
322
323  #D-
324  # ------------------------------------------------------------------
325  #D-   4.3 CALL OPTIONNAL FUNCTION WITHIN DRIVER
326  #D-       - to set special variables used in lists (Param, Init or Bc).
327  # ------------------------------------------------------------------
328  IGCM_comp_PeriodStart
329
330  #D-
331  # ------------------------------------------------------------------
332  #D-   4.4 GET PARAMETERS TEXT FILES UPDATED BY JOB
333  #D-       - (.def, namelist ...)
334  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
335  # ------------------------------------------------------------------
336  IGCM_comp_GetInputParametersFiles
337
338  #D-
339  # ------------------------------------------------------------------
340  #D-   4.5 GET INITIAL STATE
341  #D-       - (Etat0, carteveg,relief...)
342  #D-       - NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
343  #D-       - AND ???_Restart=NO
344  #D-       - READ AND USE BY GCM FOR ONLY ONE EXECUTION.
345  # ------------------------------------------------------------------
346  IGCM_comp_GetInputInitialStateFiles
347
348  #D-
349  # ------------------------------------------------------------------
350  #D-   4.6 GET BOUNDARIES CONDITIONS
351  #D-       - (SST, WIND[X,Y,Z], LAI ...)
352  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
353  # ------------------------------------------------------------------
354  IGCM_comp_GetInputBoundaryFiles
355
356  #D-
357  # ------------------------------------------------------------------
358  #D-   4.7 GET SmoothFiles CONDITIONS
359  #D-       - (SST, WIND[X,Y,Z], LAI ...)
360  #D-       - READ AND USE BY GCM AT EACH EXECUTION
361  #D-       - Do not change at each ${Period}
362  # ------------------------------------------------------------------
363  IGCM_comp_GetInputSmoothFiles
364
365  #D-
366  # ------------------------------------------------------------------
367  #D-   4.8 GET RESTART FILES
368  #D-       - (restartphy.nc, orca_restart.nc ...)
369  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
370  #D-
371  #D-       - IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
372  #D-       - ONLY IF CumulPeriod= (=> NUMERO) == 1
373  #D-       - MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
374  #D-
375  #D-       - IF CumulPeriod == 1
376  #D-       - AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
377  #D-       - MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
378  #D-
379  #D-       - ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
380  # ------------------------------------------------------------------
381  IGCM_comp_GetInputRestartFiles
382
383  # ------------------------------------------------------------------
384  # Test if all was right before Update
385  # ------------------------------------------------------------------
386  IGCM_debug_Verif_Exit
387
388  #D-
389  # ------------------------------------------------------------------
390  #D-   4.9 ACTIVATE RUNNING ENVIRONNEMENT VARIABLES
391  # ------------------------------------------------------------------
392  IGCM_sys_activ_variables
393
394  #D-
395  # ------------------------------------------------------------------
396  #D-   4.10 UPDATE ParametersFiles
397  #D-        - (.def, namelist ...)
398  #D-        - with current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
399  # ------------------------------------------------------------------
400  IGCM_comp_Update
401
402  # ------------------------------------------------------------------
403  # Test if all was right before execution
404  # ------------------------------------------------------------------
405  IGCM_debug_Verif_Exit
406
407  #D-
408  #D---------------------------------------------------------------------==
409  #D- 5. EXECUTION
410  #D---------------------------------------------------------------------==
411
412  echo
413  echo "#######################################"
414  echo "#      DIR BEFORE RUN EXECUTION       #"
415  echo "#######################################"
416  echo
417  ls -lrt
418
419  echo "========================================================================"
420  if [ ${DRYRUN} -le 1 ] ; then
421    REAL_DATE_INIT=$( date )
422    echo                                                                                   > ${Exe_Output}
423    echo "#######################################"                                        >> ${Exe_Output}
424    echo "EXECUTION of : ${EXECUTION}"
425    echo "EXECUTION of : ${EXECUTION}"                                                    >> ${Exe_Output}
426    echo                                                                                  >> ${Exe_Output}
427    typeset RET
428    RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
429    ${EXECUTION}  >> ${Exe_Output} 2>&1
430    RET=$?
431    RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
432    if [ ${RET} -gt 0 ] ; then
433      echo "Return code of executable :" ${RET}
434      IGCM_debug_Exit "EXECUTABLE"
435      IGCM_sys_Mkdir ${SUBMIT_DIR}/Debug
436      IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/Debug/${PREFIX}_${Exe_Output}_error
437      ExecutionFail=true
438    fi
439    echo                                            >> ${Exe_Output}
440    echo "#######################################"  >> ${Exe_Output}
441    echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
442    echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
443    echo                                            >> ${Exe_Output}
444  else
445    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN
446    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN > ${Exe_Output}
447    if ( $DEBUG_debug ) ; then
448      echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
449    fi
450  fi
451  echo "========================================================================"
452
453  echo
454  echo "#######################################"
455  echo "#       DIR AFTER RUN EXECUTION       #"
456  echo "#######################################"
457  echo
458  ls -lrt
459
460  #D-
461  #D---------------------------------------------------------------------==
462  #D- 6. POST EXECUTION OPERATIONS
463  #D---------------------------------------------------------------------==
464
465  #D-
466  # ------------------------------------------------------------------
467  #D-   6.1. DESACTIVATE RUNNING ENVIRONNEMENT VARIABLES
468  # ------------------------------------------------------------------
469  IGCM_sys_desactiv_variables
470
471  #D-
472  # ------------------------------------------------------------------
473  #D-   6.2. SAVE OUTPUTS
474  #D-        - On archive/storage machine
475  #D-        - netcdf outputs, restarts and text files of models
476  # ------------------------------------------------------------------
477  IGCM_comp_Finalize
478
479  # ------------------------------------------------------------------
480  # Test if all was right after Finalize
481  # ------------------------------------------------------------------
482  IGCM_debug_Verif_Exit
483
484  #D-
485  # ------------------------------------------------------------------
486  #D-   6.3. CONFIGURE POST-PROCESSING
487  #D-        - asynchronous post-treatment (Level 0) when necessary (rebuild/pack)
488  #D-        - classic post-treatment      (Level 1) when necessary (TimeSeries/Seasonal)
489  # ------------------------------------------------------------------
490  IGCM_post_Configure
491
492  #D-
493  # ------------------------------------------------------------------
494  #D-   6.4. SUBMIT POST-PROCESSING
495  #D-        - remote rebuild or/and post-treatment process if necessary
496  # ------------------------------------------------------------------
497  IGCM_post_Submit
498
499  #D-
500  # ------------------------------------------------------------------
501  #D-   6.5. FINALYZE CURRENT LOOP AND PREPARE NEXT ONE
502  #D-        - Manage executable size to save only different binary
503  #D-        - Write in run.card user, system and elapse time
504  #D-        - Check that everything went well ! No ? then we stop.
505  #D-        - Determine next computed period
506  # ------------------------------------------------------------------
507  IGCM_config_PeriodEnd
508
509  echo "Ending iteration ${Period}"
510  (( Period = Period + 1 ))
511
512  # End loop if date end is reached
513  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
514    break ;
515  fi
516
517  PeriodContinue=false
518  if [ ${Period} -le ${PeriodNb} ]; then
519    PeriodContinue=true
520  fi
521done
522
523#D-
524#D---------------------------------------------------------------------==
525#D- 7. SUBMIT NEXT JOB OR SEND MAIL IF SIMULATION IS OVER.
526#D---------------------------------------------------------------------==
527IGCM_config_Finalize
528
529date
Note: See TracBrowser for help on using the repository browser.