source: trunk/libIGCM/AA_job @ 1059

Last change on this file since 1059 was 1053, checked in by sdipsl, 10 years ago

bug fixes affecting only debug mode

  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 20.1 KB
Line 
1#-Q- curie #!/bin/ksh
2#-Q- curie ######################
3#-Q- curie ## CURIE   TGCC/CEA ##
4#-Q- curie ######################
5#-Q- curie #MSUB -r ::Jobname::        # Job Name
6#-Q- curie #MSUB -o Script_Output_::Jobname::.000001    # standard output
7#-Q- curie #MSUB -e Script_Output_::Jobname::.000001    # error output
8#-Q- curie #MSUB -eo
9#-Q- curie #MSUB -n ::JobNumProcTot::  # reservation des processeurs pour le job
10#-Q- curie ##MSUB -N 8                 # Number of nodes (16 cores per node)
11#-Q- curie ##MSUB -x                    # exclusive node
12#-Q- curie ##MSUB -E '--cpu_bind=none'
13#-Q- curie #MSUB -T 86400              # Wall clock limit (seconds)
14#-Q- curie #MSUB -q standard           # thin nodes
15#-Q- curie #MSUB -A ::default_project::
16#-Q- curie BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
17#-Q- curie set +x
18#-Q- ada #!/bin/ksh
19#-Q- ada # ######################
20#-Q- ada # ##  ADA       IDRIS ##
21#-Q- ada # ######################
22#-Q- ada # Nom de la requete
23#-Q- ada # @ job_name = ::Jobname::
24#-Q- ada # Type de travail
25#-Q- ada # @ job_type = parallel
26#-Q- ada # Fichier de sortie standard
27#-Q- ada # @ output = Script_Output_::Jobname::.000001
28#-Q- ada # Fichier de sortie erreur (le meme)
29#-Q- ada # @ error = Script_Output_::Jobname::.000001
30#-Q- ada # Nombre de processus demandes
31#-Q- ada # @ total_tasks = ::JobNumProcTot::
32#-Q- ada # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
33#-Q- ada # Temps CPU max. par processus MPI hh:mm:ss
34#-Q- ada # @ wall_clock_limit = 1:00:00
35#-Q- ada # Nombre de taches OpenMP/pthreads par processus MPI
36#-Q- ada ### @ parallel_threads = 4
37#-Q- ada # Fin de l entete
38#-Q- ada # @ queue
39#-Q- lxiv8 ######################
40#-Q- lxiv8 ## OBELIX      LSCE ##
41#-Q- lxiv8 ######################
42#-Q- lxiv8 #PBS -N ::Jobname::
43#-Q- lxiv8 #PBS -m a
44#-Q- lxiv8 #PBS -j oe
45#-Q- lxiv8 #PBS -q medium
46#-Q- lxiv8 #PBS -o Script_Output_::Jobname::.000001
47#-Q- lxiv8 #PBS -S /bin/ksh
48#-Q- lxiv8 #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
49#-Q- lxiv8 #PBS -l nodes=1:ppn=::JobNumProcTot::
50#-Q- default #!/bin/ksh
51#-Q- default ##################
52#-Q- default ## DEFAULT HOST ##
53#-Q- default ##################
54#-Q- default #For MPI use, uncomment next line :
55#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
56
57#**************************************************************
58# Author: Sebastien Denvil
59# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
60# $Revision::                                          $ Revision of last commit
61# $Author::                                            $ Author of last commit
62# $Date::                                              $ Date of last commit
63# IPSL (2006)
64#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
65#
66#**************************************************************
67
68#set -eu
69#set -vx
70
71date
72echo
73echo "#######################################"
74echo "#       ANOTHER GREAT SIMULATION      #"
75echo "#######################################"
76echo
77
78#D--------------------------------------------------------------------==
79#D-
80#D-                      Job to launch IGCM models
81#D-
82#D--------------------------------------------------------------------==
83#D-
84
85MODIPSL=::modipsl::
86libIGCM=${MODIPSL}/libIGCM
87
88#D--------------------------------------------------------------------==
89#D- -1. PLACE FOR USER MODIFICATION
90#D-     - Job Verbosity
91#D-     - PeriodNb
92#D-     - Experience type : DEB(ug), DEV(elopment), RUN
93#D--------------------------------------------------------------------==
94
95#D- Task type (computing or post-processing)
96TaskType=computing
97#D- Increased verbosity (1, 2, 3)
98Verbosity=3
99#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
100JobType=RUN
101#D- Number of execution in one job
102PeriodNb=1
103#-Q- sx9mercure PeriodNb=60
104
105#D-
106#D- --------------------------------------------------------------------------------------------------#
107#D-                          ! OPTIONNAL FEATURES /!\ USE WITH CARE !                                 #
108#D- --------------------------------------------------------------------------------------------------#
109
110
111#D- Low level debug : to bypass lib test checks and stack construction
112#D- Default=true
113#D-
114DEBUG_debug=false
115
116#D- Messaging : all activities and call stacks will be sent to ipsl servers
117#D- If true will imply DEBUG_debug=true
118#D- Default=false
119#D-
120BigBrother=false
121
122#D- Set DEBUG_sys to false to disable output calls of function
123#D- true when JobType=DEB # means DEB(ug)
124#D-
125#DEBUG_sys=false
126
127#D- Define running directory
128#D- Default=${TMPDIR} ie temporary batch directory
129#D-
130#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
131
132#D- Define submit directory
133#D- Default= where you launch qsub (variable from scheduler)
134#D-
135#SUBMIT_DIR=$( pwd )
136
137#D- Define input file root directory
138#D- Default=/IGCMG/common/account/of/this/machine
139#D-
140#R_IN=/u/rech/por/rpor111/DATA
141
142#D- Turn in dry run mode ? (0,1,2,3)
143#D- Default=0
144#D-
145#DRYRUN=3
146#D-# YOU HAVE TO COMPILE YOUR EXE FILES to USE DRYRUN MODE !
147#D-# -------------------------------------------------------------------------------------
148#D-# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
149#D-# |          |  Cp/Exe/param/files |            |         |                           |
150#D-# |          |  Chmod Qsub         |            |         |                           |
151#D-# -------------------------------------------------------------------------------------
152#D-# |    0     |       yes           |    yes     |  yes    |      yes                  |
153#D-# -------------------------------------------------------------------------------------
154#D-# |    1     |       yes           |    yes     |  yes    |      no                   |
155#D-# -------------------------------------------------------------------------------------
156#D-# |    2     |       yes           |    yes     |  no     |      no                   |
157#D-# -------------------------------------------------------------------------------------
158#D-# |    3     |       yes           |    no      |  no     |      no                   |
159#D-# -------------------------------------------------------------------------------------
160
161#D-
162# --------------------------------------------------------------------------------------------------#
163#D- ! END OF OPTIONNAL FEATURES /!\ DO NOT MODIFY ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING #
164# --------------------------------------------------------------------------------------------------#
165
166#D-
167#D--------------------------------------------------------------------==
168#D- 0.0 SYSTEM ENVIRONMENT
169#D-     - Define variables for Number of processors
170#D-     - Define MPI variables
171#D-     - Define batch scheduler variables
172#D-     - Source IGCM Library
173#D-     - Get RESOLution in .resol file (temporary)
174#D--------------------------------------------------------------------==
175
176#---------------------------------------------------------------------==
177
178. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
179. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
180. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
181#-------
182. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
183. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
184. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
185. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
186#-------
187( ${DEBUG_debug} ) && IGCM_debug_Check
188( ${DEBUG_debug} ) && IGCM_card_Check
189( ${DEBUG_debug} ) && IGCM_date_Check
190
191#--------------------------------------------------------------------==
192
193#-- Resolution FLAG for oasis input
194#RESOL="ORCA2xLMD9671"
195[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
196#-- Chemistrie FLAG to ease chemistries configurations management
197#CHEM="AER"
198[ -f ${SUBMIT_DIR}/../.chimie ] && eval $(grep CHEM ${SUBMIT_DIR}/../.chimie)
199
200#D-
201#D--------------------------------------------------------------------==
202#D- 0.1 COMMON ENVIRONMENT
203#D-     - Read libIGCM compatibility version in config.card
204#D-     - Read UserChoices section
205#D-     - Read Ensemble section
206#D-     - Read Post section
207#D-     - Define all netcdf output directories
208#D--------------------------------------------------------------------==
209IGCM_config_CommonConfiguration ${SUBMIT_DIR}/config.card
210
211if [ ! -r ${SUBMIT_DIR}/run.card ] ; then
212  #================================================#
213  #         The file run.card doesn't exist        #
214  #================================================#
215  FirstInitialize=true
216  #copy initial run.card
217  IGCM_sys_Cp ${SUBMIT_DIR}/run.card.init ${SUBMIT_DIR}/run.card
218  IGCM_debug_Print 2 "run.card copied from run.card.init"
219else
220  FirstInitialize=false
221  IGCM_debug_Print 2 "run.card exists"
222fi
223
224# ------------------------------------------------------------------
225# Activate BigBrother so as to supervise this simulation
226# ------------------------------------------------------------------
227IGCM_debug_ActivateBigBro
228
229#D-
230#D--------------------------------------------------------------------==
231#D- 1. INITIALIZE CONFIGURATION
232#D-    - Simulation configuration
233#D-    - Simulation parameters
234#D-    - Execution parameter
235#D-    - Define input files directory
236#D-    - Read or initialize CumulPeriod
237#D-    - run.card
238#D--------------------------------------------------------------------==
239IGCM_config_Initialize
240
241# --------------------------------------------------------------------==
242# Define, create and cd RUN_DIR
243# --------------------------------------------------------------------==
244RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
245IGCM_sys_MkdirWork ${RUN_DIR}
246IGCM_sys_Cd ${RUN_DIR}
247
248# ------------------------------------------------------------------
249# Test if all was right before proceeding further
250# ------------------------------------------------------------------
251IGCM_debug_Verif_Exit
252
253#D-
254#D---------------------------------------------------------------------==
255#D- 2. INITIALIZE ALL COMPONENT OF THE CONFIGURATION
256#D-    - Define component executable
257#D-    - Define Write Frequency by component "1M" "1D" ...
258#D-    - Source ${comp}.driver
259#D-    - Define Storage Places by component
260#D-    - ${comp}_Initialize :
261#D-    - Patterns to sed
262#D-    - Variables from namelist
263#D---------------------------------------------------------------------==
264IGCM_comp_Initialize
265
266#D-
267#D---------------------------------------------------------------------==
268#D- 3. PERFORM GENERAL VERIFICATION
269#D-    - Verify compatibility of period length, write frequencies, ...
270#D-    - Verify compatibility of rebuild choices and post-process choices
271#D-    - Will stop here if something is wrong
272#D---------------------------------------------------------------------==
273IGCM_config_Check
274
275# ------------------------------------------------------------------
276# Test if all was right before entering the period loop
277# ------------------------------------------------------------------
278IGCM_debug_Verif_Exit
279
280#D-
281#D---------------------------------------------------------------------==
282#D- 4. ENTER THE DEEP EXECUTION LOOP
283#D-    ! FASTEN YOUR SEAT BELTS PLEASE. !
284#D-    !   WE WILL CROSS TURBULENCES.   !
285#D---------------------------------------------------------------------==
286
287Period=1
288
289PeriodContinue=false
290if [ ${Period} -le ${PeriodNb} ]; then
291  PeriodContinue=true
292fi
293while ( ${PeriodContinue} ) ; do
294
295  echo
296  echo "Starting iteration ${Period} / ${PeriodNb}"
297
298  #D-
299  # ------------------------------------------------------------------
300  #D-   4.1 COMPUTE AND DEFINE DATE INFORMATION.
301  #D-       - ${PeriodDateBegin}
302  #D-       - ${PeriodDateEnd}
303  #D-       - ${CumulPeriod}
304  #D-       - ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
305  #D-       - ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
306  #D-       - update run.card value
307  # ------------------------------------------------------------------
308  IGCM_config_PeriodStart
309
310  #D-
311  # ------------------------------------------------------------------
312  #D-   4.2 BEGIN COMPUTATION ONLY IF NEEDED
313  # ------------------------------------------------------------------
314  if [ ${SimulationLengthInDays} -gt ${ExperienceLengthInDays} ] ; then
315    IGCM_debug_Print 1 "break because '${SimulationLengthInDays} -gt ${ExperienceLengthInDays}' : ${SimulationLengthInDays} -gt ${ExperienceLengthInDays}"
316    break ;
317  fi
318
319  #D-
320  # ------------------------------------------------------------------
321  #D-   4.3 CALL OPTIONNAL FUNCTION WITHIN DRIVER
322  #D-       - to set special variables used in lists (Param, Init or Bc).
323  # ------------------------------------------------------------------
324  IGCM_comp_PeriodStart
325
326  #D-
327  # ------------------------------------------------------------------
328  #D-   4.4 GET PARAMETERS TEXT FILES UPDATED BY JOB
329  #D-       - (.def, namelist ...)
330  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
331  # ------------------------------------------------------------------
332  IGCM_comp_GetInputParametersFiles
333
334  #D-
335  # ------------------------------------------------------------------
336  #D-   4.5 GET INITIAL STATE
337  #D-       - (Etat0, carteveg,relief...)
338  #D-       - NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
339  #D-       - AND ???_Restart=NO
340  #D-       - READ AND USE BY GCM FOR ONLY ONE EXECUTION.
341  # ------------------------------------------------------------------
342  IGCM_comp_GetInputInitialStateFiles
343
344  #D-
345  # ------------------------------------------------------------------
346  #D-   4.6 GET BOUNDARIES CONDITIONS
347  #D-       - (SST, WIND[X,Y,Z], LAI ...)
348  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
349  # ------------------------------------------------------------------
350  IGCM_comp_GetInputBoundaryFiles
351
352  #D-
353  # ------------------------------------------------------------------
354  #D-   4.7 GET SmoothFiles CONDITIONS
355  #D-       - (SST, WIND[X,Y,Z], LAI ...)
356  #D-       - READ AND USE BY GCM AT EACH EXECUTION
357  #D-       - Do not change at each ${Period}
358  # ------------------------------------------------------------------
359  IGCM_comp_GetInputSmoothFiles
360
361  #D-
362  # ------------------------------------------------------------------
363  #D-   4.8 GET RESTART FILES
364  #D-       - (restartphy.nc, orca_restart.nc ...)
365  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
366  #D-
367  #D-       - IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
368  #D-       - ONLY IF CumulPeriod= (=> NUMERO) == 1
369  #D-       - MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
370  #D-
371  #D-       - IF CumulPeriod == 1
372  #D-       - AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
373  #D-       - MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
374  #D-
375  #D-       - ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
376  # ------------------------------------------------------------------
377  IGCM_comp_GetInputRestartFiles
378
379  # ------------------------------------------------------------------
380  # Test if all was right before Update
381  # ------------------------------------------------------------------
382  IGCM_debug_Verif_Exit
383
384  #D-
385  # ------------------------------------------------------------------
386  #D-   4.9 ACTIVATE RUNNING ENVIRONNEMENT VARIABLES
387  # ------------------------------------------------------------------
388  IGCM_sys_activ_variables
389
390  #D-
391  # ------------------------------------------------------------------
392  #D-   4.10 UPDATE ParametersFiles
393  #D-        - (.def, namelist ...)
394  #D-        - with current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
395  # ------------------------------------------------------------------
396  IGCM_comp_Update
397
398  # ------------------------------------------------------------------
399  # Test if all was right before execution
400  # ------------------------------------------------------------------
401  IGCM_debug_Verif_Exit
402
403  #D-
404  #D---------------------------------------------------------------------==
405  #D- 5. EXECUTION
406  #D---------------------------------------------------------------------==
407
408  echo
409  echo "#######################################"
410  echo "#      DIR BEFORE RUN EXECUTION       #"
411  echo "#######################################"
412  echo
413  ls -lrt
414
415  echo "========================================================================"
416  if [ ${DRYRUN} -le 1 ] ; then
417    REAL_DATE_INIT=$( date )
418    echo                                                                                   > ${Exe_Output}
419    echo "#######################################"                                        >> ${Exe_Output}
420    echo "EXECUTION of : ${EXECUTION}"
421    echo "EXECUTION of : ${EXECUTION}"                                                    >> ${Exe_Output}
422    echo                                                                                  >> ${Exe_Output}
423    typeset RET
424    RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
425    ${EXECUTION}  >> ${Exe_Output} 2>&1
426    RET=$?
427    RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
428    if [ ${RET} -gt 0 ] ; then
429      echo "Return code of executable :" ${RET}
430      IGCM_debug_Exit "EXECUTABLE"
431      IGCM_sys_Mkdir ${SUBMIT_DIR}/Debug
432      IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/Debug/${PREFIX}_${Exe_Output}_error
433      ExecutionFail=true
434    fi
435    echo                                            >> ${Exe_Output}
436    echo "#######################################"  >> ${Exe_Output}
437    echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
438    echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
439    echo                                            >> ${Exe_Output}
440  else
441    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN
442    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN > ${Exe_Output}
443    if ( $DEBUG_debug ) ; then
444      echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
445    fi
446  fi
447  echo "========================================================================"
448
449  echo
450  echo "#######################################"
451  echo "#       DIR AFTER RUN EXECUTION       #"
452  echo "#######################################"
453  echo
454  ls -lrt
455
456  #D-
457  #D---------------------------------------------------------------------==
458  #D- 6. POST EXECUTION OPERATIONS
459  #D---------------------------------------------------------------------==
460
461  #D-
462  # ------------------------------------------------------------------
463  #D-   6.1. DESACTIVATE RUNNING ENVIRONNEMENT VARIABLES
464  # ------------------------------------------------------------------
465  IGCM_sys_desactiv_variables
466
467  #D-
468  # ------------------------------------------------------------------
469  #D-   6.2. SAVE OUTPUTS
470  #D-        - On archive/storage machine
471  #D-        - netcdf outputs, restarts and text files of models
472  # ------------------------------------------------------------------
473  IGCM_comp_Finalize
474
475  # ------------------------------------------------------------------
476  # Test if all was right after Finalize
477  # ------------------------------------------------------------------
478  IGCM_debug_Verif_Exit
479
480  #D-
481  # ------------------------------------------------------------------
482  #D-   6.3. CONFIGURE POST-PROCESSING
483  #D-        - asynchronous post-treatment (Level 0) when necessary (rebuild/pack)
484  #D-        - classic post-treatment      (Level 1) when necessary (TimeSeries/Seasonal)
485  # ------------------------------------------------------------------
486  IGCM_post_Configure
487
488  #D-
489  # ------------------------------------------------------------------
490  #D-   6.4. SUBMIT POST-PROCESSING
491  #D-        - remote rebuild or/and post-treatment process if necessary
492  # ------------------------------------------------------------------
493  IGCM_post_Submit
494
495  #D-
496  # ------------------------------------------------------------------
497  #D-   6.5. FINALYZE CURRENT LOOP AND PREPARE NEXT ONE
498  #D-        - Manage executable size to save only different binary
499  #D-        - Write in run.card user, system and elapse time
500  #D-        - Check that everything went well ! No ? then we stop.
501  #D-        - Determine next computed period
502  # ------------------------------------------------------------------
503  IGCM_config_PeriodEnd
504
505  echo "Ending iteration ${Period}"
506  (( Period = Period + 1 ))
507
508  # End loop if date end is reached
509  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
510    break ;
511  fi
512
513  PeriodContinue=false
514  if [ ${Period} -le ${PeriodNb} ]; then
515    PeriodContinue=true
516  fi
517done
518
519#D-
520#D---------------------------------------------------------------------==
521#D- 7. SUBMIT NEXT JOB OR SEND MAIL IF SIMULATION IS OVER.
522#D---------------------------------------------------------------------==
523IGCM_config_Finalize
524
525date
Note: See TracBrowser for help on using the repository browser.