source: trunk/libIGCM/AA_job @ 154

Last change on this file since 154 was 154, checked in by sdipsl, 15 years ago
  • Use generic function IGCM_sys_SendMail
  • BigCleanUp? at the end of the simulation if in production mode
  • Do the CleanUp? After the resubmission ...
  • ${RunDir?} depends on PID because of the BigCleanUp?
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 17.2 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J ::Jobname::       # nom de la requete
6#-Q- platine #BSUB -N                   # message a la fin du job
7#-Q- platine #BSUB -o Script_Output_::Jobname::.000001    # nom du fichier de sortie
8#-Q- platine #BSUB -n ::JobNumProcTot::             # reservation des processeurs pour le job
9#-Q- platine #BSUB -W 1:00           # Limite temps
10#-Q- platine BATCH_NUM_PROC_TOT=$SLURM_NPROCS
11#-Q- sx8brodie #######################
12#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
13#-Q- sx8brodie #######################
14#-Q- sx8brodie #PBS -N ::Jobname::         # nom de la requete
15#-Q- sx8brodie #PBS -m a                   # message a la fin du job
16#-Q- sx8brodie #PBS -j o                   # regroupement des sorties du job : standard error et output
17#-Q- sx8brodie #PBS -o Script_Output_::Jobname::.000001  # nom du fichier de sortie
18#-Q- sx8brodie #PBS -S /usr/bin/ksh        # shell de soumission
19#-Q- sx8brodie #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
20#-Q- sx8brodie #PBS -q ::JobClass::                # classe multi pour BATCH_NUM_PROC_TOT superieur a 2, mono sinon
21#-Q- sx8brodie #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}  # reservation des processeurs pour le job
22#-Q- sx8brodie #PBS -l memsz_job=8.0gb     # Limite memoire
23#-Q- sx8brodie #PBS -l cputim_job=1:00:00  # Limite temps
24#-Q- aix6 #!/usr/bin/ksh
25#-Q- aix6 # ######################
26#-Q- aix6 # ##   VARGAS IDRIS   ##
27#-Q- aix6 # ######################
28#-Q- aix6 # Nom de la requete
29#-Q- aix6 # @ job_name = ::Jobname::
30#-Q- aix6 # Type de travail
31#-Q- aix6 # @ job_type = parallel
32#-Q- aix6 # Fichier de sortie standard
33#-Q- aix6 # @ output = $(job_name).$(jobid)
34#-Q- aix6 # Fichier de sortie erreur (le meme)
35#-Q- aix6 # @ error =  $(job_name).$(jobid)
36#-Q- aix6 # Nombre de processus demandes
37#-Q- aix6 # @ total_tasks = ::JobNumProcTot::
38#-Q- aix6 # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
39#-Q- aix6 # Temps CPU max. par processus MPI hh:mm:ss
40#-Q- aix6 # @ wall_clock_limit = 1:00:00
41#-Q- aix6 # Memoire max. utilisee par processus
42#-Q- aix6 # @ data_limit = 3.2gb
43#-Q- aix6 # Memoire stack demandee
44#-Q- aix6 # @ stack_limit = 0.3gb,0.3gb
45#-Q- aix6 # Pas d OpenMP
46#-Q- aix6 # @ resources = ConsumableCpus(1)
47#-Q- aix6 # Fin de l entete
48#-Q- aix6 # @ queue
49#-Q- sx8mercure #!/bin/ksh
50#-Q- sx8mercure ######################
51#-Q- sx8mercure ## SX8MERCURE   CEA ##
52#-Q- sx8mercure ######################
53#-Q- sx8mercure #PBS -N ::Jobname::           # nom de la requete
54#-Q- sx8mercure #PBS -m a                     # message a la fin du job
55#-Q- sx8mercure #PBS -j o                     # regroupement des sorties du job : standard error et output
56#-Q- sx8mercure ####PBS -A gen6178
57#-Q- sx8mercure #PBS -o Script_Output_::Jobname::.000001   # nom du fichier de sortie
58#-Q- sx8mercure #PBS -S /usr/bin/ksh          # shell de soumission
59#-Q- sx8mercure #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
60#-Q- sx8mercure #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}          # reservation des processeurs pour le job
61#-Q- sx8mercure #PBS -l memsz_job=8.0gb       # limite memoire
62#-Q- sx8mercure #PBS -l elapstim_req=01:00:00 # limite en temps elapsed
63#-Q- default #!/bin/ksh
64#-Q- default ##################
65#-Q- default ## DEFAULT HOST ##
66#-Q- default ##################
67#-Q- default #For MPI use, uncomment next line :
68#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
69
70# $Date$
71# $Author$
72# $Revision$
73# IPSL (2006)
74#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
75
76#set -eu
77#set -vx
78
79date
80echo
81echo "#######################################"
82echo "#       ANOTHER GREAT SIMULATION      #"
83echo "#######################################"
84echo
85
86#D--------------------------------------------------------------------==
87#D--------------------------------------------------------------------==
88#D-
89#D-                      Job to launch IGCM models
90#D-
91#D--------------------------------------------------------------------==
92#D--------------------------------------------------------------------==
93#D-
94
95MODIPSL=::modipsl::
96libIGCM=${MODIPSL}/libIGCM
97
98#D--------------------------------------------------------------------==
99#D-                   -1. User modification place
100#D-                      - Job Verbosity
101#D-                      - PeriodNb
102#D-                      - Experience type : DEB(ug), DEV(elopment), RUN
103#D--------------------------------------------------------------------==
104
105#D- Increased verbosity (1, 2, 3)
106Verbosity=3
107#D- Number of execution in one job
108PeriodNb=1
109
110#D-
111# --------------------------------------------------------------------
112#D- ! Optionnal features /!\ Use with care !
113# --------------------------------------------------------------------
114
115#D- Low level debug : to bypass lib test checks and stack construction
116#D- Default=true
117DEBUG_debug=false
118
119#D- Set DEBUG_sys to false to disable output calls of function
120#D- Default=true
121#DEBUG_sys=false
122
123#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
124JobType=RUN
125
126#D- Define running directory
127#D- Default=${TMPDIR} ie temporary batch directory
128#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
129
130#D- Define submit directory
131#D- Default= where you launch qsub (variable from scheduler)
132#SUBMIT_DIR=$( pwd )
133
134#D- Turn in dry run mode ? (0,1,2,3)
135#D- Default=0
136#DRYRUN=3
137# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
138# -------------------------------------------------------------------------------------
139# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
140# |          |  Cp/Exe param files |            |  Chmod  |                           |
141# |          |      Qsub           |            |         |                           |
142# -------------------------------------------------------------------------------------
143# |    0     |       yes           |    yes     |  yes    |      yes                  |
144# -------------------------------------------------------------------------------------
145# |    1     |       yes           |    yes     |  yes    |      no                   |
146# -------------------------------------------------------------------------------------
147# |    2     |       yes           |    yes     |  no     |      no                   |
148# -------------------------------------------------------------------------------------
149# |    3     |       yes           |    no      |  no     |      no                   |
150# -------------------------------------------------------------------------------------
151
152#D- Define input file root directory
153#D- Default=/IGCMG/common/account/of/this/machine
154#R_IN=/u/rech/por/rpor111/DATA
155
156#D-
157#D--------------------------------------------------------------------==
158#D-                   0. System Environment
159#D-                      - Define variables for Number of processors
160#D-                      - Define MPI variables
161#D-                      - Define batch scheduler variables
162#D-                      - Source IGCM Library
163#D-                      - Get RESOLution in .resol file (temporary)
164#D--------------------------------------------------------------------==
165
166#D--------------------------------------------------------------------==
167
168. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
169       ( ${DEBUG_debug} ) && IGCM_debug_Check
170. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
171       ( ${DEBUG_debug} ) && IGCM_card_Check
172. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
173       ( ${DEBUG_debug} ) && IGCM_date_Check
174#-------
175. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
176. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
177. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
178. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
179
180#D--------------------------------------------------------------------==
181
182#-- Resolution FLAG for oasis input
183[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
184#RESOL="ORCA2xLMD9671"
185
186#D-
187#D--------------------------------------------------------------------==
188#D-           1. INITIALIZE CONFIGURATION AND THEN COMPONENTS
189#D--------------------------------------------------------------------==
190
191# --------------------------------------------------------------------==
192#D- Initialize configuration :
193#D-              - Simulation configuration
194#D-              - Simulation parameters
195#D-              - Execution parameter
196#D-              - Define input files directory
197#D-              - Read or initialize CumulPeriod
198#D-              - run.card
199#D-
200# --------------------------------------------------------------------==
201IGCM_config_Initialize
202
203# --------------------------------------------------------------------==
204#D- Define, create and cd RUN_DIR
205#D- 
206# --------------------------------------------------------------------==
207RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_TagName}/${config_UserChoices_JobName}.${$}
208IGCM_sys_MkdirWork ${RUN_DIR}
209IGCM_sys_Cd ${RUN_DIR}
210
211# --------------------------------------------------------------------==
212#D- Initialize each components taking part
213#D- into simulation configuration :
214#D-         - Define component executable
215#D-         - Define Write Frequency by component "1M" "1D" ...
216#D-         - Source ${comp}.driver
217#D-         - Define Storage Places by component
218#D-         - ${comp}_Initialize :
219#D-         - Patterns to sed
220#D-         - Variables from namelist
221#D-
222# --------------------------------------------------------------------==
223IGCM_comp_Initialize
224
225# --------------------------------------------------------------------==
226#D- Verify compatibility of period length, write frequencies, ...
227#D- Verify compatibility of rebuild choices and post-process choices
228#D- Will stop here if something is wrong
229# --------------------------------------------------------------------==
230#IGCM_config_Check
231
232#D--------------------------------------------------------------------==
233#D-                  2. ENTER THE DEEP EXECUTION LOOP
234#D-                  !!!! FASTEN YOUR SEAT BELTS !!!!
235#D-                  !!!!!      PLEASE.          !!!!
236#D-                  !! WE WILL CROSS TURBULENCES. !!
237#D--------------------------------------------------------------------==
238
239Period=1
240
241PeriodContinue=false
242if [ ${Period} -le ${PeriodNb} ]; then
243    PeriodContinue=true
244fi
245while ( ${PeriodContinue} ) ; do
246
247    echo
248    echo "Starting iteration ${Period} / ${PeriodNb}"
249
250    #D-
251    # ------------------------------------------------------------------
252    #D- COMPUTE AND DEFINE DATE STUFF.
253    # ------------------------------------------------------------------
254    IGCM_config_PeriodStart
255    #D- => ${PeriodDateBegin}
256    #D- => ${PeriodDateEnd}
257    #D- => ${CumulPeriod}
258    #D- => ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
259    #D- => ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
260    #D- => update run.card value
261   
262    #D-
263    # ------------------------------------------------------------------
264    #D- Get parameters text files updated by job (.def, namelist ...)
265    #D- READ AND USE BY GCM AT EACH EXECUTION.
266    # ------------------------------------------------------------------
267    IGCM_comp_GetInputParametersFiles
268
269    #D-
270    # ------------------------------------------------------------------
271    #D- Get initial state (Etat0, carteveg,relief...)
272    #D- NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
273    #D- AND ???_Restart=NO
274    #D- READ AND USE BY GCM FOR ONLY ONE EXECUTION.
275    # ------------------------------------------------------------------
276    IGCM_comp_GetInputInitialStateFiles
277
278    #D-
279    # ------------------------------------------------------------------
280    #D- Get Boundaries Conditions (SST, WIND[X,Y,Z], LAI ...)
281    #D- READ AND USE BY GCM AT EACH EXECUTION.
282    # ------------------------------------------------------------------
283    IGCM_comp_GetInputBoundaryFiles
284
285    #D-
286    # ------------------------------------------------------------------
287    #D- Get restart files (restartphy.nc, orca_restart.nc ...)
288    #D- READ AND USE BY GCM AT EACH EXECUTION.
289    #D-
290    #D- IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
291    #D- ONLY IF CumulPeriod= (=> NUMERO) == 1
292    #D- MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
293    #D-
294    #D- IF CumulPeriod == 1
295    #D- AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
296    #D- MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
297    #D-
298    #D- ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
299    # ------------------------------------------------------------------
300    IGCM_comp_GetInputRestartFiles
301   
302    #D-
303    # ------------------------------------------------------------------
304    #D- Test if all was right before Update
305    # ------------------------------------------------------------------
306    IGCM_debug_Verif_Exit
307
308    # ------------------------------------------------------------------
309    #D- Activate running environnement variables
310    # ------------------------------------------------------------------
311    IGCM_sys_activ_variables
312
313    #D-
314    # ------------------------------------------------------------------
315    #D- UPDATE ParametersFiles (.def, namelist ...) with
316    #D- current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
317    # ------------------------------------------------------------------
318    IGCM_comp_Update
319
320    #D-
321    # ------------------------------------------------------------------
322    #D- Test if all was right before execution
323    # ------------------------------------------------------------------
324    IGCM_debug_Verif_Exit
325
326    #D-
327    #D----------------------------------------------------------------==
328    #D-                          3. Execution
329    #D----------------------------------------------------------------==
330
331    echo
332    echo "#######################################"
333    echo "#      DIR BEFORE RUN EXECUTION       #"
334    echo "#######################################"
335    echo
336    ls -lrt
337
338    echo "========================================================================"
339    if ( [ X${BATCH_NUM_PROC_TOT} != X ] && [ "${BATCH_NUM_PROC_TOT}" -gt 1 ] ) ; then
340        MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND}
341        eval MPIRUN_OPTIONS=${config_UserChoices_JobRunOptions}
342    else
343        MPIRUN_COMMAND=${MPIRUN_COMMAND:="time "}
344        MPIRUN_OPTIONS=${MPIRUN_OPTIONS:=""}
345    fi
346
347    if [ ${DRYRUN} -le 1 ] ; then
348        REAL_DATE_INIT=$( date )
349        echo "EXECUTION of : ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name} > ${Exe_Output} 2>&1"
350        typeset RET
351        RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
352        ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name} > ${Exe_Output} 2>&1
353        RET=$?
354        RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
355        if [ ${RET} -gt 0 ] ; then
356            echo "Return code of executable :" ${RET}
357            IGCM_debug_Exit "EXECUTABLE"
358            IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/${PREFIX}_${Exe_Output}_error
359        fi
360        echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"  >> ${Exe_Output}
361        echo "libIGCM RunDateEnd=${RUN_DATE_END}" >> ${Exe_Output}
362    else
363        echo "EXECUTION of : ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name} simulated for DRYRUN = " $DRYRUN
364        if ( $DEBUG_debug ) ; then
365            echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
366        fi
367    fi
368    echo "========================================================================"
369
370    echo
371    echo "#######################################"
372    echo "#       DIR AFTER RUN EXECUTION       #"
373    echo "#######################################"
374    echo
375    ls -lrt
376
377    #D-
378    # ------------------------------------------------------------------
379    #D- Desactivate running environnement variables
380    # ------------------------------------------------------------------
381    IGCM_sys_desactiv_variables
382
383    #D-
384    #D----------------------------------------------------------------==
385    #D-                    4. Post execution operations
386    #D----------------------------------------------------------------==
387
388    #D-
389    # ------------------------------------------------------------------
390    #D- Save on archive machine netcdf outputs, restarts
391    #D- and text files of models
392    # ------------------------------------------------------------------
393    IGCM_comp_Finalize
394
395    #D-
396    # ------------------------------------------------------------------
397    #D- Test if all was right after Finalize
398    # ------------------------------------------------------------------
399    IGCM_debug_Verif_Exit
400
401    #D-
402    # ------------------------------------------------------------------
403    #D- Launch remote rebuild or/and post-treatment process if necessary
404    # ------------------------------------------------------------------
405    IGCM_post_Configure
406
407    #D-
408    # ------------------------------------------------------------------
409    #D- Save job output
410    #D- Manage executable size to save only different binary
411    #D- Write in run.card user, system and elapse time
412    #D- Check that everything went well ! No ? then we stop.
413    #D- Determine next computed period
414    # ------------------------------------------------------------------
415    IGCM_config_PeriodEnd
416
417    echo "Ending iteration ${Period}"
418    (( Period = Period + 1 ))
419
420    # End loop if date end is reached
421    if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
422        break ;
423    fi
424
425    PeriodContinue=false
426    if [ ${Period} -le ${PeriodNb} ]; then
427        PeriodContinue=true
428    fi
429done
430
431#D-
432#D--------------------------------------------------------------------==
433#D-     5. Submit next job or send mail if simulation over.
434#D--------------------------------------------------------------------==
435IGCM_config_Finalize
436
437date
Note: See TracBrowser for help on using the repository browser.