source: trunk/libIGCM/AA_pack_debug

Last change on this file was 1612, checked in by aclsce, 4 months ago

Needed to define Script_Post_Output variable in stand alone mode in order to handle exit in case of error
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 12.6 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r PACKDEBUG      # Nom du job
5#-Q- curie #MSUB -o PACKDEBUG.out_%I
6#-Q- curie #MSUB -e PACKDEBUG.out_%I
7#-Q- curie #MSUB -n 1              # Reservation du processus
8#-Q- curie #MSUB -T 3600          # Limite de temps elapsed du job
9#-Q- curie #MSUB -q ::default_node::
10#-Q- curie #MSUB -c ::default_core::
11#-Q- curie #MSUB -Q normal
12#-Q- curie #MSUB -A ::default_project::
13#-Q- curie set +x
14#-Q- irene ######################
15#-Q- irene ## IRENE   TGCC/CEA ##
16#-Q- irene ######################
17#-Q- irene #MSUB -r PACKDEBUG     # Job name
18#-Q- irene #MSUB -o PACKDEBUG.out_%I
19#-Q- irene #MSUB -e PACKDEBUG.out_%I
20#-Q- irene #MSUB -n 1             # Number of cores
21#-Q- irene #MSUB -T 3600          # Maximum elapsed time
22#-Q- irene #MSUB -q ::default_node::
23#-Q- irene #MSUB -c ::default_core::
24#-Q- irene #MSUB -Q normal
25#-Q- irene #MSUB -A ::default_post_project::
26#-Q- irene #MSUB -m store,work,scratch
27#-Q- irene set +x
28#-Q- irene-amd ######################
29#-Q- irene-amd ## IRENE   TGCC/CEA ##
30#-Q- irene-amd ######################
31#-Q- irene-amd #MSUB -r PACKDEBUG     # Job name
32#-Q- irene-amd #MSUB -o PACKDEBUG.out_%I
33#-Q- irene-amd #MSUB -e PACKDEBUG.out_%I
34#-Q- irene-amd #MSUB -n 1              # Number of cores
35#-Q- irene-amd #MSUB -T 3600          # Maximum elapsed time
36#-Q- irene-amd #MSUB -q ::default_node::
37#-Q- irene-amd #MSUB -c ::default_core::
38#-Q- irene-amd #MSUB -Q normal
39#-Q- irene-amd #MSUB -A ::default_post_project::
40#-Q- irene-amd #MSUB -m store,work,scratch
41#-Q- irene-amd set +x
42#-Q- jeanzay #!/bin/ksh
43#-Q- jeanzay ######################
44#-Q- jeanzay ## JEANZAY    IDRIS ##
45#-Q- jeanzay ######################
46#-Q- jeanzay #SBATCH --job-name=PACKDEBUG         # Job Name
47#-Q- jeanzay #SBATCH --output=PACKDEBUG.out_%J    # standard output
48#-Q- jeanzay #SBATCH --error=PACKDEBUG.out_%J     # error output
49#-Q- jeanzay #SBATCH -N  1                        # Number of core
50#-Q- jeanzay #SBATCH --partition=prepost          # Post-processing partition
51#-Q- jeanzay #SBATCH --time=10:00:00              # Wall clock limit (seconds)
52#-Q- jeanzay #SBATCH --account ::default_project::@cpu
53#-Q- jeanzay set +x
54#-Q- ada #!/bin/ksh
55#-Q- ada #######################
56#-Q- ada ## ADA         IDRIS ##
57#-Q- ada #######################
58#-Q- ada # @ job_type = mpich
59#-Q- ada # @ requirements = (Feature == "prepost")
60#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
61#-Q- ada # @ wall_clock_limit = 10:00:00
62#-Q- ada # Nom du travail LoadLeveler
63#-Q- ada # @ job_name   = PACKDEBUG
64#-Q- ada # Fichier de sortie standard du travail
65#-Q- ada # @ output     = $(job_name).$(jobid)
66#-Q- ada # Fichier de sortie d'erreur du travail
67#-Q- ada # @ error      =  $(job_name).$(jobid)
68#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
69#-Q- ada # @ notification = error
70#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $POST_DIR ; $Script_Post_Output ; $SUBMIT_DIR ; $DateBegin ; $DateEnd ; $PeriodPack ; $StandAlone ; $MASTER ; wall_clock_limit=$(wall_clock_limit)
71#-Q- ada # @ queue
72#-Q- obelix ######################
73#-Q- obelix ## OBELIX      LSCE ##
74#-Q- obelix ######################
75#-Q- obelix #PBS -N PACKDEBUG
76#-Q- obelix #PBS -m a
77#-Q- obelix #PBS -j oe
78#-Q- obelix #PBS -q medium
79#-Q- obelix #PBS -o PACKDEBUG.$$
80#-Q- obelix #PBS -S /bin/ksh
81#-Q- ifort_CICLAD ######################
82#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
83#-Q- ifort_CICLAD ######################
84#-Q- ifort_CICLAD #PBS -N PACKDEBUG
85#-Q- ifort_CICLAD #PBS -m a
86#-Q- ifort_CICLAD #PBS -j oe
87#-Q- ifort_CICLAD #PBS -q std
88#-Q- ifort_CICLAD #PBS -S /bin/ksh
89#-Q- default #!/bin/ksh
90#-Q- default ##################
91#-Q- default ## DEFAULT HOST ##
92#-Q- default ##################
93
94#**************************************************************
95# Author: Sebastien Denvil
96# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
97# $Revision::                                          $ Revision of last commit
98# $Author::                                            $ Author of last commit
99# $Date::                                              $ Date of last commit
100# IPSL (2006)
101#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
102#
103#**************************************************************
104
105#set -eu
106#set -vx
107
108date
109
110#D- Task type DO NOT CHANGE (computing, post-processing or checking)
111TaskType=post-processing
112
113########################################################################
114
115#D- Flag to determine if this job in a standalone mode
116#D- Default : value from AA_job if any
117StandAlone=${StandAlone:=true}
118
119#D- Path to libIGCM
120#D- Default : value from AA_job if any
121# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
122# WARNING you must check MirrorlibIGCM variable in sys library.
123# WARNING If this variable is true, you must use libIGCM_POST path instead
124# WARNING of your running libIGCM directory.
125libIGCM=${libIGCM:=::modipsl::/libIGCM}
126
127#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
128MASTER=${MASTER:=ada|curie}
129
130#D- Flag to determine begin date for restart pack
131#D- Default : value from AA_job if any
132DateBegin=${DateBegin:=20000101}
133
134#D- Flag to determine end date for restart pack
135#D- Default : value from AA_job if any
136DateEnd=${DateEnd:=20691231}
137
138#D- Flag to determine pack period
139#D- Default : value from AA_job if any
140PeriodPack=${PeriodPack:=10Y}
141
142#D- Uncomment to run interactively
143#D- For testing purpose, will be remove
144#SUBMIT_DIR=${PWD}
145#RUN_DIR_PATH=${SCRATCHDIR}/Pack_Test
146
147#D- Increased verbosity (1, 2, 3)
148#D- Default : value from AA_job if any
149Verbosity=${Verbosity:=3}
150
151#D- Low level debug : to bypass lib test checks and stack construction
152#D- Default : value from AA_job if any
153DEBUG_debug=${DEBUG_debug:=false}
154
155#D- Needed to define this variable in order to handle exit in case of error
156Script_Post_Output=${Script_Post_Output:=pack_debug}
157
158#D- Keep following comments for now. Will be remove.
159
160#on veut archiver les fichiers restart
161#Dans le cadre du demenagement ils seront renommes
162#ATM/Restart/****.nc
163#OCE/Restart/****.nc
164#...
165#sous la forme
166#RESTART/ATM_****.nc
167#RESTART/OCE_****.nc
168#Attention de bien conserver les dates d'origines
169#
170#Dans le cadre de la production nouvelle chaine
171#ils auront directement le bon nom
172#
173#On veut egalement que l'archive soit par periode de temps
174# cette periode est definie par la taille du pack donc connue
175# en entree du script (1M, 1Y, 5Y, 10Y etc...)
176#
177# $1= PATH/config_card d'une simulation avec
178# TagName/JobName/ExperimentName/SpaceName/DateBegin/DateEnd
179# $2=PeriodPack en M/m ou Y/y
180
181########################################################################
182
183. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
184. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
185. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
186#-------
187. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
188. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
189. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
190#-------
191RUN_DIR=${RUN_DIR_PATH}
192IGCM_sys_MkdirWork ${RUN_DIR}
193IGCM_sys_Cd ${RUN_DIR}
194#-------
195( ${DEBUG_debug} ) && IGCM_debug_Check
196( ${DEBUG_debug} ) && IGCM_card_Check
197( ${DEBUG_debug} ) && IGCM_date_Check
198
199########################################################################
200
201#set -vx
202
203# ------------------------------------------------------------------
204# Test if all was right before proceeding further
205# ------------------------------------------------------------------
206IGCM_debug_Verif_Exit
207
208if [ ${StandAlone} = true ] ; then
209    CARD_DIR=${SUBMIT_DIR}
210else
211    CARD_DIR=${RUN_DIR_PATH}
212    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
213    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
214    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
215    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
216fi
217
218#==================================
219# First of all
220#
221# Read libIGCM compatibility version in config.card
222# Read UserChoices section
223# Read Ensemble section
224# Read Post section
225# Define all netcdf output directories
226#==================================
227IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
228
229# ------------------------------------------------------------------
230# Activate BigBrother so as to supervise this job
231# ------------------------------------------------------------------
232IGCM_debug_BigBro_Initialize
233
234#==================================
235# Test and set up directories
236#==================================
237IGCM_sys_TestDirArchive ${R_SAVE}
238[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
239
240STORE_DEBUG=${R_SAVE}/DEBUG
241IGCM_sys_MkdirArchive ${STORE_DEBUG}
242
243# Switch to script variables meaning (try to be compatible with ipsl_pack TGCC moving procedure)
244JobName=${config_UserChoices_JobName}
245echo $JobName $DateBegin $DateEnd
246
247# Clear txt file lists
248rm -f ${RUN_DIR}/liste_debug_files.txt
249rm -f ${RUN_DIR}/liste_out_files.txt
250rm -f ${RUN_DIR}/new_liste_debug_files.txt
251rm -f ${RUN_DIR}/liste_pack_*.txt
252
253# Create sorted debug list to work with
254find ${R_BUFR}/*/Debug -type f | sort >> ${RUN_DIR}/liste_debug_files.txt
255
256# Rename debug files we will work with using symlinks. use ???_ prefix ie NF-2
257#gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("cp --preserve " $0 " "dest_name)}' ${RUN_DIR}/liste_debug_files.txt
258gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_debug_files.txt
259
260# Add out_ : output of execution files
261find ${R_BUFR}/Out -type f -name '*out_*' | sort >> ${RUN_DIR}/liste_out_files.txt
262
263# Rename out files we will work with using symlinks. use Out_ prefix ie NF-1
264gawk -F/ '{dest_name=$(NF-1)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_out_files.txt
265
266# List renamed debug files. Exclude Job_xxx files.
267find ${RUN_DIR} -maxdepth 1 -mindepth 1 -name "???_*${JobName}*" -ls | sort -k 11 > ${RUN_DIR}/new_liste_debug_files.txt
268
269# ------------------------------------------------------------------
270# Test if all was right before proceeding further
271# ------------------------------------------------------------------
272IGCM_debug_Verif_Exit
273
274# Init loop
275date_begin_pack=${DateBegin}
276date_end_simulation=${DateEnd}
277number_pack=1
278
279IGCM_debug_PrintVariables 3 date_begin_pack
280IGCM_debug_PrintVariables 3 date_end_simulation
281
282while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do
283
284  IGCM_debug_PrintVariables 3 number_pack
285
286  DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} )
287  date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )
288
289  for file in $( gawk '{print $11}' ${RUN_DIR}/new_liste_debug_files.txt ); do
290    extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" )
291    date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' )
292#   echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack}
293    if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then
294        echo $( basename ${file} ) >> ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
295        echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
296    fi
297  done
298
299  # Create tar files
300  IGCM_debug_Print 1 "Tar ongoing"
301  tar -cvf ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar --dereference --files-from ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
302  # Test return code of tar
303  [ $? != 0 ] && IGCM_debug_Exit "tar failed"
304  # Save it
305  IGCM_sys_Put_Out ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar ${STORE_DEBUG}
306  # Clean it
307  IGCM_sys_Rm ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar
308  # ------------------------------------------------------------------
309  # Test if all was right before proceeding cleaning
310  # ------------------------------------------------------------------
311  IGCM_debug_Verif_Exit
312  # Clean links
313  cat ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
314  # Clean files links pointed to
315  cat ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
316  # Save the list of files that has been pack (tar)
317  #mv ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt ${STORE_DEBUG}
318  IGCM_debug_Print 1 "Tar and cleaning done"
319
320  (( number_pack = number_pack + 1 ))
321
322  # Add 1 day to date_end_pack to have the new date_begin_pack
323  date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 )
324
325done
326
327# Clean RUN_DIR_PATH (necessary for cesium and titane only)
328IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
329
330# ------------------------------------------------------------------
331# Finalize BigBrother to inform that the jobs end
332# ------------------------------------------------------------------
333IGCM_debug_BigBro_Finalize
334
335date
Note: See TracBrowser for help on using the repository browser.