source: trunk/libIGCM/AA_pack_debug @ 1206

Last change on this file since 1206 was 1206, checked in by sdipsl, 9 years ago
  • Remove IGCM_debug_Verif_Exit_Post. Only IGCM_debug_Verif_Exit will manage exit cases.
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 10.6 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r PACKDEBUG      # Nom du job
5#-Q- curie #MSUB -eo
6#-Q- curie #MSUB -n 1              # Reservation du processus
7#-Q- curie #MSUB -T 3600          # Limite de temps elapsed du job
8#-Q- curie #MSUB -q ::default_node::
9#-Q- curie #MSUB -Q normal
10#-Q- curie #MSUB -A ::default_project::
11#-Q- curie set +x
12#-Q- ada #!/bin/ksh
13#-Q- ada #######################
14#-Q- ada ## ADA         IDRIS ##
15#-Q- ada #######################
16#-Q- ada # @ job_type = serial
17#-Q- ada # @ requirements = (Feature == "prepost")
18#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
19#-Q- ada # @ wall_clock_limit = 10:00:00
20#-Q- ada # Nom du travail LoadLeveler
21#-Q- ada # @ job_name   = PACKDEBUG
22#-Q- ada # Fichier de sortie standard du travail
23#-Q- ada # @ output     = $(job_name).$(jobid)
24#-Q- ada # Fichier de sortie d'erreur du travail
25#-Q- ada # @ error      =  $(job_name).$(jobid)
26#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
27#-Q- ada # @ notification = error
28#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $POST_DIR ; $Script_Post_Output ; $SUBMIT_DIR ; $DateBegin ; $DateEnd ; $PeriodPack ; $StandAlone ; $MASTER ; wall_clock_limit=$(wall_clock_limit)
29#-Q- ada # @ queue
30#-Q- lxiv8 ######################
31#-Q- lxiv8 ## OBELIX      LSCE ##
32#-Q- lxiv8 ######################
33#-Q- lxiv8 #PBS -N PACKDEBUG
34#-Q- lxiv8 #PBS -m a
35#-Q- lxiv8 #PBS -j oe
36#-Q- lxiv8 #PBS -q medium
37#-Q- lxiv8 #PBS -o PACKDEBUG.$$
38#-Q- lxiv8 #PBS -S /bin/ksh
39#-Q- ifort_CICLAD ######################
40#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
41#-Q- ifort_CICLAD ######################
42#-Q- ifort_CICLAD #PBS -N PACKDEBUG
43#-Q- ifort_CICLAD #PBS -m a
44#-Q- ifort_CICLAD #PBS -j oe
45#-Q- ifort_CICLAD #PBS -q std
46#-Q- ifort_CICLAD #PBS -S /bin/ksh
47#-Q- default #!/bin/ksh
48#-Q- default ##################
49#-Q- default ## DEFAULT HOST ##
50#-Q- default ##################
51
52#**************************************************************
53# Author: Sebastien Denvil
54# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
55# $Revision::                                          $ Revision of last commit
56# $Author::                                            $ Author of last commit
57# $Date::                                              $ Date of last commit
58# IPSL (2006)
59#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
60#
61#**************************************************************
62
63#set -eu
64#set -vx
65
66date
67
68#D- Task type (computing or post-processing)
69TaskType=post-processing
70
71########################################################################
72
73#D- Flag to determine if this job in a standalone mode
74#D- Default : value from AA_job if any
75StandAlone=${StandAlone:=true}
76
77#D- Path to libIGCM
78#D- Default : value from AA_job if any
79# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
80# WARNING you must check MirrorlibIGCM variable in sys library.
81# WARNING If this variable is true, you must use libIGCM_POST path instead
82# WARNING of your running libIGCM directory.
83libIGCM=${libIGCM:=::modipsl::/libIGCM}
84
85#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
86MASTER=${MASTER:=ada|curie}
87
88#D- Flag to determine begin date for restart pack
89#D- Default : value from AA_job if any
90DateBegin=${DateBegin:=20000101}
91
92#D- Flag to determine end date for restart pack
93#D- Default : value from AA_job if any
94DateEnd=${DateEnd:=20691231}
95
96#D- Flag to determine pack period
97#D- Default : value from AA_job if any
98PeriodPack=${PeriodPack:=10Y}
99
100#D- Uncomment to run interactively
101#D- For testing purpose, will be remove
102#SUBMIT_DIR=${PWD}
103#RUN_DIR_PATH=${SCRATCHDIR}/Pack_Test
104
105#D- Increased verbosity (1, 2, 3)
106#D- Default : value from AA_job if any
107Verbosity=${Verbosity:=3}
108
109#D- Low level debug : to bypass lib test checks and stack construction
110#D- Default : value from AA_job if any
111DEBUG_debug=${DEBUG_debug:=false}
112
113#D- Keep following comments for now. Will be remove.
114
115#on veut archiver les fichiers restart
116#Dans le cadre du demenagement ils seront renommes
117#ATM/Restart/****.nc
118#OCE/Restart/****.nc
119#...
120#sous la forme
121#RESTART/ATM_****.nc
122#RESTART/OCE_****.nc
123#Attention de bien conserver les dates d'origines
124#
125#Dans le cadre de la production nouvelle chaine
126#ils auront directement le bon nom
127#
128#On veut egalement que l'archive soit par periode de temps
129# cette periode est definie par la taille du pack donc connue
130# en entree du script (1M, 1Y, 5Y, 10Y etc...)
131#
132# $1= PATH/config_card d'une simulation avec
133# TagName/JobName/ExperimentName/SpaceName/DateBegin/DateEnd
134# $2=PeriodPack en M/m ou Y/y
135
136########################################################################
137
138. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
139. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
140. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
141#-------
142. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
143. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
144. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
145#-------
146RUN_DIR=${RUN_DIR_PATH}
147IGCM_sys_MkdirWork ${RUN_DIR}
148IGCM_sys_Cd ${RUN_DIR}
149#-------
150( ${DEBUG_debug} ) && IGCM_debug_Check
151( ${DEBUG_debug} ) && IGCM_card_Check
152( ${DEBUG_debug} ) && IGCM_date_Check
153
154########################################################################
155
156#set -vx
157
158# ------------------------------------------------------------------
159# Test if all was right before proceeding further
160# ------------------------------------------------------------------
161IGCM_debug_Verif_Exit
162
163if [ ${StandAlone} = true ] ; then
164    CARD_DIR=${SUBMIT_DIR}
165else
166    CARD_DIR=${RUN_DIR_PATH}
167    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
168    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
169    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
170    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
171fi
172
173#==================================
174# First of all
175#
176# Read libIGCM compatibility version in config.card
177# Read UserChoices section
178# Read Ensemble section
179# Read Post section
180# Define all netcdf output directories
181#==================================
182IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
183
184# ------------------------------------------------------------------
185# Activate BigBrother so as to supervise this job
186# ------------------------------------------------------------------
187IGCM_debug_BigBro_Initialize
188
189#==================================
190# Test and set up directories
191#==================================
192IGCM_sys_TestDirArchive ${R_SAVE}
193[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
194
195STORE_DEBUG=${R_SAVE}/DEBUG
196IGCM_sys_MkdirArchive ${STORE_DEBUG}
197
198# Switch to script variables meaning (try to be compatible with ipsl_pack TGCC moving procedure)
199JobName=${config_UserChoices_JobName}
200echo $JobName $DateBegin $DateEnd
201
202# Clear txt file lists
203rm -f ${RUN_DIR}/liste_debug_files.txt
204rm -f ${RUN_DIR}/liste_out_files.txt
205rm -f ${RUN_DIR}/new_liste_debug_files.txt
206rm -f ${RUN_DIR}/liste_pack_*.txt
207
208# Create sorted debug list to work with
209find ${R_BUFR}/*/Debug -type f | sort >> ${RUN_DIR}/liste_debug_files.txt
210
211# Rename debug files we will work with using symlinks. use ???_ prefix ie NF-2
212#gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("cp --preserve " $0 " "dest_name)}' ${RUN_DIR}/liste_debug_files.txt
213gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_debug_files.txt
214
215# Add out_ : output of execution files
216find ${R_BUFR}/Out -type f -name '*out_*' | sort >> ${RUN_DIR}/liste_out_files.txt
217
218# Rename out files we will work with using symlinks. use Out_ prefix ie NF-1
219gawk -F/ '{dest_name=$(NF-1)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_out_files.txt
220
221# List renamed debug files. Exclude Job_xxx files.
222find ${RUN_DIR} -maxdepth 1 -mindepth 1 -name "???_*${JobName}*" -ls | sort -k 11 > ${RUN_DIR}/new_liste_debug_files.txt
223
224# ------------------------------------------------------------------
225# Test if all was right before proceeding further
226# ------------------------------------------------------------------
227IGCM_debug_Verif_Exit
228
229# Init loop
230date_begin_pack=${DateBegin}
231date_end_simulation=${DateEnd}
232number_pack=1
233
234IGCM_debug_PrintVariables 3 date_begin_pack
235IGCM_debug_PrintVariables 3 date_end_simulation
236
237while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do
238
239  IGCM_debug_PrintVariables 3 number_pack
240
241  DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} )
242  date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )
243
244  for file in $( gawk '{print $11}' ${RUN_DIR}/new_liste_debug_files.txt ); do
245    extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" )
246    date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' )
247#   echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack}
248    if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then
249        echo $( basename ${file} ) >> ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
250        echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
251    fi
252  done
253
254  # Create tar files
255  IGCM_debug_Print 1 "Tar ongoing"
256  tar -cvf ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar --dereference --files-from ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
257  # Save it
258  IGCM_sys_Put_Out ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar ${STORE_DEBUG}
259  # Clean it
260  IGCM_sys_Rm ${JobName}_${date_begin_pack}_${date_end_pack}_debug.tar
261  # ------------------------------------------------------------------
262  # Test if all was right before proceeding cleaning
263  # ------------------------------------------------------------------
264  IGCM_debug_Verif_Exit
265  # Clean links
266  cat ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
267  # Clean files links pointed to
268  cat ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
269  # Save the list of files that has been pack (tar)
270  #mv ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt ${STORE_DEBUG}
271  IGCM_debug_Print 1 "Tar and cleaning done"
272
273  (( number_pack = number_pack + 1 ))
274
275  # Add 1 day to date_end_pack to have the new date_begin_pack
276  date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 )
277
278done
279
280# Clean RUN_DIR_PATH (necessary for cesium and titane only)
281IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
282
283# ------------------------------------------------------------------
284# Finalize BigBrother to inform that the jobs end
285# ------------------------------------------------------------------
286IGCM_debug_BigBro_Finalize
287
288date
Note: See TracBrowser for help on using the repository browser.