source: tags/libIGCM_v2.8/AA_pack_restart

Last change on this file was 1290, checked in by sdipsl, 8 years ago
  • broadcast postProcessingStopLevel value to every jobs. Default value is zero.
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 10.8 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r PACKRESTART    # Nom du job
5#-Q- curie #MSUB -eo
6#-Q- curie #MSUB -n 1              # Reservation du processus
7#-Q- curie #MSUB -T 7200          # Limite de temps elapsed du job
8#-Q- curie #MSUB -q ::default_node::
9#-Q- curie #MSUB -c ::default_core::
10#-Q- curie #MSUB -Q normal
11#-Q- curie #MSUB -A ::default_project::
12#-Q- curie set +x
13#-Q- ada #!/bin/ksh
14#-Q- ada #######################
15#-Q- ada ## ADA         IDRIS ##
16#-Q- ada #######################
17#-Q- ada # @ job_type = serial
18#-Q- ada # @ requirements = (Feature == "prepost")
19#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
20#-Q- ada # @ wall_clock_limit = 10:00:00
21#-Q- ada # Nom du travail LoadLeveler
22#-Q- ada # @ job_name   = PACKRESTART
23#-Q- ada # Fichier de sortie standard du travail
24#-Q- ada # @ output     = $(job_name).$(jobid)
25#-Q- ada # Fichier de sortie d'erreur du travail
26#-Q- ada # @ error      =  $(job_name).$(jobid)
27#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
28#-Q- ada # @ notification = error
29#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $POST_DIR ; $Script_Post_Output ; $SUBMIT_DIR ; $DateBegin ; $DateEnd ; $PeriodPack ; $StandAlone ; $MASTER ; wall_clock_limit=$(wall_clock_limit)
30#-Q- ada # @ queue
31#-Q- lxiv8 ######################
32#-Q- lxiv8 ## OBELIX      LSCE ##
33#-Q- lxiv8 ######################
34#-Q- lxiv8 #PBS -N PACKRESTART
35#-Q- lxiv8 #PBS -m a
36#-Q- lxiv8 #PBS -j oe
37#-Q- lxiv8 #PBS -q medium
38#-Q- lxiv8 #PBS -o PACKRESTART.$$
39#-Q- lxiv8 #PBS -S /bin/ksh
40#-Q- ifort_CICLAD ######################
41#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
42#-Q- ifort_CICLAD ######################
43#-Q- ifort_CICLAD #PBS -N PACKRESTART
44#-Q- ifort_CICLAD #PBS -m a
45#-Q- ifort_CICLAD #PBS -j oe
46#-Q- ifort_CICLAD #PBS -q std
47#-Q- ifort_CICLAD #PBS -S /bin/ksh
48#-Q- default #!/bin/ksh
49#-Q- default ##################
50#-Q- default ## DEFAULT HOST ##
51#-Q- default ##################
52
53#**************************************************************
54# Author: Sebastien Denvil
55# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
56# $Revision::                                          $ Revision of last commit
57# $Author::                                            $ Author of last commit
58# $Date::                                              $ Date of last commit
59# IPSL (2006)
60#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
61#
62#**************************************************************
63
64#set -eu
65#set -vx
66
67date
68
69#D- Task type (computing or post-processing)
70TaskType=post-processing
71
72########################################################################
73
74#D- Flag to determine if this job in a standalone mode
75#D- Default : value from AA_job if any
76StandAlone=${StandAlone:=true}
77
78#D- Path to libIGCM
79#D- Default : value from AA_job if any
80# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
81# WARNING you must check MirrorlibIGCM variable in sys library.
82# WARNING If this variable is true, you must use libIGCM_POST path instead
83# WARNING of your running libIGCM directory.
84libIGCM=${libIGCM:=::modipsl::/libIGCM}
85
86#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
87MASTER=${MASTER:=ada|curie}
88
89#D- Flag to determine begin date for restart pack
90#D- Default : value from AA_job if any
91DateBegin=${DateBegin:=20000101}
92
93#D- Flag to determine end date for restart pack
94#D- Default : value from AA_job if any
95DateEnd=${DateEnd:=20691231}
96
97#D- Flag to determine pack period
98#D- Default : value from AA_job if any
99PeriodPack=${PeriodPack:=10Y}
100
101#D- Uncomment to run interactively
102#D- For testing purpose, will be remove
103#SUBMIT_DIR=${PWD}
104#RUN_DIR_PATH=${SCRATCHDIR}/Pack_Test
105
106#D- Increased verbosity (1, 2, 3)
107#D- Default : value from AA_job if any
108Verbosity=${Verbosity:=3}
109
110#D- Low level debug : to bypass lib test checks and stack construction
111#D- Default : value from AA_job if any
112DEBUG_debug=${DEBUG_debug:=false}
113
114#D- Keep following comments for now. Will be remove.
115
116#on veut archiver les fichiers restart
117#Dans le cadre du demenagement ils seront renommes
118#ATM/Restart/****.nc
119#OCE/Restart/****.nc
120#...
121#sous la forme
122#RESTART/ATM_****.nc
123#RESTART/OCE_****.nc
124#Attention de bien conserver les dates d'origines
125#
126#Dans le cadre de la production nouvelle chaine
127#ils auront directement le bon nom
128#
129#On veut egalement que l'archive soit par periode de temps
130# cette periode est definie par la taille du pack donc connue
131# en entree du script (1M, 1Y, 5Y, 10Y etc...)
132#
133# $1= PATH/config_card d'une simulation avec
134# TagName/JobName/ExperimentName/SpaceName/DateBegin/DateEnd
135# $2=PeriodPack en M/m ou Y/y
136
137########################################################################
138
139. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
140. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
141. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
142#-------
143. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
144. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
145. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
146#-------
147RUN_DIR=${RUN_DIR_PATH}
148IGCM_sys_MkdirWork ${RUN_DIR}
149IGCM_sys_Cd ${RUN_DIR}
150#-------
151( ${DEBUG_debug} ) && IGCM_debug_Check
152( ${DEBUG_debug} ) && IGCM_card_Check
153( ${DEBUG_debug} ) && IGCM_date_Check
154
155########################################################################
156
157#set -vx
158
159# ------------------------------------------------------------------
160# Test if all was right before proceeding further
161# ------------------------------------------------------------------
162IGCM_debug_Verif_Exit
163
164if [ ${StandAlone} = true ] ; then
165    CARD_DIR=${SUBMIT_DIR}
166else
167    CARD_DIR=${RUN_DIR_PATH}
168    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
169    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
170    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
171    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
172fi
173
174#==================================
175# First of all
176#
177# Read libIGCM compatibility version in config.card
178# Read UserChoices section
179# Read Ensemble section
180# Read Post section
181# Define all netcdf output directories
182#==================================
183IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
184
185# ------------------------------------------------------------------
186# Activate BigBrother so as to supervise this job
187# ------------------------------------------------------------------
188IGCM_debug_BigBro_Initialize
189
190#==================================
191# Test and set up directories
192#==================================
193IGCM_sys_TestDirArchive ${R_SAVE}
194[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
195
196STORE_RESTART=${R_SAVE}/RESTART
197IGCM_sys_MkdirArchive ${STORE_RESTART}
198
199# Switch to script variables meaning (try to be compatible with ipsl_pack TGCC moving procedure)
200JobName=${config_UserChoices_JobName}
201echo $JobName $R_BUFR $DateBegin $DateEnd
202
203# Clear txt file lists
204rm -f ${RUN_DIR}/liste_restart_files.txt
205rm -f ${RUN_DIR}/new_liste_restart_files.txt
206rm -f ${RUN_DIR}/liste_pack_*.txt
207
208# Create sorted restart list to work with
209find ${R_BUFR}/*/Restart -name "*.*" | sort >> ${RUN_DIR}/liste_restart_files.txt
210
211# Rename restart files we will work with using symlinks
212#gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("cp --preserve " $0 " "dest_name)}' ${RUN_DIR}/liste_restart_files.txt
213gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_restart_files.txt
214
215# List renamed restart files
216find ${RUN_DIR} -maxdepth 1 -mindepth 1 -name "???_*${JobName}*" -ls | sort -k 11 > ${RUN_DIR}/new_liste_restart_files.txt
217
218# ------------------------------------------------------------------
219# Test if all was right before proceeding further
220# ------------------------------------------------------------------
221IGCM_debug_Verif_Exit
222
223# Init loop
224date_begin_pack=${DateBegin}
225date_end_simulation=${DateEnd}
226number_pack=1
227
228IGCM_debug_PrintVariables 3 date_begin_pack
229IGCM_debug_PrintVariables 3 date_end_simulation
230
231# last restart of the previous period : to be deleted
232typeset date_end_previous
233date_end_previous=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} -1 )
234
235while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do
236
237  IGCM_debug_PrintVariables 3 number_pack
238
239  DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} )
240  date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )
241
242  for file in $( gawk '{print $11}' ${RUN_DIR}/new_liste_restart_files.txt ); do
243    extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_//" )
244    date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' )
245    # warning demenagement. delete last restart of the previous pack
246    [  ${date_file} -eq ${date_end_previous} ] && echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
247    # echo pack number ${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack}
248    if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then
249      echo $( basename ${file} ) >> ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
250      # warning demenagement. do not delete last restart of this pack
251      [ ${date_file} -eq ${date_end_pack} ] || echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
252    fi
253  done
254
255  # Create tar files
256  IGCM_debug_Print 1 "Tar ongoing"
257  tar -cvf ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar --dereference --files-from ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
258  # Save it
259  IGCM_sys_Put_Out ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar ${STORE_RESTART}
260  # Clean it
261  IGCM_sys_Rm ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar
262  # ------------------------------------------------------------------
263  # Test if all was right before proceeding cleaning
264  # ------------------------------------------------------------------
265  IGCM_debug_Verif_Exit
266  # Clean links
267  cat ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
268  # Clean files links pointed to
269  cat ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
270  # Save the list of files that has been pack (tar)
271  #mv ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt ${STORE_RESTART}
272  IGCM_debug_Print 1 "Tar and cleaning done"
273
274  (( number_pack = number_pack + 1 ))
275
276  # Add 1 day to date_end_pack to have the new date_begin_pack
277  date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 )
278
279done
280
281# Clean RUN_DIR_PATH (necessary for cesium and titane only)
282IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
283
284# ------------------------------------------------------------------
285# Finalize BigBrother to inform that the jobs end
286# ------------------------------------------------------------------
287IGCM_debug_BigBro_Finalize
288
289date
Note: See TracBrowser for help on using the repository browser.