source: trunk/libIGCM/AA_pack_restart @ 1517

Last change on this file since 1517 was 1501, checked in by acosce, 4 years ago

change in post treatment Jean Zay job header to add jobid in the log out file

  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 12.7 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r PACKRESTART    # Nom du job
5#-Q- curie #MSUB -o PACKRESTART.out_%I
6#-Q- curie #MSUB -e PACKRESTART.out_%I
7#-Q- curie #MSUB -n 1              # Reservation du processus
8#-Q- curie #MSUB -T 7200          # Limite de temps elapsed du job
9#-Q- curie #MSUB -q ::default_node::
10#-Q- curie #MSUB -c ::default_core::
11#-Q- curie #MSUB -Q normal
12#-Q- curie #MSUB -A ::default_project::
13#-Q- curie set +x
14#-Q- irene ######################
15#-Q- irene ## IRENE   TGCC/CEA ##
16#-Q- irene ######################
17#-Q- irene #MSUB -r PACKRESTART    # Job name
18#-Q- irene #MSUB -o PACKRESTART.out_%I
19#-Q- irene #MSUB -e PACKRESTART.out_%I
20#-Q- irene #MSUB -n 1              # Number of cores
21#-Q- irene #MSUB -T 7200           # Maximum elapsed time
22#-Q- irene #MSUB -q ::default_node::
23#-Q- irene #MSUB -c ::default_core::
24#-Q- irene #MSUB -Q normal
25#-Q- irene #MSUB -A ::default_post_project::
26#-Q- irene #MSUB -m store,work,scratch
27#-Q- irene set +x
28#-Q- jeanzay #!/bin/ksh
29#-Q- jeanzay ######################
30#-Q- jeanzay ## JEANZAY    IDRIS ##
31#-Q- jeanzay ######################
32#-Q- jeanzay #SBATCH --job-name=PACKRESTART         # Job Name
33#-Q- jeanzay #SBATCH --output=PACKRESTART.out_%J    # standard output
34#-Q- jeanzay #SBATCH --error=PACKRESTART.out_%J     # error output
35#-Q- jeanzay #SBATCH -N  1                        # Number of core
36#-Q- jeanzay #SBATCH --partition=prepost          # Post-processing partition
37#-Q- jeanzay #SBATCH --time=10:00:00                # Wall clock limit (seconds)
38#-Q- jeanzay #SBATCH --account ::default_project::@cpu
39#-Q- jeanzay set +x
40#-Q- ada #!/bin/ksh
41#-Q- ada #######################
42#-Q- ada ## ADA         IDRIS ##
43#-Q- ada #######################
44#-Q- ada # @ job_type = mpich
45#-Q- ada # @ requirements = (Feature == "prepost")
46#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
47#-Q- ada # @ wall_clock_limit = 10:00:00
48#-Q- ada # Nom du travail LoadLeveler
49#-Q- ada # @ job_name   = PACKRESTART
50#-Q- ada # Fichier de sortie standard du travail
51#-Q- ada # @ output     = $(job_name).$(jobid)
52#-Q- ada # Fichier de sortie d'erreur du travail
53#-Q- ada # @ error      =  $(job_name).$(jobid)
54#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
55#-Q- ada # @ notification = error
56#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $POST_DIR ; $Script_Post_Output ; $SUBMIT_DIR ; $PeriodDateBegin ; $DateBegin ; $DateEnd ; $PeriodPack ; $StandAlone ; $MASTER ; wall_clock_limit=$(wall_clock_limit)
57#-Q- ada # @ queue
58#-Q- lxiv8 ######################
59#-Q- lxiv8 ## OBELIX      LSCE ##
60#-Q- lxiv8 ######################
61#-Q- lxiv8 #PBS -N PACKRESTART
62#-Q- lxiv8 #PBS -m a
63#-Q- lxiv8 #PBS -j oe
64#-Q- lxiv8 #PBS -q medium
65#-Q- lxiv8 #PBS -o PACKRESTART.$$
66#-Q- lxiv8 #PBS -S /bin/ksh
67#-Q- ifort_CICLAD ######################
68#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
69#-Q- ifort_CICLAD ######################
70#-Q- ifort_CICLAD #PBS -N PACKRESTART
71#-Q- ifort_CICLAD #PBS -m a
72#-Q- ifort_CICLAD #PBS -j oe
73#-Q- ifort_CICLAD #PBS -q std
74#-Q- ifort_CICLAD #PBS -S /bin/ksh
75#-Q- default #!/bin/ksh
76#-Q- default ##################
77#-Q- default ## DEFAULT HOST ##
78#-Q- default ##################
79
80#**************************************************************
81# Author: Sebastien Denvil
82# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
83# $Revision::                                          $ Revision of last commit
84# $Author::                                            $ Author of last commit
85# $Date::                                              $ Date of last commit
86# IPSL (2006)
87#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
88#
89#**************************************************************
90
91#set -eu
92#set -vx
93
94date
95
96#D- Task type DO NOT CHANGE (computing, post-processing or checking)
97TaskType=post-processing
98
99########################################################################
100
101#D- Flag to determine if this job in a standalone mode
102#D- Default : value from AA_job if any
103StandAlone=${StandAlone:=true}
104
105#D- Path to libIGCM
106#D- Default : value from AA_job if any
107# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
108# WARNING you must check MirrorlibIGCM variable in sys library.
109# WARNING If this variable is true, you must use libIGCM_POST path instead
110# WARNING of your running libIGCM directory.
111libIGCM=${libIGCM:=::modipsl::/libIGCM}
112
113#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
114MASTER=${MASTER:=ada|curie}
115
116#D- Flag to determine begin date for restart pack
117#D- Default : value from AA_job if any
118DateBegin=${DateBegin:=20000101}
119
120#D- Flag to determine end date for restart pack
121#D- Default : value from AA_job if any
122DateEnd=${DateEnd:=20691231}
123
124#D- Flag to determine pack period
125#D- Default : value from AA_job if any
126PeriodPack=${PeriodPack:=10Y}
127
128#D- Uncomment to run interactively
129#D- For testing purpose, will be remove
130#SUBMIT_DIR=${PWD}
131#RUN_DIR_PATH=${SCRATCHDIR}/Pack_Test
132
133#D- Increased verbosity (1, 2, 3)
134#D- Default : value from AA_job if any
135Verbosity=${Verbosity:=3}
136
137#D- Low level debug : to bypass lib test checks and stack construction
138#D- Default : value from AA_job if any
139DEBUG_debug=${DEBUG_debug:=false}
140
141#D- Keep following comments for now. Will be remove.
142
143#on veut archiver les fichiers restart
144#Dans le cadre du demenagement ils seront renommes
145#ATM/Restart/****.nc
146#OCE/Restart/****.nc
147#...
148#sous la forme
149#RESTART/ATM_****.nc
150#RESTART/OCE_****.nc
151#Attention de bien conserver les dates d'origines
152#
153#Dans le cadre de la production nouvelle chaine
154#ils auront directement le bon nom
155#
156#On veut egalement que l'archive soit par periode de temps
157# cette periode est definie par la taille du pack donc connue
158# en entree du script (1M, 1Y, 5Y, 10Y etc...)
159#
160# $1= PATH/config_card d'une simulation avec
161# TagName/JobName/ExperimentName/SpaceName/DateBegin/DateEnd
162# $2=PeriodPack en M/m ou Y/y
163
164########################################################################
165
166. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
167. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
168. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
169#-------
170. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
171. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
172. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
173#-------
174RUN_DIR=${RUN_DIR_PATH}
175IGCM_sys_MkdirWork ${RUN_DIR}
176IGCM_sys_Cd ${RUN_DIR}
177#-------
178( ${DEBUG_debug} ) && IGCM_debug_Check
179( ${DEBUG_debug} ) && IGCM_card_Check
180( ${DEBUG_debug} ) && IGCM_date_Check
181
182########################################################################
183
184#set -vx
185
186# ------------------------------------------------------------------
187# Test if all was right before proceeding further
188# ------------------------------------------------------------------
189IGCM_debug_Verif_Exit
190
191if [ ${StandAlone} = true ] ; then
192    CARD_DIR=${SUBMIT_DIR}
193else
194    CARD_DIR=${RUN_DIR_PATH}
195    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
196    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
197    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
198    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
199fi
200
201#==================================
202# First of all
203#
204# Read libIGCM compatibility version in config.card
205# Read UserChoices section
206# Read Ensemble section
207# Read Post section
208# Define all netcdf output directories
209#==================================
210IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
211
212# ------------------------------------------------------------------
213# Activate BigBrother so as to supervise this job
214# ------------------------------------------------------------------
215IGCM_debug_BigBro_Initialize
216
217#==================================
218# Test and set up directories
219#==================================
220IGCM_sys_TestDirArchive ${R_SAVE}
221[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
222
223STORE_RESTART=${R_SAVE}/RESTART
224IGCM_sys_MkdirArchive ${STORE_RESTART}
225
226# Switch to script variables meaning (try to be compatible with ipsl_pack TGCC moving procedure)
227JobName=${config_UserChoices_JobName}
228echo $JobName $R_BUFR $DateBegin $DateEnd
229
230# Clear txt file lists
231rm -f ${RUN_DIR}/liste_restart_files.txt
232rm -f ${RUN_DIR}/new_liste_restart_files.txt
233rm -f ${RUN_DIR}/liste_pack_*.txt
234rm -f ${RUN_DIR}/liste_pack_light_*.txt
235
236# Create sorted restart list to work with
237find ${R_BUFR}/*/Restart -name "*.*" | sort >> ${RUN_DIR}/liste_restart_files.txt
238
239# Rename restart files we will work with using symlinks
240#gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("cp --preserve " $0 " "dest_name)}' ${RUN_DIR}/liste_restart_files.txt
241gawk -F/ '{dest_name=$(NF-2)("_") $(NF); system("ln -s " $0 " "dest_name)}' ${RUN_DIR}/liste_restart_files.txt
242
243# List renamed restart files
244find ${RUN_DIR} -maxdepth 1 -mindepth 1 -name "???_*${JobName}*" -ls | sort -k 11 > ${RUN_DIR}/new_liste_restart_files.txt
245
246# ------------------------------------------------------------------
247# Test if all was right before proceeding further
248# ------------------------------------------------------------------
249IGCM_debug_Verif_Exit
250
251# Init loop
252date_begin_pack=${DateBegin}
253date_end_simulation=${DateEnd}
254number_pack=1
255
256IGCM_debug_PrintVariables 3 date_begin_pack
257IGCM_debug_PrintVariables 3 date_end_simulation
258
259# last restart of the previous period : to be deleted
260typeset date_end_previous
261date_end_previous=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} -1 )
262
263while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do
264
265  IGCM_debug_PrintVariables 3 number_pack
266
267  DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} )
268  date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )
269
270  for file in $( gawk '{print $11}' ${RUN_DIR}/new_liste_restart_files.txt ); do
271    extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_//" )
272    date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' )
273    # warning demenagement. delete last restart of the previous pack
274    [  ${date_file} -eq ${date_end_previous} ] && echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
275    # echo pack number ${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack}
276    if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then
277      echo $( basename ${file} ) >> ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
278      # warning demenagement. do not delete last restart of this pack
279      [ ${date_file} -eq ${date_end_pack} ] || echo $( readlink ${file} ) >> ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt
280    fi
281    # light pack list of restarts files that will be used
282    if [ X${config_Post_LightRestartPack} = XTRUE ] ; then
283      if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${PeriodDateBegin} ] ; then
284        echo $( basename ${file} ) >> ${RUN_DIR}/liste_pack_light_${date_begin_pack}_${date_end_pack}.txt
285      fi
286    fi
287  done
288
289  # Create tar files
290  IGCM_debug_Print 1 "Tar ongoing"
291  if [ X${config_Post_LightRestartPack} = XTRUE ] ; then
292    tar -cvf ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar --dereference --files-from ${RUN_DIR}/liste_pack_light_${date_begin_pack}_${date_end_pack}.txt
293  else
294    tar -cvf ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar --dereference --files-from ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt
295  fi
296  # Test return code of tar
297  [ $? != 0 ] && IGCM_debug_Exit "tar failed in pack_restart job"
298  # Save it
299  IGCM_sys_Put_Out ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar ${STORE_RESTART}
300  # Clean it
301  IGCM_sys_Rm ${JobName}_${date_begin_pack}_${date_end_pack}_restart.tar
302  # ------------------------------------------------------------------
303  # Test if all was right before proceeding cleaning
304  # ------------------------------------------------------------------
305  IGCM_debug_Verif_Exit
306  # Clean links
307  cat ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
308  # Clean files links pointed to
309  cat ${RUN_DIR}/liste_remove_pack_${date_begin_pack}_${date_end_pack}.txt | xargs rm
310  # Save the list of files that has been pack (tar)
311  #mv ${RUN_DIR}/liste_pack_${date_begin_pack}_${date_end_pack}.txt ${STORE_RESTART}
312  IGCM_debug_Print 1 "Tar and cleaning done"
313
314  (( number_pack = number_pack + 1 ))
315
316  # Add 1 day to date_end_pack to have the new date_begin_pack
317  date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 )
318
319done
320
321# Clean RUN_DIR_PATH (necessary for cesium and titane only)
322IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
323
324# ------------------------------------------------------------------
325# Finalize BigBrother to inform that the jobs end
326# ------------------------------------------------------------------
327IGCM_debug_BigBro_Finalize
328
329date
Note: See TracBrowser for help on using the repository browser.