source: branches/libIGCM_CESMEP/AA_pack_output

Last change on this file was 1585, checked in by ssenesi, 12 months ago

merge branch libIGCM_CESMEP with trunk that handles RedHat8 on Irene
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 14.4 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r PACKOUTPUT     # Nom du job
5#-Q- curie #MSUB -o PACKOUTPUT.out_%I
6#-Q- curie #MSUB -e PACKOUTPUT.out_%I
7#-Q- curie #MSUB -n 1              # Reservation du processus
8#-Q- curie #MSUB -T 36000          # Limite de temps elapsed du job
9#-Q- curie #MSUB -q ::default_node::
10#-Q- curie #MSUB -c ::default_core::
11#-Q- curie #MSUB -Q normal
12#-Q- curie #MSUB -A ::default_project::
13#-Q- curie set +x
14#-Q- irene ######################
15#-Q- irene ## IRENE   TGCC/CEA ##
16#-Q- irene ######################
17#-Q- irene #MSUB -r PACKOUTPUT     # Job name
18#-Q- irene #MSUB -o PACKOUTPUT.out_%I
19#-Q- irene #MSUB -e PACKOUTPUT.out_%I
20#-Q- irene #MSUB -n 1              # Number of cores
21#-Q- irene #MSUB -T 36000          # Maximum elapsed time
22#-Q- irene #MSUB -q ::default_node::
23#-Q- irene #MSUB -c ::default_core::
24#-Q- irene #MSUB -Q normal
25#-Q- irene #MSUB -A ::default_post_project::
26#-Q- irene #MSUB -m store,work,scratch
27#-Q- irene set +x
28#-Q- irene-amd ######################
29#-Q- irene-amd ## IRENE   TGCC/CEA ##
30#-Q- irene-amd ######################
31#-Q- irene-amd #MSUB -r PACKOUTPUT     # Job name
32#-Q- irene-amd #MSUB -o PACKOUTPUT.out_%I
33#-Q- irene-amd #MSUB -e PACKOUTPUT.out_%I
34#-Q- irene-amd #MSUB -n 1              # Number of cores
35#-Q- irene-amd #MSUB -T 36000          # Maximum elapsed time
36#-Q- irene-amd #MSUB -q ::default_node::
37#-Q- irene-amd #MSUB -c ::default_core::
38#-Q- irene-amd #MSUB -Q normal
39#-Q- irene-amd #MSUB -A ::default_post_project::
40#-Q- irene-amd #MSUB -m store,work,scratch
41#-Q- irene-amd set +x
42#-Q- jeanzay #!/bin/ksh
43#-Q- jeanzay ######################
44#-Q- jeanzay ## JEANZAY    IDRIS ##
45#-Q- jeanzay ######################
46#-Q- jeanzay #SBATCH --job-name=PACKOUTPUT         # Job Name
47#-Q- jeanzay #SBATCH --output=PACKOUTPUT.out_%J    # standard output
48#-Q- jeanzay #SBATCH --error=PACKOUTPUT.out_%J     # error output
49#-Q- jeanzay #SBATCH -N  1                        # Number of core
50#-Q- jeanzay #SBATCH --partition=prepost          # Post-processing partition
51#-Q- jeanzay #SBATCH --time=10:00:00               # Wall clock limit (seconds)
52#-Q- jeanzay #SBATCH --account ::default_project::@cpu
53#-Q- jeanzay set +x
54#-Q- ada #!/bin/ksh
55#-Q- ada #######################
56#-Q- ada ## ADA         IDRIS ##
57#-Q- ada #######################
58#-Q- ada # @ job_type = mpich
59#-Q- ada # @ requirements = (Feature == "prepost")
60#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
61#-Q- ada # @ wall_clock_limit = 10:00:00
62#-Q- ada # Memory required for ncrcat
63#-Q- ada # @ as_limit = 30Gb
64#-Q- ada # Nom du travail LoadLeveler
65#-Q- ada # @ job_name   = PACKOUTPUT
66#-Q- ada # Fichier de sortie standard du travail
67#-Q- ada # @ output     = $(job_name).$(jobid)
68#-Q- ada # Fichier de sortie d'erreur du travail
69#-Q- ada # @ error      =  $(job_name).$(jobid)
70#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
71#-Q- ada # @ notification = error
72#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $POST_DIR ; $Script_Post_Output ; $SUBMIT_DIR ; $DateBegin ; $DateEnd ; $PeriodPack ; $StandAlone ; $MASTER ; wall_clock_limit=$(wall_clock_limit)
73#-Q- ada # @ queue
74#-Q- obelix ######################
75#-Q- obelix ## OBELIX      LSCE ##
76#-Q- obelix ######################
77#-Q- obelix #PBS -N PACKOUTPUT
78#-Q- obelix #PBS -m a
79#-Q- obelix #PBS -j oe
80#-Q- obelix #PBS -q medium
81#-Q- obelix #PBS -o PACKOUTPUT.$$
82#-Q- obelix #PBS -S /bin/ksh
83#-Q- ifort_CICLAD ######################
84#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
85#-Q- ifort_CICLAD ######################
86#-Q- ifort_CICLAD #PBS -N PACKOUTPUT
87#-Q- ifort_CICLAD #PBS -m a
88#-Q- ifort_CICLAD #PBS -j oe
89#-Q- ifort_CICLAD #PBS -q std
90#-Q- ifort_CICLAD #PBS -S /bin/ksh
91#-Q- default #!/bin/ksh
92#-Q- default ##################
93#-Q- default ## DEFAULT HOST ##
94#-Q- default ##################
95
96#**************************************************************
97# Author: Sebastien Denvil
98# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
99# $Revision::                                          $ Revision of last commit
100# $Author::                                            $ Author of last commit
101# $Date::                                              $ Date of last commit
102# IPSL (2006)
103#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
104#
105#**************************************************************
106
107#set -eu
108#set -vx
109
110date
111
112#D- Task type DO NOT CHANGE (computing, post-processing or checking)
113TaskType=post-processing
114
115########################################################################
116
117#D- Flag to determine if this job in a standalone mode
118#D- Default : value from AA_job if any
119StandAlone=${StandAlone:=true}
120
121#D- Path to libIGCM
122#D- Default : value from AA_job if any
123# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
124# WARNING you must check MirrorlibIGCM variable in sys library.
125# WARNING If this variable is true, you must use libIGCM_POST path instead
126# WARNING of your running libIGCM directory.
127libIGCM=${libIGCM:=::modipsl::/libIGCM}
128
129#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
130MASTER=${MASTER:=ada|curie}
131
132#D- Flag to determine begin date for restart pack
133#D- Default : value from AA_job if any
134DateBegin=${DateBegin:=20000101}
135
136#D- Flag to determine end date for restart pack
137#D- Default : value from AA_job if any
138DateEnd=${DateEnd:=20691231}
139
140#D- Flag to determine pack period
141#D- Default : value from AA_job if any
142PeriodPack=${PeriodPack:=10Y}
143
144#D- Uncomment to run interactively
145#D- For testing purpose, will be remove
146#SUBMIT_DIR=${PWD}
147#RUN_DIR_PATH=${SCRATCHDIR}/Pack_Test
148
149#D- Increased verbosity (1, 2, 3)
150#D- Default : value from AA_job if any
151Verbosity=${Verbosity:=3}
152
153#D- Low level debug : to bypass lib test checks and stack construction
154#D- Default : value from AA_job if any
155DEBUG_debug=${DEBUG_debug:=false}
156
157########################################################################
158
159. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
160. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
161. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
162#-------
163. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
164. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
165. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
166#-------
167RUN_DIR=${RUN_DIR_PATH}
168IGCM_sys_MkdirWork ${RUN_DIR}
169IGCM_sys_Cd ${RUN_DIR}
170#-------
171( ${DEBUG_debug} ) && IGCM_debug_Check
172( ${DEBUG_debug} ) && IGCM_card_Check
173( ${DEBUG_debug} ) && IGCM_date_Check
174
175########################################################################
176
177#set -vx
178
179# ------------------------------------------------------------------
180# Test if all was right before proceeding further
181# ------------------------------------------------------------------
182IGCM_debug_Verif_Exit
183
184if [ ${StandAlone} = true ] ; then
185    CARD_DIR=${SUBMIT_DIR}
186else
187    CARD_DIR=${RUN_DIR_PATH}
188    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
189    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
190    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
191    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
192fi
193
194#==================================
195# First of all
196#
197# Read libIGCM compatibility version in config.card
198# Read UserChoices section
199# Read Ensemble section
200# Read Post section
201# Define all netcdf output directories
202#==================================
203IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
204
205# ------------------------------------------------------------------
206# Activate BigBrother so as to supervise this job
207# ------------------------------------------------------------------
208IGCM_debug_BigBro_Initialize
209
210#==================================
211# Read ListOfComponents section
212# to drive the loop over find
213IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
214
215#==================================
216# Test and set up directories
217#==================================
218IGCM_sys_TestDirArchive ${R_SAVE}
219[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
220
221# Where to store used file list /!\ TEMPORARY /!\
222STORE_DEBUG=${R_SAVE}/DEBUG
223
224# Switch to script variables meaning (try to be compatible with ipsl_pack TGCC moving procedure)
225JobName=${config_UserChoices_JobName}
226echo $JobName $DateBegin $DateEnd
227
228# ------------------------------------------------------------------
229# Test if all was right before proceeding further
230# ------------------------------------------------------------------
231IGCM_debug_Verif_Exit
232
233IGCM_debug_Print 1 "Check coherence between PackFrequency and PeriodLength"
234IGCM_post_CheckModuloFrequency PeriodPack config_UserChoices_PeriodLength NbPeriodPerFrequency
235# ------------------------------------------------------------------
236# Test if all was right before proceeding further
237# ------------------------------------------------------------------
238IGCM_debug_Verif_Exit
239
240IGCM_debug_Print 1 "We must process ${NbPeriodPerFrequency} files for each pack"
241
242# Init loop
243date_begin_pack=${DateBegin}
244date_end_simulation=${DateEnd}
245number_pack=1
246
247IGCM_debug_PrintVariables 3 date_begin_pack
248IGCM_debug_PrintVariables 3 date_end_simulation
249
250while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do
251
252  IGCM_debug_PrintVariables 3 number_pack
253  DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} )
254  date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )
255
256  for comp in ${config_ListOfComponents[*]} ; do
257    dirList=$( find ${R_BUFR}/${comp}/Output -maxdepth 1 -mindepth 1 -type d )
258    for dir in ${dirList} ; do
259      # dirID is like ATM.Output.MO
260      dirID=$( echo $dir | sed "s:${R_BUFR}/::" | sed "s:/:.:g" )
261      # Sort what's in the directory
262      find ${dir} -type f -name "${JobName}*.nc" -ls | sort -k 11 > liste_files.${dirID}.txt
263      # How much file type. Example : 1M_histmthCOSP.nc, 1M_histmth.nc, 1M_histmthNMC.nc, 1M_paramLMDZ_phy.nc
264      # /!\ fileType include the .nc extension /!\
265      fileType=$( gawk '{print $11}' liste_files.${dirID}.txt | gawk -F$dir/ '{print $2}' | sed "s:${JobName}_[0-9]\{8,9\}_[0-9]\{8,9\}_::g" | sort | uniq )
266      # Loop over the file type and pack them when in between date_begin_pack and date_end_pack
267      for myType in ${fileType} ; do
268        grep ${myType} liste_files.${dirID}.txt > liste_files.${dirID}.${myType}.txt
269        nbfile=0
270        for file in $( gawk '{print $11}' liste_files.${dirID}.${myType}.txt ); do
271          extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" )
272          date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' )
273          # echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack}
274          if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then
275            echo ${file} >> liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt
276            ncdump -h ${file} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> liste_variables_${myType}_${date_begin_pack}_${date_end_pack}.txt
277            (( nbfile = nbfile + 1 ))
278          fi
279        done
280
281        if [ ${nbfile} = 0 ] ; then
282          IGCM_debug_Print 1 "We found no file to process"
283          IGCM_debug_Print 1 "We should have found ${NbPeriodPerFrequency} files"
284          IGCM_debug_Print 1 "As some files can be produced only for some selected period we consider we can move to the next file type"
285          continue
286        fi
287
288        # Select list of variables to work with
289        list_var=$( cat liste_variables_${myType}_${date_begin_pack}_${date_end_pack}.txt | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 != nbfile) {print $2}}' | paste -s -d ',' )
290        liste_file_tmp=$( for i in $( cat liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt ) ; do basename $i ; done )
291        # Create packed files
292        IGCM_debug_Print 1 "Ncrcat ongoing for ${dir} and ${myType}"
293        if [ ! ${nbfile} = ${NbPeriodPerFrequency} ] ; then
294          IGCM_debug_Print 1 "Number of files to process is not equal to what it should be"
295          IGCM_debug_Print 1 "We found ${nbfile} files and it should have been ${NbPeriodPerFrequency} files"
296          IGCM_debug_Exit "ERROR in number of files to process. STOP HERE INCLUDING THE COMPUTING JOB"
297          IGCM_debug_Verif_Exit
298        fi
299        output=${JobName}_${date_begin_pack}_${date_end_pack}_${myType}
300        #cat liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt | xargs ncrcat -v ${list_var} -o ${output}
301        if [ X${list_var} = X ] ; then
302          IGCM_sys_ncrcat -p ${dir} ${liste_file_tmp} --output ${output}
303        else
304          IGCM_sys_ncrcat -x -v ${list_var} -p ${dir} ${liste_file_tmp} --output ${output}
305        fi
306        # ------------------------------------------------------------------
307        # Test if all was right before proceeding further
308        # ------------------------------------------------------------------
309        IGCM_debug_Verif_Exit
310        # Save it
311        IGCM_sys_Put_Out ${output} ${R_SAVE}/$( echo $dir | sed "s:${R_BUFR}/::" )/${output}
312        # Clean file produced by ncrcat
313        IGCM_sys_Rm ${output}
314        # ------------------------------------------------------------------
315        # Test if all was right before proceeding further
316        # ------------------------------------------------------------------
317        IGCM_debug_Verif_Exit
318        # Clean files used by ncrcat
319        cat liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt | xargs rm
320        # Save the list of files that has been pack (ncrcat)
321        #mv liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt ${STORE_DEBUG}
322        IGCM_debug_Print 1 "Ncrcat and cleaning done for ${dir} and ${myType}"
323        echo
324      done
325    done
326  done
327  (( number_pack = number_pack + 1 ))
328  # Add 1 day to date_end_pack to have the new date_begin_pack
329  date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 )
330done
331
332# Execute C-ESM-EP driver (which launches jobs)
333if [ $PostCesmep = Pack ] ; then
334    ${SUBMIT_DIR}/cesmep_lite/libIGCM_post.sh $DateBegin $DateEnd
335fi
336
337# Flush post-processing submission
338if [ -f ${R_BUFR}/FlushPost_${DateEnd}.ksh ] ; then
339  . ${R_BUFR}/FlushPost_${DateEnd}.ksh
340  IGCM_FlushPost
341  #IGCM_sys_Rm -f ${R_BUFR}/FlushPost_${DateEnd}.ksh
342fi
343
344# Clean RUN_DIR_PATH (necessary for cesium and titane only)
345IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
346
347# ------------------------------------------------------------------
348# Finalize BigBrother to inform that the jobs end
349# ------------------------------------------------------------------
350IGCM_debug_BigBro_Finalize
351
352date
Note: See TracBrowser for help on using the repository browser.