source: tags/libIGCM_v1_5/AA_create_ts @ 326

Last change on this file since 326 was 229, checked in by mmaipsl, 14 years ago

Littles modifications in StandAlone? mode for post-treatment scripts.

Change libIGCM default path for StandAlone? :
Now ins_job will give libIGCM path on host for run
(add a warning for blind system that require mirroring).
In rebuild jobs :
1) change NbRebuildDir? default value to 1000

(ie "all" old rebuild that haven't been done).

2) Change name of internal 'PeriodDateBegin?' for "LastPeriodForRebuild?".
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 30.7 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ###################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ###################
5#-Q- platine #BSUB -J TS                     # Nom du job
6#-Q- platine #BSUB -N                        # message a la fin du job
7#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
8#-Q- platine #BSUB -W 1:00                   # Limite temps
9#-Q- platine #BSUB -q post                   # Passage en queue post
10#-Q- sx8brodie #!/bin/ksh
11#-Q- sx8brodie #######################
12#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
13#-Q- sx8brodie #######################
14#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16#-Q- sx8brodie # Nom du travail LoadLeveler
17#-Q- sx8brodie # @ job_name   = TS
18#-Q- sx8brodie # Fichier de sortie standard du travail       
19#-Q- sx8brodie # @ output     = $(job_name).$(jobid)
20#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21#-Q- sx8brodie # @ error      =  $(job_name).$(jobid)
22#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23#-Q- sx8brodie # @ notification = error
24#-Q- sx8brodie # @ environment  = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25#-Q- aix6 #!/bin/ksh
26#-Q- aix6 #######################
27#-Q- aix6 ##   VARGAS   IDRIS  ##
28#-Q- aix6 #######################
29#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30#-Q- aix6 # @ wall_clock_limit = 20:00:00
31#-Q- aix6 # Nom du travail LoadLeveler
32#-Q- aix6 # @ job_name   = TS
33#-Q- aix6 # Fichier de sortie standard du travail
34#-Q- aix6 # @ output     = $(job_name).$(jobid)
35#-Q- aix6 # Fichier de sortie d'erreur du travail
36#-Q- aix6 # @ error      =  $(job_name).$(jobid)
37#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38#-Q- aix6 # @ notification = error
39#-Q- aix6 # @ environment  = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40#-Q- aix6 # @ queue
41#-Q- sx8brodie # @ queue
42#-Q- sx8mercure #!/bin/ksh
43#-Q- sx8mercure ######################
44#-Q- sx8mercure ## SX8MERCURE   CEA ##
45#-Q- sx8mercure ######################
46#-Q- sx8mercure #PBS -N TS                   # Nom du job
47#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
48#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
49#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
50#-Q- sx8mercure #PBS -l cputim_job=24:00:00   # Limite temps a 2 heures
51#-Q- sx8mercure #PBS -q scalaire
52#-Q- sx9mercure #!/bin/ksh
53#-Q- sx9mercure ######################
54#-Q- sx9mercure ## SX9MERCURE   CEA ##
55#-Q- sx9mercure ######################
56#-Q- sx9mercure #MSUB -r TS             # Nom du job               
57#-Q- sx9mercure #MSUB -N 1              # Reservation du noeud
58#-Q- sx9mercure #MSUB -n 1              # Reservation du processus
59#-Q- sx9mercure #MSUB -T 86400          # Limite de temps elapsed du job
60#-Q- sx9mercure #MSUB -E "-j o"
61#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62#-Q- sx9mercure ##MSUB -e nco.out        # Sortie standard
63#-Q- sx9mercure ##MSUB -o nco.out        # Sortie standard
64#-Q- titane #!/bin/ksh
65#-Q- titane ######################
66#-Q- titane ## TITANE   CEA ##
67#-Q- titane ######################
68#-Q- titane #MSUB -r TS             # Nom du job               
69#-Q- titane #MSUB -N 1              # Reservation du noeud
70#-Q- titane #MSUB -n 1              # Reservation du processus
71#-Q- titane #MSUB -T 86400          # Limite de temps elapsed du job
72#-Q- titane #MSUB -E "-j o"
73#-Q- titane #MSUB -E "-S /bin/ksh"
74#-Q- titane ##MSUB -e nco.out        # Sortie standard
75#-Q- titane ##MSUB -o nco.out        # Sortie standard
76#-Q- default #!/bin/ksh
77#-Q- default ##################
78#-Q- default ## DEFAULT HOST ##
79#-Q- default ##################
80
81# $Date$
82# $Author$
83# $Revision$
84# IPSL (2006)
85#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
86
87#set -eu
88#set -vx
89
90date
91
92#-Q- sx8brodie export OMP_NUM_THREADS=1
93#-Q- aix6 export OMP_NUM_THREADS=1
94
95########################################################################
96
97#D- Flag to determine if this job in a standalone mode
98#D- Default : value from AA_job if any
99StandAlone=${StandAlone:=true}
100
101#D- Low level debug : to bypass lib test checks and stack construction
102#D- Default : value from AA_job if any
103libIGCM=${libIGCM:=::modipsl::/libIGCM}
104# WARNING for StandAlone used : To run this script on some machine,
105# you must check MirrorlibIGCM variable in sys library.
106# If this variable is true, you must use libIGCM_POST path instead
107# of your running libIGCM directory.
108
109#D- Flag to determine allready produced time series. Empty if you start from the beginning
110#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
111CompletedFlag=${CompletedFlag:=}
112
113#D- Flag to determine task type this script will perform.
114#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
115#D- Default : Value from AA_job if any. Usefull in StandAlone case.
116TsTask=${TsTask:=2D}
117
118#D- Flag to determine if rebuild process has been performed asynchronously
119#D- Possible value true or false.
120#D- If true raw files has allready been patched by rebuild job
121#D- If false create_ts will patch the raw files
122#D- Default : Value from AA_job if any. Usefull in StandAlone case.
123RebuildFrequency=${RebuildFrequency:=true}
124
125#D- Flag to determine job's output directory
126#D- Default : value from libIGCM_post.ksh if any
127POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
128
129#D- Increased verbosity (1, 2, 3)
130#D- Default : value from AA_job if any
131Verbosity=${Verbosity:=3}
132
133#D- Low level debug : to bypass lib test checks and stack construction
134#D- Default : value from AA_job if any
135DEBUG_debug=${DEBUG_debug:=false}
136
137#D- TEMPORARY Flag to determine atmospheric resolution
138#D- Default : value from atmospheric driver if any
139RESOL_ATM=${RESOL_ATM:=ALL}
140
141#D- TEMPORARY Flag to determine ocean resolution
142#D- Default : value from ocean driver if any
143RESOL_OCE=${RESOL_OCE:=ORCA2}
144
145#D- TEMPORARY Flag to determine ice resolution
146#D- Default : value from ice driver if any
147RESOL_ICE=${RESOL_ICE:=ORCA2}
148
149#D- TEMPORARY Flag to determine marine biogeochemistry resolution
150#D- Default : value from ice driver if any
151RESOL_MBG=${RESOL_MBG:=ORCA2}
152
153#D- Flag to determine surface resolution
154#D- Default : value from surface driver if any
155RESOL_SRF=ALL
156
157#D- Flag to determine surface resolution
158#D- Default : value from surface driver if any
159RESOL_SBG=ALL
160
161########################################################################
162
163. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
164      ( ${DEBUG_debug} ) && IGCM_debug_Check
165. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
166     ( ${DEBUG_debug} ) && IGCM_card_Check
167. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
168     ( ${DEBUG_debug} ) && IGCM_date_Check
169#-------
170. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
171
172########################################################################
173
174#set -vx
175
176# Useful cleaning function
177MENAGE () 
178{
179    [ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
180}
181
182########################################################################
183
184IGCM_sys_MkdirWork ${RUN_DIR_PATH}
185IGCM_sys_Cd ${RUN_DIR_PATH}
186
187if [ ${StandAlone} = true ] ; then
188    CARD_DIR=${SUBMIT_DIR}
189else
190    CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
191    IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
192fi
193
194#
195# First of all
196#
197IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
198typeset option
199for option in ${config_UserChoices[*]} ; do
200    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
201done
202#
203echo
204IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
205IGCM_debug_PrintVariables 3 config_UserChoices_JobName
206IGCM_debug_PrintVariables 3 config_UserChoices_LongName
207IGCM_debug_PrintVariables 3 config_UserChoices_TagName
208IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
209IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
210IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
211IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
212echo
213
214#==================================
215R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
216#
217# Determine component to take care of depending on kind of task create_ts will perform
218#
219( [ X${TsTask} = XChunck2D ] || [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} || IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
220#
221# Determine period for time series. Default : value from AA_job or from launch_TS if any
222#
223DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
224#
225PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
226#
227# Determine Dimension of the produced time series : 2D or 3D
228if (   [ X${TsTask} = XChunck2D ] || [ X${TsTask} = X2D ] ) ; then
229    Dimension=2D
230elif ( [ X${TsTask} = XChunck3D ] || [ X${TsTask} = X3D ] ) ; then
231    Dimension=3D
232else
233    Dimension=""
234fi
235
236
237########################################################################
238#
239#      Depending on requested time series fill following variables :
240#
241#                           - LISTE_FILE_${comp}[*]
242#                           - LISTE_VARS_${file}[*]
243#                           - LISTE_AXIS_${file}[*]
244#                           - LISTE_PATCH_${file}[*]
245########################################################################
246
247#
248# For each selected component determine which files need post-processing
249#
250DoJob=false
251for comp in ${config_ListOfComponents[*]} ; do
252    #
253    ActiveComp=false
254    #
255    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
256    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
257    #
258    card=${CARD_DIR}/COMP/${compname}.card
259
260    IGCM_card_DefineArrayFromOption ${card} OutputFiles List
261    #
262    ListFilesName=${compname}_OutputFiles_List
263    eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
264    #
265    if [ X${FileName0} != X${NULL_STR} ] ; then
266        #
267        #IGCM_debug_Print 1 "Component      : ${compname}"
268        #
269        # INITIALISATION
270        #
271        typeset i
272        if ( [ X${TsTask} = XChunck2D ] || [ X${TsTask} = XChunck3D ] ) ; then
273            ((NbFiles=${FlagToRead}))
274            ((i=${NbFiles}-2))
275        else
276            eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
277            i=0
278        fi
279        #
280        until [ $i -ge $NbFiles ]; do
281            #
282            (( i_ = i+2 ))
283            eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
284            eval flag_post=${flag_post_}
285            #
286            if [ X${flag_post} != XNONE ] ; then
287                #
288                # For each comp and each file who needs post-processing
289                # determine which variables need time series.
290                #
291                FILE=$( echo ${flag_post} | awk "-FPost_" '{print $2}' )
292                IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
293                #
294                if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
295                    #
296                    IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
297                    (( i=i+3 ))
298                    continue
299                    #
300                fi
301                #
302                # If we are not a Chunck type task, skip files/variables handled  by chunck task
303                #
304                if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
305                    IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
306                    if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
307                        IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
308                        IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
309                        (( i=i+3 ))
310                        continue
311                    fi
312                fi
313                #
314                #  We need coordinates axis to properly fill produced time series
315                #
316                IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
317                #
318                if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
319                    #
320                    IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
321                    IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
322                    (( i=i+3 ))
323                    continue
324                    #
325                fi
326                #
327                # We need LIST of variables not allready produced (useful for standalone mode)
328                #
329                FlagDir=$( echo ${FILE} | awk -F "_" '{print $1}' )
330                case ${FlagDir} in
331                    *Y)  TS_Dir=TS_YE  ;;
332                    *M)  TS_Dir=TS_MO  ;;
333                    *D)  TS_Dir=TS_DA  ;;
334                    HF)  TS_Dir=TS_HF  ;;
335                    INS) TS_Dir=TS_INS ;;
336                esac
337                #
338                count=0
339                for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
340                    #
341                    DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
342                    TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
343                    #
344                    IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
345                    if [ ! $? = 0 ] ; then
346                        eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
347                        ActiveComp=true
348                        (( count = count + 1 ))
349                    fi
350                done
351                #
352                if [ ${count} -gt 0 ] ; then
353                    #
354                    eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
355                    eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
356                    #
357                    if [ ! X${RebuildFrequency} = Xtrue ] ; then
358                        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
359                        eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
360                        if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
361                            for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
362                                if [ Xload${Patch} != X ] ; then
363                                    . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
364                                    eval load${Patch}=loaded
365                                fi
366                            done
367                        fi
368                    fi
369                fi
370            fi
371            (( i=i+3 ))
372        done
373    fi
374    #
375    if [ ${ActiveComp} = true ] ; then
376        set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
377    fi
378done
379
380set -A config_ListOfComponents ${ActiveListOfComponents[*]}
381[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
382
383########################################################################
384#
385#                          DEFINE DATE FUNCTIONS
386#
387########################################################################
388
389# Give the end of the next $1 period from $2 date
390function create_ts_next_date {
391    IGCM_debug_PushStack "create_ts_next_date" $@
392    typeset Length
393    case ${config_UserChoices_PeriodLength} in
394        *Y|*y) 
395            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
396            IGCM_date_GetYearMonth ${1} year month
397            (( Length=0 ))
398            (( i=0 ))
399            until [ $i -ge $PeriodLengthInYears ] ; do
400                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) )) 
401                (( i=i+1 ))
402            done
403            ;;
404        *M|*m) 
405            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
406            IGCM_date_GetYearMonth ${1} year month
407            (( year0 = year ))
408            if [ $(( month + 1 )) -lt 13 ] ; then
409                month0=$(( month + 1 ))
410            else
411                month0=$(( month + 1 - 12 ))
412                (( year = year0 + 1 ))
413            fi
414            (( Length=0 ))
415            (( i=0 ))
416            until [ $i -ge $PeriodLengthInMonths ] ; do
417                if [ $(( month0 + i )) -lt 13 ] ; then
418                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
419                else
420                    (( year = year0 + 1 ))
421                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
422                fi
423                (( i=i+1 ))
424            done
425            ;;
426        *D|*d)
427            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
428        *)
429            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
430            IGCM_debug_Verif_Exit_Post ;;
431    esac
432    echo ${Length}
433
434    IGCM_debug_PopStack "create_ts_next_date"
435}
436
437# Give the end of the actual $1 period from $2 date
438function create_ts_begin_date {
439    IGCM_debug_PushStack "create_ts_begin_date" $@
440    typeset Length
441    case ${config_UserChoices_PeriodLength} in
442        *Y|*y) 
443            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
444            IGCM_date_GetYearMonth ${1} year month
445            if [ X${2} = Xend ] ; then
446                (( year = year - PeriodLengthInYears + 1))
447            fi
448            (( Length=0 ))
449            (( i=0 ))
450            until [ $i -ge $PeriodLengthInYears ] ; do
451                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) )) 
452                (( i=i+1 ))
453            done
454            ;;
455        *M|*m) 
456            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
457            IGCM_date_GetYearMonth ${1} year month
458            if [ X${2} = Xend ] ; then
459                (( month = month - PeriodLengthInMonths + 1 ))
460            fi     
461            (( year0 = year ))
462            if [ $month -le 0 ] ; then
463                (( month = month + 12 ))
464                year=$( printf "%04i\n" $(( year - 1 )) )
465            fi
466            month=$( printf "%02i\n" ${month} )
467            (( Length=0 ))
468            (( i=0 ))
469            until [ $i -ge $PeriodLengthInMonths ] ; do
470                if [ $(( month + i )) -lt 13 ] ; then
471                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
472                else
473                    (( year = year0 + 1 ))
474                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
475                fi
476                (( i=i+1 ))
477            done
478            ;;
479        *D|*d)
480            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
481        *)
482            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
483            IGCM_debug_Verif_Exit_Post ;;
484    esac
485    echo ${Length}
486
487    IGCM_debug_PopStack "create_ts_begin_date"
488}
489
490########################################################################
491#
492#               IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
493#                 IF SO BRING THEM IN THE WORKING DIRECTORY
494#
495# PS : Keep in mind that IGCM_sys_Get here is a weak link :
496#      - especially for IDRiS
497#      - no special protection against "command too long" !
498#      - libIGCM_sys should handle that or modify following code part
499#
500########################################################################
501
502if [ ${StandAlone} != true ] ; then
503    Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
504       ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
505       echo \${run_PostProcessing_TimeSeriesRunning}" )
506
507    #if [ X${Running_Flag} = Xy ] ; then
508    #    # Time Series Job allready running
509    #   IGCM_debug_Print 1 "Time Series Job allready running exit"
510    #   exit
511    #fi
512    # Now run.card know we are running
513
514
515
516
517    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
518
519
520
521
522
523    # Get information from last execution
524    #CompletedFlag=$( IGCM_sys_RshMaster \
525    #   "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
526    #        echo \${run_PostProcessing_TimeSeriesCompleted}" )
527fi
528
529# Is it the first submission or not ?
530
531if ( [ X${CompletedFlag} = X ] || [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
532    # First Time Series Submission
533    FIRST_PASS=TRUE
534    Length=$( create_ts_begin_date ${DateBegin} begin)
535    DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
536else
537    # Last Completed Time Series
538    FIRST_PASS=FALSE
539    DATE_FIN_JOB_B=${CompletedFlag}
540    date=${DateBegin}_${DATE_FIN_JOB_B}
541    for comp in ${config_ListOfComponents[*]} ; do
542        #
543        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
544        #
545        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
546        #
547        #set -A liste_file
548        unset liste_file
549        #
550        file_path=${R_SAVE}/${comp}/Analyse
551        #
552        i=0
553        file=${NULL_STR}
554        #
555        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
556            #
557            # Determine in which output can we find file
558            # IE : Analyse/TS or Analyse/DA2TS or ...
559            #
560            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
561            case ${FlagDir} in
562                *Y)  TS_Dir=TS_YE  ;;
563                *M)  TS_Dir=TS_MO  ;;
564                *D)  TS_Dir=TS_DA  ;;
565                HF)  TS_Dir=TS_HF  ;;
566                INS) TS_Dir=TS_INS ;;
567            esac
568            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
569                #
570                #set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
571                liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
572                #
573            done
574        done
575        if [ X${file} != X${NULL_STR} ] ; then
576            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
577            ExitFlag=false
578            IGCM_debug_Verif_Exit_Post
579        fi
580    done
581    #
582    LEVEL=0
583    #
584fi
585
586########################################################################
587#
588#                          DEFINE LOOP PARAMETERS
589#
590########################################################################
591
592DATE_COUNT=${DATE_FIN_JOB_B}
593DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
594
595( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 || NBRE_FILE_TOT=1
596
597if [ ${DoJob} = true ] ; then
598    while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
599        (( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
600        Length=$( create_ts_next_date ${DATE_COUNT} )
601        DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
602    done
603else
604    NBRE_FILE_TOT=0
605fi
606
607if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
608    NBRE_TOUR=1
609elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
610    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT} 
611else
612    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
613fi
614
615CURRENT_LOOP=1
616# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
617# WE HAVE 3 LEVELS
618#    - LEVEL 1 : JUST ONE LOOP TO COMPLETE
619#    - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
620#    - LEVEL 3 : WE ARE IN THE LAST LOOP
621
622while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] ; do
623     
624    if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then 
625        # - LEVEL 1 : JUST ONE LOOP TO COMPLETE
626        NBRE_FILE_LOOP=${NBRE_FILE_TOT}
627
628        if [ ${FIRST_PASS} = TRUE ] ; then
629            DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
630        else
631            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
632            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
633        fi
634
635        DATE_FIN=${PeriodDateEnd}
636        DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
637
638        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
639        [ -n "${LEVEL}" ] && FLAG_B="TRUE" || FLAG_B="FALSE"
640
641        LEVEL=1
642        DEPOT="TRUE"     
643    elif [ ${CURRENT_LOOP} -ne  ${NBRE_TOUR} ] ; then
644        # - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
645        NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
646
647        if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
648            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
649            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
650        elif [ -z "${DATE_COURANTE}" ] ; then
651            DATE_COURANTE=${DATE_FIN_JOB_B}
652        else
653            Length=$( create_ts_next_date ${DATE_FIN} )
654            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
655        fi
656         
657        (( TotLength = 0 ))
658        COMPTEUR=1
659        DATE_LOOP=${DATE_COURANTE}
660
661        while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
662            #
663            Length=$( create_ts_next_date ${DATE_LOOP} )
664            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
665            (( TotLength = TotLength + Length ))
666            (( COMPTEUR = COMPTEUR + 1 ))
667            #
668        done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
669        DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
670
671        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
672        [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" || [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" || FLAG_B="FALSE"
673        DEPOT="FALSE"
674        LEVEL=2
675    else
676        # - LEVEL 3 : WE ARE IN THE LAST LOOP
677        NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
678        [ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
679
680        Length=$( create_ts_next_date ${DATE_FIN} )
681        DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
682        DATE_FIN=${PeriodDateEnd}
683        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
684
685        LEVEL=3
686        DEPOT="TRUE"
687        FLAG_B="TRUE"
688    fi
689
690    # THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
691    for comp in ${config_ListOfComponents[*]} ; do
692        #
693        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
694        #
695        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
696        #
697        eval R_OUT_${comp}=${R_SAVE}/${comp}
698        #
699        i=0
700        #
701        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
702            #
703            # Determine in which output can we find file
704            # IE : Output/MO or Output/DA or ...
705            #
706            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
707            case ${FlagDir} in
708                *Y)    TS_Dir=TS_YE  ; FreqDir=YE  ;;
709                *M)    TS_Dir=TS_MO  ; FreqDir=MO  ;;
710                *D)    TS_Dir=TS_DA  ; FreqDir=DA  ;;
711                HF)    TS_Dir=TS_HF  ; FreqDir=HF  ;;
712                INS)   TS_Dir=TS_INS ; FreqDir=INS ;;
713                *) 
714                       IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
715                       IGCM_sys_Exit "Job create_ts" ;;
716            esac
717
718            # Initialize array
719            #set -A liste_file
720            #set -A liste_file_tmp
721            unset liste_file
722            unset liste_file_tmp
723            eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
724
725            COMPTEUR=1
726            DATE_LOOP=${DATE_COURANTE}
727
728            while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
729                #
730                Length1=$( create_ts_begin_date ${DATE_LOOP} end)
731                DATE_TAB=$(  IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
732
733                Length2=$( create_ts_next_date ${DATE_LOOP} end)
734                DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
735
736                #set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
737                #set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
738                liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
739                liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
740                (( COMPTEUR = COMPTEUR + 1 ))
741                #
742            done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
743            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
744
745            # Get selected files
746            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
747            ExitFlag=false
748            IGCM_debug_Verif_Exit_Post
749
750            # Apply IGCM_Patch if needed
751            if [ ! X${RebuildFrequency} = Xtrue ] ; then
752                if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
753                    for file_t in $( eval echo ${liste_file_tmp[*]} ); do               
754                        for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
755                            IGCM_${Patch} ${file_t}
756                        done
757                    done
758                fi
759            fi
760
761            # WE CAN CONCATENATE FILES AT THIS POINT
762            [ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
763            Length=$( create_ts_begin_date ${DATE_COURANTE} end)
764            DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
765            DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
766            DATE_BUILD_END=${DateBegin}_${DATE_FIN}
767
768            liste_coord=" "
769            for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
770              liste_coord=${liste_coord}${axis}","
771            done
772
773            # Time axis must be the last dimension
774            time_axis=${axis}
775
776            # SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
777            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
778                #
779                IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
780                #
781#                 # Change time axis name to be CF compliant
782#                 if [ ${time_axis} != time_counter ]; then
783#                   IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
784#                 fi
785                #
786                if [ "${FLAG_B}" = "TRUE" ] ; then
787                    # WE CONCATENATE WITH EXISTING FILES
788                    file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
789                    file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
790                    file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
791                    IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
792                    #
793                    IGCM_sys_Rm ${file1} ${file2}
794                fi
795                #
796                if [ "${DEPOT}" = "TRUE" ] ; then
797                    # WE PUT FINAL FILE ON FILE SYSTEM
798                    [ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
799                    #${ncatted} -a modulo,time_counter,c,c," " ${file_out}
800                    file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
801                    eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
802                    [ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
803                    IGCM_sys_Rm ${file_out}
804                    #
805                fi
806                #
807                ExitFlag=false
808                IGCM_debug_Verif_Exit_Post
809            done     # for var in ${liste_var} ; do
810            IGCM_sys_Rm ${liste_file_tmp[*]}
811        done         # for file in ${liste_file_brut} do
812    done             # for comp in ${config_ListOfComponents[*]} ; do
813    ExitFlag=false
814    IGCM_debug_Verif_Exit_Post
815    # GO AHEAD IN DEEPEST LOOP
816    (( CURRENT_LOOP = CURRENT_LOOP + 1 ))
817done                 # while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] do
818# FINISH
819
820# DODS copy
821for comp in ${config_ListOfComponents[*]} ; do
822    for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
823        IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
824        [ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
825    done
826done
827
828export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
829export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
830
831listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
832export listVarEnv
833
834#D-
835# --------------------------------------------------------------------
836#D- Test if all was right
837# --------------------------------------------------------------------
838IGCM_debug_Verif_Exit_Post
839# --------------------------------------------------------------------
840#D- Submit MONITORING
841# --------------------------------------------------------------------
842( [ X${TsTask} = X2D ] || [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
843
844#D-
845# --------------------------------------------------------------------
846#D- Update the run.card
847# --------------------------------------------------------------------
848if [ ${StandAlone} != true ] ; then
849    # Put in run.card end period of time series
850    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
851    # We have finish the job !
852    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
853fi
854
855# Clean RUN_DIR_PATH (necessary for cesium)
856IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
857
858date
Note: See TracBrowser for help on using the repository browser.