source: branches/AllPostFred/AA_create_ts @ 13

Last change on this file since 13 was 10, checked in by mmaipsl, 16 years ago

MM: All frequencies in post-treatement.
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 26.1 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J TS                     # Nom du job
6#-Q- platine #BSUB -N                        # message a la fin du job
7#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
8#-Q- platine #BSUB -W 1:00                   # Limite temps
9#-Q- sx8brodie #######################
10#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
11#-Q- sx8brodie #######################
12#-Q- sx8brodie #QSUB -r TS                   # Nom du job
13#-Q- sx8brodie #QSUB -lT 1:00:00             # limite en temps total
14#-Q- sx8brodie #QSUB -lt 1:00:00             # limite en temps par process
15#-Q- sx8brodie #QSUB -lM 1Gb 
16#-Q- sx8brodie #QSUB -eo                     # rassemble standard error et output
17#-Q- sx8brodie #QSUB -J m
18#-Q- sx8brodie #QSUB -l mpp_p=4
19#-Q- sx8brodie #QSUB -s /bin/ksh             # shell du job
20#-Q- sx8mercure #!/bin/ksh
21#-Q- sx8mercure ######################
22#-Q- sx8mercure ## SX8MERCURE   CEA ##
23#-Q- sx8mercure ######################
24#-Q- sx8mercure #PBS -N TS                   # Nom du job
25#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
26#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
27#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
28#-Q- sx8mercure #PBS -l cputim_job=2:00:00   # Limite temps a 2 heures
29#-Q- sx8mercure #PBS -q scalaire
30#-Q- default #!/bin/ksh
31#-Q- default ##################
32#-Q- default ## DEFAULT HOST ##
33#-Q- default ##################
34#-Q- default date
35
36#set -vx
37
38date
39
40#-Q- sx8brodie export OMP_NUM_THREADS=1
41
42# $Date$
43# $Author$
44# $Revision$
45# IPSL (2006)
46#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
47
48########################################################################
49
50#D- Flag to determine if this job in a standalone mode
51#D- Default : value from AA_job if any
52StandAlone=${StandAlone:=true}
53
54#D- Flag to determine job's output directory
55#D- Default : value from libIGCM_post.ksh if any
56POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
57
58#D- Increased verbosity (1, 2, 3)
59#D- Default : value from AA_job if any
60Verbosity=${Verbosity:=3}
61
62#D- Low level debug : to bypass lib test checks and stack construction
63#D- Default : value from AA_job if any
64DEBUG_debug=${DEBUG_debug:=false}
65
66#D- Low level debug : to bypass lib test checks and stack construction
67#D- Default : value from AA_job if any
68libIGCM=${libIGCM:=/work/cont003/p86denv/PARA_SX8_ORCA2xLMD144142/Prelude/modipsl/libIGCM}
69
70#D- TEMPORARY Flag to determine atmospheric resolution
71#D- Default : value from atmospheric driver if any
72RESOL_ATM=${RESOL_ATM:=LMD144142}
73
74#D- TEMPORARY Flag to determine ocean resolution
75#D- Default : value from ocean driver if any
76RESOL_OCE=${RESOL_OCE:=ORCA2}
77
78#D- TEMPORARY Flag to determine ice resolution
79#D- Default : value from ice driver if any
80RESOL_ICE=${RESOL_ICE:=ORCA2}
81
82#D- Flag to determine surface resolution
83#D- Default : value from surface driver if any
84RESOL_SRF=${RESOL_SRF:=LMD144142}
85
86########################################################################
87
88. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh ;
89. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh     #; IGCM_debug_Check
90. ${libIGCM}/libIGCM_card/libIGCM_card.ksh   #; IGCM_card_Check
91. ${libIGCM}/libIGCM_date/libIGCM_date.ksh   #; IGCM_date_Check
92
93########################################################################
94
95#set -vx
96
97# Max number of files we use with a command (ex : ncrcat, ...)
98UNIX_MAX_LIMIT=120
99
100# Useful cleaning function
101MENAGE () 
102{ 
103    [ -f $3 ] & ( rm -f $1/$2 ;) 
104}
105
106########################################################################
107
108#set -vx
109
110if [ ${DEBUG_debug} = true ] ; then
111    IGCM_sys_MkdirWork ${RUN_DIR_PATH}
112    echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
113fi
114IGCM_sys_Cd ${RUN_DIR_PATH}
115
116if [ ${StandAlone} = true ] ; then
117    CARD_DIR=${SUBMIT_DIR}
118else
119    CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
120    IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
121fi
122
123#
124# First of all
125#
126IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
127typeset option
128for option in ${config_UserChoices[*]} ; do
129    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
130done
131#
132echo
133IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
134IGCM_debug_PrintVariables 3 config_UserChoices_JobName
135IGCM_debug_PrintVariables 3 config_UserChoices_LongName
136IGCM_debug_PrintVariables 3 config_UserChoices_TagName
137IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
138IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
139IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
140IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
141echo
142
143#==================================
144
145R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
146#
147IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card ListOfComponents
148#
149# Determine period for time series. Default : value from AA_job if any
150#
151DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
152#
153PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
154
155########################################################################
156#
157#      Depending on requested time series fill following variables :
158#
159#                           - LISTE_FILE_${comp}[*]
160#                           - LISTE_VARS_${file}[*]
161#                           - LISTE_AXIS_${file}[*]
162#                           - LISTE_PATCH_${file}[*]
163########################################################################
164
165#
166# For each selected component determine which files need post-processing
167#
168DoJob=false
169for comp in ${config_ListOfComponents[*]} ; do
170    #
171    ActiveComp=false
172    #
173    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
174    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
175    #
176    card=${CARD_DIR}/COMP/${compname}.card
177    IGCM_card_DefineArrayFromOption ${card} OutputFiles List
178    #
179    ListFilesName=${compname}_OutputFiles_List
180    eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
181    #
182    if [ X${FileName0} != X${NULL_STR} ] ; then
183        #
184        #IGCM_debug_Print 1 "Component      : ${compname}"
185        #
186        # INITIALISATION
187        #
188        eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
189        typeset i=0
190        #
191        until [ $i -eq $NbFiles ]; do
192            #
193            eval file_in_=\${${ListFilesName}[$i]} > /dev/null 2>&1
194            eval file_in=${file_in_}
195            let $(( i_ = i+2 ))
196            eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
197            eval flag_post=${flag_post_}
198            #
199            if [ X${flag_post} != XNONE ] ; then
200                #
201                # Initialize ${flag_post}_${compname}_${post_freq}
202                #
203                eval ${flag_post}_${compname}_TimeSeriesFrequency=true
204            fi
205            let $(( i=i+3 ))
206        done
207    fi
208
209    for element in $( set | grep Post_ | grep ${compname}_TimeSeriesFrequency | grep -i "=true" ) ; do
210        #
211        FILE=$( echo ${element} | awk -F "_${compname}" '{print $1}' | awk "-FPost_" '{print $2}' )
212        #
213        # For each comp and each file who needs post-processing
214        # determine which variables need time series.
215        #
216        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars
217        #
218        if [ $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars} ) = ${NULL_STR} ] ; then
219            #
220            IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
221            continue
222            #
223        fi
224        #
225        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
226        #
227        # So we need coordinates axis to properly fill produced time series
228        #
229        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
230        #
231        if [ $( eval echo \$\{${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
232            #
233            IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
234            IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
235            continue
236            #
237        fi
238        #
239        # CREATE LIST
240        #
241        FlagDir=$( echo ${FILE} | awk -F "_" '{print $1}' )
242        case ${FlagDir} in
243            *Y)    TS_Dir=TS_YE  ;;
244            *M)    TS_Dir=TS_MO  ;;
245            5D|1D|*D) TS_Dir=TS_DA  ;;
246            HF)    TS_Dir=TS_HF  ;;
247            INS)   TS_Dir=TS_INS ;;
248        esac
249
250        count=0
251        for var in $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars[*]} ) ; do
252            #
253            DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
254            TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
255            #
256            if [ ! -f ${DIRECTORY}/${TestedFile} ] ; then
257                eval set +A LISTE_VARS_${FILE} \$\{LISTE_VARS_${FILE}[*]} ${var}
258                ActiveComp=true
259                (( count = count + 1 ))
260            fi
261        done
262
263        if [ ${count} -gt 0 ] ; then
264            #
265            eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
266            eval set +A LISTE_AXIS_${FILE} \$\{${compname}_Post_${FILE}_GatherWithInternal[*]}
267            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
268            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
269                for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
270                    if [ Xload${Patch} != X ] ; then
271                        . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
272                        eval load${Patch}=loaded
273                    fi
274                done
275            fi
276            #
277        fi
278    done
279
280    if [ ${ActiveComp} = true ] ; then
281        set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
282    fi
283
284done
285
286set -A config_ListOfComponents ${ActiveListOfComponents[*]}
287[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
288
289########################################################################
290#
291#                          DEFINE DATE FUNCTIONS
292#
293########################################################################
294
295# Give the end of the next $1 period from $2 date
296function create_ts_next_date {
297    IGCM_debug_PushStack "create_ts_next_date" $@
298    typeset year month Length
299    case ${config_UserChoices_PeriodLength} in
300        1Y|1y)
301            IGCM_date_GetYearMonth ${1} year month
302            Length=$( IGCM_date_DaysInYear $(( year + 1 )) ) ;;
303        *Y|*y) 
304            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
305            IGCM_date_GetYearMonth ${1} year month
306            (( Length=0 ))
307            (( i=0 ))
308            until [ $i -ge $PeriodLengthInYears ] ; do
309                (( Length = Length + $( IGCM_date_DaysInYear $(( year + PeriodLengthInYears + i )) ) )) 
310                (( i=i+1 ))
311            done
312            ;;
313        1M|1m)
314            IGCM_date_GetYearMonth ${1} year month
315            if [ $month -lt 12 ] ; then
316                Length=$( IGCM_date_DaysInMonth $year $(( month + 1 )) )
317            else
318                Length=$( IGCM_date_DaysInMonth $(( year + 1 )) 1 )
319            fi ;;
320        *M|*m) 
321            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
322            IGCM_date_GetYearMonth ${1} year month
323            (( year0=year ))
324            if [ $(( month + 1 )) -lt 13 ] ; then
325                month0=$(( month + 1 ))
326            else
327                month0=$(( month + 1 - 12 ))
328                (( year = year0 + 1 ))
329            fi
330            (( Length=0 ))
331            (( i=0 ))
332            until [ $i -ge $PeriodLengthInMonths ] ; do
333                if [ $(( month0 + i )) -lt 13 ] ; then
334                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
335                else
336                    (( year = year0 + 1 ))
337                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
338                fi
339                (( i=i+1 ))
340            done
341            ;;
342        *D|*d)
343            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
344        *)
345            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
346            IGCM_debug_Verif_Exit_Post ;;
347    esac
348    echo ${Length}
349
350    IGCM_debug_PopStack "create_ts_next_date"
351}
352
353# Give the end of the actual $1 period from $2 date
354function create_ts_begin_date {
355    IGCM_debug_PushStack "create_ts_begin_date" $@
356    typeset year month Length
357    case ${config_UserChoices_PeriodLength} in
358        1Y|1y)
359            IGCM_date_GetYearMonth ${1} year month
360            Length=$( IGCM_date_DaysInYear $year ) ;;
361        *Y|*y) 
362            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
363            IGCM_date_GetYearMonth ${1} year month
364            if [ X${2} = Xend ] ; then
365                (( year = year - PeriodLengthInYears + 1))
366            fi
367            (( Length=0 ))
368            (( i=0 ))
369            until [ $i -ge $PeriodLengthInYears ] ; do
370                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) )) 
371                (( i=i+1 ))
372            done
373            ;;
374        1M|1m)
375            IGCM_date_GetYearMonth ${1} year month
376            Length=$( IGCM_date_DaysInMonth $year $month ) ;;
377        *M|*m) 
378            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
379            IGCM_date_GetYearMonth ${1} year month
380            if [ X${2} = Xend ] ; then
381                (( month = $month - $PeriodLengthInMonths + 1 ))
382            fi     
383            (( year0=$year ))
384            if [ $month -le 0 ] ; then
385                (( month = $month + 12 ))
386                year=$( printf "%04i\n" $(( year - 1 )) )
387            fi
388            month=$( printf "%02i\n" ${month} )
389            (( Length=0 ))
390            (( i=0 ))
391            until [ $i -ge $PeriodLengthInMonths ] ; do
392                if [ $(( month + i )) -lt 13 ] ; then
393                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
394                else
395                    (( year = year0 + 1 ))
396                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
397                fi
398                (( i=i+1 ))
399            done
400            ;;
401        *D|*d)
402            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
403        *)
404            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
405            IGCM_debug_Verif_Exit_Post ;;
406    esac
407    echo ${Length}
408
409    IGCM_debug_PopStack "create_ts_begin_date"
410}
411
412########################################################################
413#
414#               IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
415#                 IF SO BRING THEM IN THE WORKING DIRECTORY
416#
417# PS : Keep in mind that IGCM_sys_Get here is a weak link :
418#      - especially for IDRiS
419#      - no special protection against "command too long" !
420#      - libIGCM_sys should handle that or modify following code part
421#
422########################################################################
423
424if [ ${StandAlone} != true ] ; then
425    Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
426       ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
427       echo \${run_PostProcessing_TimeSeriesRunning}" )
428
429    if [ X${Running_Flag} = Xy ] ; then
430        # Time Series Job allready running
431        IGCM_debug_Print 1 "Time Series Job allready running exit"
432        exit
433    fi
434    # Now run.card know we are running
435    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
436    # Get information from last execution
437    CompletedFlag=$( IGCM_sys_RshMaster \
438        "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
439        echo \${run_PostProcessing_TimeSeriesCompleted}" )
440fi
441
442# Is it the first submission or not ?
443if [ X${CompletedFlag} = X ] ; then
444    # First Time Series Submission
445    FIRST_PASS=TRUE
446    Length=$( create_ts_begin_date ${DateBegin} begin)
447    DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
448else
449    # Last Completed Time Series
450    FIRST_PASS=FALSE
451    DATE_FIN_JOB_B=${CompletedFlag}
452    date=${DateBegin}_${DATE_FIN_JOB_B}
453    for comp in ${config_ListOfComponents[*]} ; do
454        #
455        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
456        #
457        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
458        #
459        set -A liste_file
460        #
461        file_path=${R_SAVE}/${comp}/Analyse
462        #
463        i=0
464        file=${NULL_STR}
465        #
466        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
467            #
468            # Determine in which output can we find file
469            # IE : Analyse/TS or Analyse/DA2TS or ...
470            #
471            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
472            case ${FlagDir} in
473                *Y)    TS_Dir=TS_YE  ;;
474                *M)    TS_Dir=TS_MO  ;;
475                5D|1D|*D) TS_Dir=TS_DA  ;;
476                HF)    TS_Dir=TS_HF  ;;
477                INS)   TS_Dir=TS_INS ;;
478            esac
479            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
480                #
481                set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
482                #
483            done
484        done
485        if [ X${file} != X${NULL_STR} ] ; then
486            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
487            IGCM_debug_Verif_Exit_Post
488        fi
489    done
490    #
491    LEVEL=0
492    #
493fi
494
495########################################################################
496#
497#                          DEFINE LOOP PARAMETERS
498#
499########################################################################
500
501DATE_COUNT=${DATE_FIN_JOB_B}
502DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
503
504( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 || NBRE_FILE_TOT=1
505
506if [ ${DoJob} = true ] ; then
507    while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
508        (( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
509        Length=$( create_ts_next_date ${DATE_COUNT} )
510        DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
511    done
512else
513    NBRE_FILE_TOT=0
514fi
515
516if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
517    NBRE_TOUR=1
518elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
519    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT} 
520else
521    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
522fi
523
524CURRENT_LOOP=1
525# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
526# WE HAVE 3 LEVELS
527#    - LEVEL 1 : JUST ONE LOOP TO COMPLETE
528#    - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
529#    - LEVEL 3 : WE ARE IN THE LAST LOOP
530
531while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] ; do
532     
533    if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then 
534        # - LEVEL 1 : JUST ONE LOOP TO COMPLETE
535        NBRE_FILE_LOOP=${NBRE_FILE_TOT}
536
537        if [ ${FIRST_PASS} = TRUE ] ; then
538            DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
539        else
540            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
541            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
542        fi
543
544        DATE_FIN=${PeriodDateEnd}
545        DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
546
547        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
548        [ -n "${LEVEL}" ] && FLAG_B="TRUE" || FLAG_B="FALSE"
549
550        LEVEL=1
551        DEPOT="TRUE"     
552    elif [ ${CURRENT_LOOP} -ne  ${NBRE_TOUR} ] ; then
553        # - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
554        NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
555
556        if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
557            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
558            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
559        elif [ -z "${DATE_COURANTE}" ] ; then
560            DATE_COURANTE=${DATE_FIN_JOB_B}
561        else
562            Length=$( create_ts_next_date ${DATE_FIN} )
563            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
564        fi
565         
566        (( TotLength = 0 ))
567        COMPTEUR=1
568        DATE_LOOP=${DATE_COURANTE}
569
570        while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
571            #
572            Length=$( create_ts_next_date ${DATE_LOOP} )
573            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
574            (( TotLength = TotLength + Length ))
575            (( COMPTEUR = COMPTEUR + 1 ))
576            #
577        done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
578        DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
579
580        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
581        [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" || [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" || FLAG_B="FALSE"
582        DEPOT="FALSE"
583        LEVEL=2
584    else
585        # - LEVEL 3 : WE ARE IN THE LAST LOOP
586        NBRE_FILE_LOOP=$( expr ${NBRE_FILE_TOT} % $(( UNIX_MAX_LIMIT * $(( CURRENT_LOOP - 1 )) )) )
587        [ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
588
589        Length=$( create_ts_next_date ${DATE_FIN} )
590        DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
591        DATE_FIN=${PeriodDateEnd}
592        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
593
594        LEVEL=3
595        DEPOT="TRUE"
596        FLAG_B="TRUE"
597    fi
598
599    # THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
600    for comp in ${config_ListOfComponents[*]} ; do
601        #
602        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
603        #
604        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
605        #
606        eval R_OUT_${comp}=${R_SAVE}/${comp}
607        #
608        i=0
609        #
610        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
611            #
612            # Determine in which output can we find file
613            # IE : Output/MO or Output/DA or ...
614            #
615            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
616            case ${FlagDir} in
617                *Y)    TS_Dir=TS_YE  ; FreqDir=YE  ;;
618                *M)    TS_Dir=TS_MO  ; FreqDir=MO  ;;
619                *D)    TS_Dir=TS_DA  ; FreqDir=DA  ;;
620                HF)    TS_Dir=TS_HF  ; FreqDir=HF  ;;
621                INS)   TS_Dir=TS_INS ; FreqDir=INS ;;
622                *) 
623                       IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
624                       IGCM_sys_Exit "Job create_ts" ;;
625            esac
626
627            # Initialize array
628            set -A liste_file
629            set -A liste_file_tmp
630
631            eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
632
633            COMPTEUR=1
634            DATE_LOOP=${DATE_COURANTE}
635
636            while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
637                #
638                Length1=$( create_ts_begin_date ${DATE_LOOP} end)
639                DATE_TAB=$(  IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
640
641                Length2=$( create_ts_next_date ${DATE_LOOP} end)
642                DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
643
644                set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
645                set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
646                (( COMPTEUR = COMPTEUR + 1 ))
647                #
648            done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
649            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
650
651            # Get selected files
652            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
653            eval ExitFlag=false
654            IGCM_debug_Verif_Exit_Post
655
656            # Apply IGCM_Patch if needed
657            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
658                for file_t in $( eval echo ${liste_file_tmp[*]} ); do           
659                    for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
660                        IGCM_${Patch} ${file_t}
661                    done
662                done
663            fi
664
665            # WE CAN CONCATENATE FILES AT THIS POINT
666            [ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
667            Length=$( create_ts_begin_date ${DATE_COURANTE} end)
668            DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
669            DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
670            DATE_BUILD_END=${DateBegin}_${DATE_FIN}
671
672            liste_coord=" "
673            for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
674              liste_coord=${liste_coord}${axis}","
675            done
676
677            # Time axis must be the last dimension
678            time_axis=${axis}
679
680            # SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
681            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
682                #
683                IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
684                #
685#                 # Change time axis name to be CF compliant
686#                 if [ ${time_axis} != time_counter ]; then
687#                   IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
688#                 fi
689                #
690                if [ "${FLAG_B}" = "TRUE" ] ; then
691                    # WE CONCATENATE WITH EXISTING FILES
692                    file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
693                    file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
694                    file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
695                    IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
696                    #
697                    IGCM_sys_Rm ${file1} ${file2}
698                fi
699                #
700                if [ "${DEPOT}" = "TRUE" ] ; then
701                    # WE PUT FINAL FILE ON FILE SYSTEM
702                    [ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
703                    #${ncatted} -a modulo,time_counter,c,c," " ${file_out}
704                    file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
705                    eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
706                    [ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
707                    IGCM_sys_Rm ${file_out}
708                    #
709                fi
710                #
711                eval ExitFlag=false
712                IGCM_debug_Verif_Exit_Post
713            done     # for var in ${liste_var} ; do
714            IGCM_sys_Rm ${liste_file_tmp[*]}
715        done         # for file in ${liste_file_brut} do
716    done             # for comp in ${config_ListOfComponents[*]} ; do
717    eval ExitFlag=false
718    IGCM_debug_Verif_Exit_Post
719    # GO AHEAD IN DEEPEST LOOP
720    (( CURRENT_LOOP = CURRENT_LOOP + 1 ))
721done                 # while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] do
722# FINISH
723
724# DODS copy
725for comp in ${config_ListOfComponents[*]} ; do
726    for TS_Dir in $( ls ${R_SAVE}/${comp}/Analyse/ 2>&1 | grep TS_ ) ; do
727        [ -d ${R_SAVE}/${comp}/Analyse/${TS_Dir} ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
728    done
729done
730
731# clean environment variables and export usefull one :
732if [ X$( hostname -s ) = Xrhodes ] ; then
733    LISTE_ENV=$( env | grep -iv MYLANG | grep -iv HOST | grep -iv LOGIN | grep -iv config_ | grep -iv R_ | grep -iv libIGCM | grep -iv PERIOD \
734        | grep -iv DMFDIR | grep -iv AN_| grep -iv YEARS | grep -iv ex | grep -iv RESOL_ | grep -iv PATH | grep -iv WORKDIR | grep -iv USER   \
735        | grep -iv LOGNAME | grep -iv PROFILE | grep -iv ENV | grep -iv TMPDIR | grep -iv POST_DIR | grep -iv StandAlone | grep -iv DB | awk -F= '{print $1}' )
736 
737    for variables in ${LISTE_ENV} ; do
738        unset ${variables}
739    done
740fi
741
742export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
743export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF
744
745listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF"
746export listVarEnv
747
748#D-
749# --------------------------------------------------------------------
750#D- Test if all was right
751# --------------------------------------------------------------------
752IGCM_debug_Verif_Exit_Post
753# --------------------------------------------------------------------
754#D- Submit MONITORING
755# --------------------------------------------------------------------
756IGCM_sys_QsubPost monitoring
757
758#D-
759# --------------------------------------------------------------------
760#D- Update the run.card
761# --------------------------------------------------------------------
762if [ ${StandAlone} != true ] ; then
763    # Put in run.card end period of time series
764    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
765    # We have finish the job !
766    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
767fi
768date
Note: See TracBrowser for help on using the repository browser.