source: trunk/libIGCM/AA_create_ts @ 29

Last change on this file since 29 was 29, checked in by sdipsl, 16 years ago

SD,MM : - Continue merging AllPostFred? branch and trunk

  • This give more flexibility to PeriodLenght? (*Y|y, *M|m, *D|d)
  • Need more test concerning userchoices consistency
  • only create_se is missing now
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 25.6 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J TS                     # Nom du job
6#-Q- platine #BSUB -N                        # message a la fin du job
7#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
8#-Q- platine #BSUB -W 1:00                   # Limite temps
9#-Q- platine #BSUB -q post              # Passage en queue post
10#-Q- sx8brodie #######################
11#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
12#-Q- sx8brodie #######################
13#-Q- sx8brodie #QSUB -r TS                   # Nom du job
14#-Q- sx8brodie #QSUB -lT 1:00:00             # limite en temps total
15#-Q- sx8brodie #QSUB -lt 1:00:00             # limite en temps par process
16#-Q- sx8brodie #QSUB -lM 1Gb 
17#-Q- sx8brodie #QSUB -eo                     # rassemble standard error et output
18#-Q- sx8brodie #QSUB -J m
19#-Q- sx8brodie #QSUB -l mpp_p=4
20#-Q- sx8brodie #QSUB -s /bin/ksh             # shell du job
21#-Q- sx8mercure #!/bin/ksh
22#-Q- sx8mercure ######################
23#-Q- sx8mercure ## SX8MERCURE   CEA ##
24#-Q- sx8mercure ######################
25#-Q- sx8mercure #PBS -N TS                   # Nom du job
26#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
27#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
28#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
29#-Q- sx8mercure #PBS -l cputim_job=2:00:00   # Limite temps a 2 heures
30#-Q- sx8mercure #PBS -q scalaire
31#-Q- default #!/bin/ksh
32#-Q- default ##################
33#-Q- default ## DEFAULT HOST ##
34#-Q- default ##################
35
36#set -vx
37
38date
39
40#-Q- sx8brodie export OMP_NUM_THREADS=1
41
42# $Date$
43# $Author$
44# $Revision$
45# IPSL (2006)
46#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
47
48########################################################################
49
50#D- Flag to determine if this job in a standalone mode
51#D- Default : value from AA_job if any
52StandAlone=${StandAlone:=true}
53
54#D- Flag to determine job's output directory
55#D- Default : value from libIGCM_post.ksh if any
56POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
57
58#D- Increased verbosity (1, 2, 3)
59#D- Default : value from AA_job if any
60Verbosity=${Verbosity:=3}
61
62#D- Low level debug : to bypass lib test checks and stack construction
63#D- Default : value from AA_job if any
64DEBUG_debug=${DEBUG_debug:=false}
65
66#D- Low level debug : to bypass lib test checks and stack construction
67#D- Default : value from AA_job if any
68libIGCM=${libIGCM:=/work/cont003/p86denv/PARA_SX8_ORCA2xLMD144142/Prelude/modipsl/libIGCM}
69
70#D- TEMPORARY Flag to determine atmospheric resolution
71#D- Default : value from atmospheric driver if any
72RESOL_ATM=${RESOL_ATM:=LMD144142}
73
74#D- TEMPORARY Flag to determine ocean resolution
75#D- Default : value from ocean driver if any
76RESOL_OCE=${RESOL_OCE:=ORCA2}
77
78#D- TEMPORARY Flag to determine ice resolution
79#D- Default : value from ice driver if any
80RESOL_ICE=${RESOL_ICE:=ORCA2}
81
82#D- Flag to determine surface resolution
83#D- Default : value from surface driver if any
84RESOL_SRF=${RESOL_SRF:=LMD144142}
85
86########################################################################
87
88. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh ;
89. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh     #; IGCM_debug_Check
90. ${libIGCM}/libIGCM_card/libIGCM_card.ksh   #; IGCM_card_Check
91. ${libIGCM}/libIGCM_date/libIGCM_date.ksh   #; IGCM_date_Check
92
93########################################################################
94
95#set -vx
96
97# Max number of files we use with a command (ex : ncrcat, ...)
98UNIX_MAX_LIMIT=120
99
100# Useful cleaning function
101MENAGE () 
102{ 
103    [ -f $3 ] & ( rm -f $1/$2 ;) 
104}
105
106########################################################################
107
108#set -vx
109
110if [ ${DEBUG_debug} = true ] ; then
111    IGCM_sys_MkdirWork ${RUN_DIR_PATH}
112    echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
113fi
114IGCM_sys_Cd ${RUN_DIR_PATH}
115
116if [ ${StandAlone} = true ] ; then
117    CARD_DIR=${SUBMIT_DIR}
118else
119    CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
120    IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
121fi
122
123#
124# First of all
125#
126IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
127typeset option
128for option in ${config_UserChoices[*]} ; do
129    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
130done
131#
132echo
133IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
134IGCM_debug_PrintVariables 3 config_UserChoices_JobName
135IGCM_debug_PrintVariables 3 config_UserChoices_LongName
136IGCM_debug_PrintVariables 3 config_UserChoices_TagName
137IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
138IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
139IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
140IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
141echo
142
143#==================================
144
145R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
146#
147IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card ListOfComponents
148#
149# Determine period for time series. Default : value from AA_job if any
150#
151DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
152#
153PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
154
155########################################################################
156#
157#      Depending on requested time series fill following variables :
158#
159#                           - LISTE_FILE_${comp}[*]
160#                           - LISTE_VARS_${file}[*]
161#                           - LISTE_AXIS_${file}[*]
162#                           - LISTE_PATCH_${file}[*]
163########################################################################
164
165#
166# For each selected component determine which files need post-processing
167#
168DoJob=false
169for comp in ${config_ListOfComponents[*]} ; do
170    #
171    ActiveComp=false
172    #
173    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
174    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
175    #
176    card=${CARD_DIR}/COMP/${compname}.card
177    IGCM_card_DefineArrayFromOption ${card} OutputFiles List
178    #
179    ListFilesName=${compname}_OutputFiles_List
180    eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
181    #
182    if [ X${FileName0} != X${NULL_STR} ] ; then
183        #
184        #IGCM_debug_Print 1 "Component      : ${compname}"
185        #
186        # INITIALISATION
187        #
188        eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
189        typeset i=0
190        #
191        until [ $i -eq $NbFiles ]; do
192            #
193            eval file_in_=\${${ListFilesName}[$i]} > /dev/null 2>&1
194            eval file_in=${file_in_}
195            let $(( i_ = i+2 ))
196            eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
197            eval flag_post=${flag_post_}
198            #
199            if [ X${flag_post} != XNONE ] ; then
200                #
201                # Initialize ${flag_post}_${compname}_${post_freq}
202                #
203                eval ${flag_post}_${compname}_TimeSeriesFrequency=true
204            fi
205            let $(( i=i+3 ))
206        done
207    fi
208
209    for element in $( set | grep Post_ | grep ${compname}_TimeSeriesFrequency | grep -i "=true" ) ; do
210        #
211        FILE=$( echo ${element} | awk -F "_${compname}" '{print $1}' | awk "-FPost_" '{print $2}' )
212        #
213        # For each comp and each file who needs post-processing
214        # determine which variables need time series.
215        #
216        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars
217        #
218        if [ $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars} ) = ${NULL_STR} ] ; then
219            #
220            IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
221            continue
222            #
223        fi
224        #
225        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
226        #
227        # So we need coordinates axis to properly fill produced time series
228        #
229        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
230        #
231        if [ $( eval echo \$\{${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
232            #
233            IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
234            IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
235            continue
236            #
237        fi
238        #
239        # CREATE LIST
240        #
241        FlagDir=$( echo ${FILE} | awk -F "_" '{print $1}' )
242        case ${FlagDir} in
243            *Y)  TS_Dir=TS_YE  ;;
244            *M)  TS_Dir=TS_MO  ;;
245            *D)  TS_Dir=TS_DA  ;;
246            HF)  TS_Dir=TS_HF  ;;
247            INS) TS_Dir=TS_INS ;;
248        esac
249
250        count=0
251        for var in $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars[*]} ) ; do
252            #
253            DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
254            TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
255            #
256            if [ ! -f ${DIRECTORY}/${TestedFile} ] ; then
257                eval set +A LISTE_VARS_${FILE} \$\{LISTE_VARS_${FILE}[*]} ${var}
258                ActiveComp=true
259                (( count = count + 1 ))
260            fi
261        done
262
263        if [ ${count} -gt 0 ] ; then
264            #
265            eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
266            eval set +A LISTE_AXIS_${FILE} \$\{${compname}_Post_${FILE}_GatherWithInternal[*]}
267            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
268            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
269                for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
270                    if [ Xload${Patch} != X ] ; then
271                        . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
272                        eval load${Patch}=loaded
273                    fi
274                done
275            fi
276            #
277        fi
278    done
279
280    if [ ${ActiveComp} = true ] ; then
281        set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
282    fi
283
284done
285
286set -A config_ListOfComponents ${ActiveListOfComponents[*]}
287[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
288
289########################################################################
290#
291#                          DEFINE DATE FUNCTIONS
292#
293########################################################################
294
295# Give the end of the next $1 period from $2 date
296function create_ts_next_date {
297    IGCM_debug_PushStack "create_ts_next_date" $@
298    typeset year month Length
299    case ${config_UserChoices_PeriodLength} in
300        *Y|*y) 
301            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
302            IGCM_date_GetYearMonth ${1} year month
303            (( Length=0 ))
304            (( i=0 ))
305            until [ $i -ge $PeriodLengthInYears ] ; do
306                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) )) 
307                (( i=i+1 ))
308            done
309            ;;
310        *M|*m) 
311            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
312            IGCM_date_GetYearMonth ${1} year month
313            (( year0=year ))
314            if [ $(( month + 1 )) -lt 13 ] ; then
315                month0=$(( month + 1 ))
316            else
317                month0=$(( month + 1 - 12 ))
318                (( year = year0 + 1 ))
319            fi
320            (( Length=0 ))
321            (( i=0 ))
322            until [ $i -ge $PeriodLengthInMonths ] ; do
323                if [ $(( month0 + i )) -lt 13 ] ; then
324                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
325                else
326                    (( year = year0 + 1 ))
327                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
328                fi
329                (( i=i+1 ))
330            done
331            ;;
332        *D|*d)
333            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
334        *)
335            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
336            IGCM_debug_Verif_Exit_Post ;;
337    esac
338    echo ${Length}
339
340    IGCM_debug_PopStack "create_ts_next_date"
341}
342
343# Give the end of the actual $1 period from $2 date
344function create_ts_begin_date {
345    IGCM_debug_PushStack "create_ts_begin_date" $@
346    typeset year month Length
347    case ${config_UserChoices_PeriodLength} in
348        *Y|*y) 
349            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
350            IGCM_date_GetYearMonth ${1} year month
351            if [ X${2} = Xend ] ; then
352                (( year = year - PeriodLengthInYears + 1))
353            fi
354            (( Length=0 ))
355            (( i=0 ))
356            until [ $i -ge $PeriodLengthInYears ] ; do
357                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) )) 
358                (( i=i+1 ))
359            done
360            ;;
361        *M|*m) 
362            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
363            IGCM_date_GetYearMonth ${1} year month
364            if [ X${2} = Xend ] ; then
365                (( month = $month - $PeriodLengthInMonths + 1 ))
366            fi     
367            (( year0=$year ))
368            if [ $month -le 0 ] ; then
369                (( month = $month + 12 ))
370                year=$( printf "%04i\n" $(( year - 1 )) )
371            fi
372            month=$( printf "%02i\n" ${month} )
373            (( Length=0 ))
374            (( i=0 ))
375            until [ $i -ge $PeriodLengthInMonths ] ; do
376                if [ $(( month + i )) -lt 13 ] ; then
377                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
378                else
379                    (( year = year0 + 1 ))
380                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
381                fi
382                (( i=i+1 ))
383            done
384            ;;
385        *D|*d)
386            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
387        *)
388            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
389            IGCM_debug_Verif_Exit_Post ;;
390    esac
391    echo ${Length}
392
393    IGCM_debug_PopStack "create_ts_begin_date"
394}
395
396########################################################################
397#
398#               IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
399#                 IF SO BRING THEM IN THE WORKING DIRECTORY
400#
401# PS : Keep in mind that IGCM_sys_Get here is a weak link :
402#      - especially for IDRiS
403#      - no special protection against "command too long" !
404#      - libIGCM_sys should handle that or modify following code part
405#
406########################################################################
407
408if [ ${StandAlone} != true ] ; then
409    Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
410       ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
411       echo \${run_PostProcessing_TimeSeriesRunning}" )
412
413    if [ X${Running_Flag} = Xy ] ; then
414        # Time Series Job allready running
415        IGCM_debug_Print 1 "Time Series Job allready running exit"
416        exit
417    fi
418    # Now run.card know we are running
419    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
420    # Get information from last execution
421    CompletedFlag=$( IGCM_sys_RshMaster \
422        "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
423        echo \${run_PostProcessing_TimeSeriesCompleted}" )
424fi
425
426# Is it the first submission or not ?
427if [ X${CompletedFlag} = X ] ; then
428    # First Time Series Submission
429    FIRST_PASS=TRUE
430    Length=$( create_ts_begin_date ${DateBegin} begin)
431    DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
432else
433    # Last Completed Time Series
434    FIRST_PASS=FALSE
435    DATE_FIN_JOB_B=${CompletedFlag}
436    date=${DateBegin}_${DATE_FIN_JOB_B}
437    for comp in ${config_ListOfComponents[*]} ; do
438        #
439        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
440        #
441        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
442        #
443        set -A liste_file
444        #
445        file_path=${R_SAVE}/${comp}/Analyse
446        #
447        i=0
448        file=${NULL_STR}
449        #
450        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
451            #
452            # Determine in which output can we find file
453            # IE : Analyse/TS or Analyse/DA2TS or ...
454            #
455            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
456            case ${FlagDir} in
457                *Y)  TS_Dir=TS_YE  ;;
458                *M)  TS_Dir=TS_MO  ;;
459                *D)  TS_Dir=TS_DA  ;;
460                HF)  TS_Dir=TS_HF  ;;
461                INS) TS_Dir=TS_INS ;;
462            esac
463            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
464                #
465                set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
466                #
467            done
468        done
469        if [ X${file} != X${NULL_STR} ] ; then
470            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
471            IGCM_debug_Verif_Exit_Post
472        fi
473    done
474    #
475    LEVEL=0
476    #
477fi
478
479########################################################################
480#
481#                          DEFINE LOOP PARAMETERS
482#
483########################################################################
484
485DATE_COUNT=${DATE_FIN_JOB_B}
486DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
487
488( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 || NBRE_FILE_TOT=1
489
490if [ ${DoJob} = true ] ; then
491    while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
492        (( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
493        Length=$( create_ts_next_date ${DATE_COUNT} )
494        DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
495    done
496else
497    NBRE_FILE_TOT=0
498fi
499
500if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
501    NBRE_TOUR=1
502elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
503    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT} 
504else
505    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
506fi
507
508CURRENT_LOOP=1
509# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
510# WE HAVE 3 LEVELS
511#    - LEVEL 1 : JUST ONE LOOP TO COMPLETE
512#    - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
513#    - LEVEL 3 : WE ARE IN THE LAST LOOP
514
515while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] ; do
516     
517    if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then 
518        # - LEVEL 1 : JUST ONE LOOP TO COMPLETE
519        NBRE_FILE_LOOP=${NBRE_FILE_TOT}
520
521        if [ ${FIRST_PASS} = TRUE ] ; then
522            DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
523        else
524            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
525            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
526        fi
527
528        DATE_FIN=${PeriodDateEnd}
529        DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
530
531        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
532        [ -n "${LEVEL}" ] && FLAG_B="TRUE" || FLAG_B="FALSE"
533
534        LEVEL=1
535        DEPOT="TRUE"     
536    elif [ ${CURRENT_LOOP} -ne  ${NBRE_TOUR} ] ; then
537        # - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
538        NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
539
540        if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
541            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
542            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
543        elif [ -z "${DATE_COURANTE}" ] ; then
544            DATE_COURANTE=${DATE_FIN_JOB_B}
545        else
546            Length=$( create_ts_next_date ${DATE_FIN} )
547            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
548        fi
549         
550        (( TotLength = 0 ))
551        COMPTEUR=1
552        DATE_LOOP=${DATE_COURANTE}
553
554        while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
555            #
556            Length=$( create_ts_next_date ${DATE_LOOP} )
557            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
558            (( TotLength = TotLength + Length ))
559            (( COMPTEUR = COMPTEUR + 1 ))
560            #
561        done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
562        DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
563
564        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
565        [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" || [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" || FLAG_B="FALSE"
566        DEPOT="FALSE"
567        LEVEL=2
568    else
569        # - LEVEL 3 : WE ARE IN THE LAST LOOP
570        NBRE_FILE_LOOP=$( expr ${NBRE_FILE_TOT} % $(( UNIX_MAX_LIMIT * $(( CURRENT_LOOP - 1 )) )) )
571        [ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
572
573        Length=$( create_ts_next_date ${DATE_FIN} )
574        DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
575        DATE_FIN=${PeriodDateEnd}
576        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
577
578        LEVEL=3
579        DEPOT="TRUE"
580        FLAG_B="TRUE"
581    fi
582
583    # THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
584    for comp in ${config_ListOfComponents[*]} ; do
585        #
586        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
587        #
588        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
589        #
590        eval R_OUT_${comp}=${R_SAVE}/${comp}
591        #
592        i=0
593        #
594        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
595            #
596            # Determine in which output can we find file
597            # IE : Output/MO or Output/DA or ...
598            #
599            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
600            case ${FlagDir} in
601                *Y)    TS_Dir=TS_YE  ; FreqDir=YE  ;;
602                *M)    TS_Dir=TS_MO  ; FreqDir=MO  ;;
603                *D)    TS_Dir=TS_DA  ; FreqDir=DA  ;;
604                HF)    TS_Dir=TS_HF  ; FreqDir=HF  ;;
605                INS)   TS_Dir=TS_INS ; FreqDir=INS ;;
606                *) 
607                       IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
608                       IGCM_sys_Exit "Job create_ts" ;;
609            esac
610
611            # Initialize array
612            set -A liste_file
613            set -A liste_file_tmp
614
615            eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
616
617            COMPTEUR=1
618            DATE_LOOP=${DATE_COURANTE}
619
620            while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
621                #
622                Length1=$( create_ts_begin_date ${DATE_LOOP} end)
623                DATE_TAB=$(  IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
624
625                Length2=$( create_ts_next_date ${DATE_LOOP} end)
626                DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
627
628                set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
629                set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
630                (( COMPTEUR = COMPTEUR + 1 ))
631                #
632            done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
633            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
634
635            # Get selected files
636            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
637            eval ExitFlag=false
638            IGCM_debug_Verif_Exit_Post
639
640            # Apply IGCM_Patch if needed
641            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
642                for file_t in $( eval echo ${liste_file_tmp[*]} ); do           
643                    for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
644                        IGCM_${Patch} ${file_t}
645                    done
646                done
647            fi
648
649            # WE CAN CONCATENATE FILES AT THIS POINT
650            [ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
651            Length=$( create_ts_begin_date ${DATE_COURANTE} end)
652            DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
653            DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
654            DATE_BUILD_END=${DateBegin}_${DATE_FIN}
655
656            liste_coord=" "
657            for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
658              liste_coord=${liste_coord}${axis}","
659            done
660
661            # Time axis must be the last dimension
662            time_axis=${axis}
663
664            # SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
665            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
666                #
667                IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
668                #
669#                 # Change time axis name to be CF compliant
670#                 if [ ${time_axis} != time_counter ]; then
671#                   IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
672#                 fi
673                #
674                if [ "${FLAG_B}" = "TRUE" ] ; then
675                    # WE CONCATENATE WITH EXISTING FILES
676                    file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
677                    file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
678                    file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
679                    IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
680                    #
681                    IGCM_sys_Rm ${file1} ${file2}
682                fi
683                #
684                if [ "${DEPOT}" = "TRUE" ] ; then
685                    # WE PUT FINAL FILE ON FILE SYSTEM
686                    [ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
687                    #${ncatted} -a modulo,time_counter,c,c," " ${file_out}
688                    file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
689                    eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
690                    [ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
691                    IGCM_sys_Rm ${file_out}
692                    #
693                fi
694                #
695                eval ExitFlag=false
696                IGCM_debug_Verif_Exit_Post
697            done     # for var in ${liste_var} ; do
698            IGCM_sys_Rm ${liste_file_tmp[*]}
699        done         # for file in ${liste_file_brut} do
700    done             # for comp in ${config_ListOfComponents[*]} ; do
701    eval ExitFlag=false
702    IGCM_debug_Verif_Exit_Post
703    # GO AHEAD IN DEEPEST LOOP
704    (( CURRENT_LOOP = CURRENT_LOOP + 1 ))
705done                 # while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] do
706# FINISH
707
708# DODS copy
709for comp in ${config_ListOfComponents[*]} ; do
710    for TS_Dir in $( ls ${R_SAVE}/${comp}/Analyse/ 2>&1 | grep TS_ ) ; do
711        [ -d ${R_SAVE}/${comp}/Analyse/${TS_Dir} ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
712    done
713done
714
715# clean environment variables and export usefull one :
716if [ X$( hostname -s ) = Xrhodes ] ; then
717    LISTE_ENV=$( env | grep -iv MYLANG | grep -iv HOST | grep -iv LOGIN | grep -iv config_ | grep -iv R_ | grep -iv libIGCM | grep -iv PERIOD \
718        | grep -iv DMFDIR | grep -iv AN_| grep -iv YEARS | grep -iv ex | grep -iv RESOL_ | grep -iv PATH | grep -iv WORKDIR | grep -iv USER   \
719        | grep -iv LOGNAME | grep -iv PROFILE | grep -iv ENV | grep -iv TMPDIR | grep -iv POST_DIR | grep -iv StandAlone | grep -iv DB | awk -F= '{print $1}' )
720 
721    for variables in ${LISTE_ENV} ; do
722        unset ${variables}
723    done
724fi
725
726export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
727export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF
728
729listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF"
730export listVarEnv
731
732#D-
733# --------------------------------------------------------------------
734#D- Test if all was right
735# --------------------------------------------------------------------
736IGCM_debug_Verif_Exit_Post
737# --------------------------------------------------------------------
738#D- Submit MONITORING
739# --------------------------------------------------------------------
740IGCM_sys_QsubPost monitoring
741
742#D-
743# --------------------------------------------------------------------
744#D- Update the run.card
745# --------------------------------------------------------------------
746if [ ${StandAlone} != true ] ; then
747    # Put in run.card end period of time series
748    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
749    # We have finish the job !
750    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
751fi
752date
Note: See TracBrowser for help on using the repository browser.