source: branches/SmoothFiles/AA_create_ts

Last change on this file was 5, checked in by mmaipsl, 16 years ago

MM: correct Date, Author and Revision svn properties.
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 24.0 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J TS                     # Nom du job
6#-Q- platine #BSUB -N                        # message a la fin du job
7#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
8#-Q- platine #BSUB -W 1:00                   # Limite temps
9#-Q- sx8brodie #######################
10#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
11#-Q- sx8brodie #######################
12#-Q- sx8brodie #QSUB -r TS                   # Nom du job
13#-Q- sx8brodie #QSUB -lT 1:00:00             # limite en temps total
14#-Q- sx8brodie #QSUB -lt 1:00:00             # limite en temps par process
15#-Q- sx8brodie #QSUB -lM 1Gb 
16#-Q- sx8brodie #QSUB -eo                     # rassemble standard error et output
17#-Q- sx8brodie #QSUB -J m
18#-Q- sx8brodie #QSUB -l mpp_p=4
19#-Q- sx8brodie #QSUB -s /bin/ksh             # shell du job
20#-Q- sx8mercure #!/bin/ksh
21#-Q- sx8mercure ######################
22#-Q- sx8mercure ## SX8MERCURE   CEA ##
23#-Q- sx8mercure ######################
24#-Q- sx8mercure #PBS -N TS                   # Nom du job
25#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
26#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
27#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
28#-Q- sx8mercure #PBS -l cputim_job=2:00:00   # Limite temps a 2 heures
29#-Q- sx8mercure #PBS -q scalaire
30#-Q- default #!/bin/ksh
31#-Q- default ##################
32#-Q- default ## DEFAULT HOST ##
33#-Q- default ##################
34
35#set -vx
36
37date
38
39#-Q- sx8brodie export OMP_NUM_THREADS=1
40
41# $Date$
42# $Author$
43# $Revision$
44# IPSL (2006)
45#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
46
47########################################################################
48
49#D- Flag to determine if this job in a standalone mode
50#D- Default : value from AA_job if any
51StandAlone=${StandAlone:=true}
52
53#D- Flag to determine job's output directory
54#D- Default : value from libIGCM_post.ksh if any
55POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
56
57#D- Increased verbosity (1, 2, 3)
58#D- Default : value from AA_job if any
59Verbosity=${Verbosity:=3}
60
61#D- Low level debug : to bypass lib test checks and stack construction
62#D- Default : value from AA_job if any
63DEBUG_debug=${DEBUG_debug:=false}
64
65#D- Low level debug : to bypass lib test checks and stack construction
66#D- Default : value from AA_job if any
67libIGCM=${libIGCM:=/work/cont003/p86denv/PARA_SX8_ORCA2xLMD144142/Prelude/modipsl/libIGCM}
68
69#D- TEMPORARY Flag to determine atmospheric resolution
70#D- Default : value from atmospheric driver if any
71RESOL_ATM=${RESOL_ATM:=LMD144142}
72
73#D- TEMPORARY Flag to determine ocean resolution
74#D- Default : value from ocean driver if any
75RESOL_OCE=${RESOL_OCE:=ORCA2}
76
77#D- TEMPORARY Flag to determine ice resolution
78#D- Default : value from ice driver if any
79RESOL_ICE=${RESOL_ICE:=ORCA2}
80
81#D- Flag to determine surface resolution
82#D- Default : value from surface driver if any
83RESOL_SRF=${RESOL_SRF:=LMD144142}
84
85########################################################################
86
87. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh ;
88. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh     #; IGCM_debug_Check
89. ${libIGCM}/libIGCM_card/libIGCM_card.ksh   #; IGCM_card_Check
90. ${libIGCM}/libIGCM_date/libIGCM_date.ksh   #; IGCM_date_Check
91
92########################################################################
93
94#set -vx
95
96# Max number of files we use with a command (ex : ncrcat, ...)
97UNIX_MAX_LIMIT=120
98
99# Useful cleaning function
100MENAGE () 
101{ 
102    [ -f $3 ] & ( rm -f $1/$2 ;) 
103}
104
105########################################################################
106
107#set -vx
108
109if [ ${DEBUG_debug} = true ] ; then
110    IGCM_sys_MkdirWork ${RUN_DIR_PATH}
111    echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
112fi
113IGCM_sys_Cd ${RUN_DIR_PATH}
114
115if [ ${StandAlone} = true ] ; then
116    CARD_DIR=${SUBMIT_DIR}
117else
118    CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
119    IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
120fi
121
122#
123# First of all
124#
125IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
126typeset option
127for option in ${config_UserChoices[*]} ; do
128    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
129done
130#
131echo
132IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
133IGCM_debug_PrintVariables 3 config_UserChoices_JobName
134IGCM_debug_PrintVariables 3 config_UserChoices_LongName
135IGCM_debug_PrintVariables 3 config_UserChoices_TagName
136IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
137IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
138IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
139IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
140echo
141
142#==================================
143
144R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
145#
146IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card ListOfComponents
147#
148# Determine period for time series. Default : value from AA_job if any
149#
150DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
151#
152PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
153
154########################################################################
155#
156#      Depending on requested time series fill following variables :
157#
158#                           - LISTE_FILE_${comp}[*]
159#                           - LISTE_VARS_${file}[*]
160#                           - LISTE_AXIS_${file}[*]
161#                           - LISTE_PATCH_${file}[*]
162########################################################################
163
164#
165# For each selected component determine which files need post-processing
166#
167DoJob=false
168for comp in ${config_ListOfComponents[*]} ; do
169    #
170    ActiveComp=false
171    #
172    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
173    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
174    #
175    card=${CARD_DIR}/COMP/${compname}.card
176    IGCM_card_DefineArrayFromOption ${card} OutputFiles List
177    #
178    ListFilesName=${compname}_OutputFiles_List
179    eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
180    #
181    if [ X${FileName0} != X${NULL_STR} ] ; then
182        #
183        #IGCM_debug_Print 1 "Component      : ${compname}"
184        #
185        # INITIALISATION
186        #
187        eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
188        typeset i=0
189        #
190        until [ $i -eq $NbFiles ]; do
191            #
192            eval file_in_=\${${ListFilesName}[$i]} > /dev/null 2>&1
193            eval file_in=${file_in_}
194            let $(( i_ = i+2 ))
195            eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
196            eval flag_post=${flag_post_}
197            #
198            if [ X${flag_post} != XNONE ] ; then
199                #
200                # Initialize ${flag_post}_${compname}_${post_freq}
201                #
202                eval ${flag_post}_${compname}_TimeSeriesFrequency=true
203            fi
204            let $(( i=i+3 ))
205        done
206    fi
207
208    for element in $( set | grep Post_ | grep ${compname}_TimeSeriesFrequency | grep -i "=true" ) ; do
209        #
210        FILE=$( echo ${element} | awk -F "_${compname}" '{print $1}' | awk "-FPost_" '{print $2}' )
211        #
212        # For each comp and each file who needs post-processing
213        # determine which variables need time series.
214        #
215        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars
216        #
217        if [ $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars} ) = ${NULL_STR} ] ; then
218            #
219            IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
220            continue
221            #
222        fi
223        #
224        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
225        #
226        # So we need coordinates axis to properly fill produced time series
227        #
228        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
229        #
230        if [ $( eval echo \$\{${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
231            #
232            IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
233            IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
234            continue
235            #
236        fi
237        #
238        # CREATE LIST
239        #
240        FlagDir=$( echo ${FILE} | awk -F "_" '{print $1}' )
241        case ${FlagDir} in
242            1Y)    TS_Dir=TS_YE  ;;
243            1M)    TS_Dir=TS_MO  ;;
244            5D|1D) TS_Dir=TS_DA  ;;
245            HF)    TS_Dir=TS_HF  ;;
246            INS)   TS_Dir=TS_INS ;;
247        esac
248
249        count=0
250        for var in $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars[*]} ) ; do
251            #
252            DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
253            TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
254            #
255            if [ ! -f ${DIRECTORY}/${TestedFile} ] ; then
256                eval set +A LISTE_VARS_${FILE} \$\{LISTE_VARS_${FILE}[*]} ${var}
257                ActiveComp=true
258                (( count = count + 1 ))
259            fi
260        done
261
262        if [ ${count} -gt 0 ] ; then
263            #
264            eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
265            eval set +A LISTE_AXIS_${FILE} \$\{${compname}_Post_${FILE}_GatherWithInternal[*]}
266            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
267            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
268                for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
269                    if [ Xload${Patch} != X ] ; then
270                        . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
271                        eval load${Patch}=loaded
272                    fi
273                done
274            fi
275            #
276        fi
277    done
278
279    if [ ${ActiveComp} = true ] ; then
280        set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
281    fi
282
283done
284
285set -A config_ListOfComponents ${ActiveListOfComponents[*]}
286[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
287
288########################################################################
289#
290#                          DEFINE DATE FUNCTIONS
291#
292########################################################################
293
294# Give the end of the next $1 period from $2 date
295function create_ts_next_date {
296    IGCM_debug_PushStack "create_ts_next_date" $@
297    typeset year month Length
298    case ${1} in
299        1Y|1y)
300            IGCM_date_GetYearMonth ${2} year month
301            Length=$( IGCM_date_DaysInYear $(( year + 1 )) ) ;;
302        1M|1m)
303            IGCM_date_GetYearMonth ${2} year month
304            if [ $month -lt 12 ] ; then
305                Length=$( IGCM_date_DaysInMonth $year $(( month + 1 )) )
306            else
307                Length=$( IGCM_date_DaysInMonth $(( year + 1 )) 1 )
308            fi ;;
309        *D|*d)
310            Length=$( echo ${1} | sed -e "s/[dD]//" ) ;;
311        *)
312            IGCM_debug_Exit "create_ts " ${1} " invalid PeriodLength : choose in 1Y, 1M, 5D, 1D."
313            IGCM_debug_Verif_Exit_Post ;;
314    esac
315    echo ${Length}
316
317    IGCM_debug_PopStack "create_ts_next_date"
318}
319
320# Give the end of the actual $1 period from $2 date
321function create_ts_begin_date {
322    IGCM_debug_PushStack "create_ts_begin_date" $@
323    typeset year month Length
324    case ${1} in
325        1Y|1y)
326            IGCM_date_GetYearMonth ${2} year month
327            Length=$( IGCM_date_DaysInYear $year ) ;;
328        1M|1m)
329            IGCM_date_GetYearMonth ${2} year month
330            Length=$( IGCM_date_DaysInMonth $year $month ) ;;
331        *D|*d)
332            Length=$( echo ${1} | sed -e "s/[dD]//" ) ;;
333        *)
334            IGCM_debug_Exit "create_ts " ${1} " invalid PeriodLength : choose in 1Y, 1M, 5D, 1D."
335            IGCM_debug_Verif_Exit_Post ;;
336    esac
337    echo ${Length}
338
339    IGCM_debug_PopStack "create_ts_begin_date"
340}
341
342########################################################################
343#
344#               IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
345#                 IF SO BRING THEM IN THE WORKING DIRECTORY
346#
347# PS : Keep in mind that IGCM_sys_Get here is a weak link :
348#      - especially for IDRiS
349#      - no special protection against "command too long" !
350#      - libIGCM_sys should handle that or modify following code part
351#
352########################################################################
353
354if [ ${StandAlone} != true ] ; then
355    Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
356       ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
357       echo \${run_PostProcessing_TimeSeriesRunning}" )
358
359    if [ X${Running_Flag} = Xy ] ; then
360        # Time Series Job allready running
361        IGCM_debug_Print 1 "Time Series Job allready running exit"
362        exit
363    fi
364    # Now run.card know we are running
365    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
366    # Get information from last execution
367    CompletedFlag=$( IGCM_sys_RshMaster \
368        "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
369        echo \${run_PostProcessing_TimeSeriesCompleted}" )
370fi
371
372# Is it the first submission or not ?
373if [ X${CompletedFlag} = X ] ; then
374    # First Time Series Submission
375    FIRST_PASS=TRUE
376    Length=$( create_ts_begin_date ${config_UserChoices_PeriodLength} ${DateBegin} )
377    DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
378else
379    # Last Completed Time Series
380    FIRST_PASS=FALSE
381    DATE_FIN_JOB_B=${CompletedFlag}
382    date=${DateBegin}_${DATE_FIN_JOB_B}
383    for comp in ${config_ListOfComponents[*]} ; do
384        #
385        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
386        #
387        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
388        #
389        set -A liste_file
390        #
391        file_path=${R_SAVE}/${comp}/Analyse
392        #
393        i=0
394        file=${NULL_STR}
395        #
396        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
397            #
398            # Determine in which output can we find file
399            # IE : Analyse/TS or Analyse/DA2TS or ...
400            #
401            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
402            case ${FlagDir} in
403                1Y)    TS_Dir=TS_YE  ;;
404                1M)    TS_Dir=TS_MO  ;;
405                5D|1D) TS_Dir=TS_DA  ;;
406                HF)    TS_Dir=TS_HF  ;;
407                INS)   TS_Dir=TS_INS ;;
408            esac
409            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
410                #
411                set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
412                #
413            done
414        done
415        if [ X${file} != X${NULL_STR} ] ; then
416            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
417            IGCM_debug_Verif_Exit_Post
418        fi
419    done
420    #
421    LEVEL=0
422    #
423fi
424
425########################################################################
426#
427#                          DEFINE LOOP PARAMETERS
428#
429########################################################################
430
431DATE_COUNT=${DATE_FIN_JOB_B}
432DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
433
434( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 || NBRE_FILE_TOT=1
435
436if [ ${DoJob} = true ] ; then
437    while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
438        (( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
439        Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_COUNT} )
440        DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
441    done
442else
443    NBRE_FILE_TOT=0
444fi
445
446if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
447    NBRE_TOUR=1
448elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
449    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT} 
450else
451    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
452fi
453
454CURRENT_LOOP=1
455# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
456# WE HAVE 3 LEVELS
457#    - LEVEL 1 : JUST ONE LOOP TO COMPLETE
458#    - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
459#    - LEVEL 3 : WE ARE IN THE LAST LOOP
460
461while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] ; do
462     
463    if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then 
464        # - LEVEL 1 : JUST ONE LOOP TO COMPLETE
465        NBRE_FILE_LOOP=${NBRE_FILE_TOT}
466
467        if [ ${FIRST_PASS} = TRUE ] ; then
468            DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
469        else
470            Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_FIN_JOB_B_LOOP} )
471            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
472        fi
473
474        DATE_FIN=${PeriodDateEnd}
475        DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
476
477        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
478        [ -n "${LEVEL}" ] && FLAG_B="TRUE" || FLAG_B="FALSE"
479
480        LEVEL=1
481        DEPOT="TRUE"     
482    elif [ ${CURRENT_LOOP} -ne  ${NBRE_TOUR} ] ; then
483        # - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
484        NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
485
486        if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
487            Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_FIN_JOB_B_LOOP} )
488            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
489        elif [ -z "${DATE_COURANTE}" ] ; then
490            DATE_COURANTE=${DATE_FIN_JOB_B}
491        else
492            Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_FIN} )
493            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
494        fi
495         
496        (( TotLength = 0 ))
497        COMPTEUR=1
498        DATE_LOOP=${DATE_COURANTE}
499
500        while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
501            #
502            Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_LOOP} )
503            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
504            (( TotLength = TotLength + Length ))
505            (( COMPTEUR = COMPTEUR + 1 ))
506            #
507        done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
508        DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
509
510        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
511        [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" || [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" || FLAG_B="FALSE"
512        DEPOT="FALSE"
513        LEVEL=2
514    else
515        # - LEVEL 3 : WE ARE IN THE LAST LOOP
516        NBRE_FILE_LOOP=$( expr ${NBRE_FILE_TOT} % $(( UNIX_MAX_LIMIT * $(( CURRENT_LOOP - 1 )) )) )
517        [ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
518
519        Length=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_FIN} )
520        DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
521        DATE_FIN=${PeriodDateEnd}
522        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
523
524        LEVEL=3
525        DEPOT="TRUE"
526        FLAG_B="TRUE"
527    fi
528
529    # THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
530    for comp in ${config_ListOfComponents[*]} ; do
531        #
532        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
533        #
534        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
535        #
536        eval R_OUT_${comp}=${R_SAVE}/${comp}
537        #
538        i=0
539        #
540        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
541            #
542            # Determine in which output can we find file
543            # IE : Output/MO or Output/DA or ...
544            #
545            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
546            case ${FlagDir} in
547                1Y)    TS_Dir=TS_YE  ; FreqDir=AN  ;;
548                1M)    TS_Dir=TS_MO  ; FreqDir=MO  ;;
549                *D)    TS_Dir=TS_DA  ; FreqDir=DA  ;;
550                HF)    TS_Dir=TS_HF  ; FreqDir=HF  ;;
551                INS)   TS_Dir=TS_INS ; FreqDir=INS ;;
552                *) 
553                       IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
554                       IGCM_sys_Exit "Job create_ts" ;;
555            esac
556
557            # Initialize array
558            set -A liste_file
559            set -A liste_file_tmp
560
561            eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
562
563            COMPTEUR=1
564            DATE_LOOP=${DATE_COURANTE}
565
566            while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
567                #
568                Length1=$( create_ts_begin_date ${config_UserChoices_PeriodLength} ${DATE_LOOP} )
569                DATE_TAB=$(  IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
570
571                Length2=$( create_ts_next_date ${config_UserChoices_PeriodLength} ${DATE_LOOP} )
572                DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
573
574                set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
575                set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
576                (( COMPTEUR = COMPTEUR + 1 ))
577                #
578            done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
579            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
580
581            # Get selected files
582            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
583            eval ExitFlag=false
584            IGCM_debug_Verif_Exit_Post
585
586            # Apply IGCM_Patch if needed
587            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
588                for file_t in $( eval echo ${liste_file_tmp[*]} ); do           
589                    for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
590                        IGCM_${Patch} ${file_t}
591                    done
592                done
593            fi
594
595            # WE CAN CONCATENATE FILES AT THIS POINT
596            [ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
597            Length=$( create_ts_begin_date ${config_UserChoices_PeriodLength} ${DATE_COURANTE} )
598            DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
599            DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
600            DATE_BUILD_END=${DateBegin}_${DATE_FIN}
601
602            liste_coord=" "
603            for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
604              liste_coord=${liste_coord}${axis}","
605            done
606
607            # Time axis must be the last dimension
608            time_axis=${axis}
609
610            # SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
611            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
612                #
613                IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
614                #
615#                 # Change time axis name to be CF compliant
616#                 if [ ${time_axis} != time_counter ]; then
617#                   IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
618#                 fi
619                #
620                if [ "${FLAG_B}" = "TRUE" ] ; then
621                    # WE CONCATENATE WITH EXISTING FILES
622                    file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
623                    file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
624                    file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
625                    IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
626                    #
627                    IGCM_sys_Rm ${file1} ${file2}
628                fi
629                #
630                if [ "${DEPOT}" = "TRUE" ] ; then
631                    # WE PUT FINAL FILE ON FILE SYSTEM
632                    [ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
633                    #${ncatted} -a modulo,time_counter,c,c," " ${file_out}
634                    file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
635                    eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
636                    [ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
637                    IGCM_sys_Rm ${file_out}
638                    #
639                fi
640                #
641                eval ExitFlag=false
642                IGCM_debug_Verif_Exit_Post
643            done     # for var in ${liste_var} ; do
644            IGCM_sys_Rm ${liste_file_tmp[*]}
645        done         # for file in ${liste_file_brut} do
646    done             # for comp in ${config_ListOfComponents[*]} ; do
647    eval ExitFlag=false
648    IGCM_debug_Verif_Exit_Post
649    # GO AHEAD IN DEEPEST LOOP
650    (( CURRENT_LOOP = CURRENT_LOOP + 1 ))
651done                 # while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] do
652# FINISH
653
654# DODS copy
655for comp in ${config_ListOfComponents[*]} ; do
656    for TS_Dir in $( ls ${R_SAVE}/${comp}/Analyse/ 2>&1 | grep TS_ ) ; do
657        [ -d ${R_SAVE}/${comp}/Analyse/${TS_Dir} ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
658    done
659done
660
661# clean environment variables and export usefull one :
662if [ X$( hostname -s ) = Xrhodes ] ; then
663    LISTE_ENV=$( env | grep -iv MYLANG | grep -iv HOST | grep -iv LOGIN | grep -iv config_ | grep -iv R_ | grep -iv libIGCM | grep -iv PERIOD \
664        | grep -iv DMFDIR | grep -iv AN_| grep -iv YEARS | grep -iv ex | grep -iv RESOL_ | grep -iv PATH | grep -iv WORKDIR | grep -iv USER   \
665        | grep -iv LOGNAME | grep -iv PROFILE | grep -iv ENV | grep -iv TMPDIR | grep -iv POST_DIR | grep -iv StandAlone | grep -iv DB | awk -F= '{print $1}' )
666 
667    for variables in ${LISTE_ENV} ; do
668        unset ${variables}
669    done
670fi
671
672export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
673export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF
674
675listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF"
676export listVarEnv
677
678#D-
679# --------------------------------------------------------------------
680#D- Test if all was right
681# --------------------------------------------------------------------
682IGCM_debug_Verif_Exit_Post
683# --------------------------------------------------------------------
684#D- Submit MONITORING
685# --------------------------------------------------------------------
686IGCM_sys_QsubPost monitoring
687
688#D-
689# --------------------------------------------------------------------
690#D- Update the run.card
691# --------------------------------------------------------------------
692if [ ${StandAlone} != true ] ; then
693    # Put in run.card end period of time series
694    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
695    # We have finish the job !
696    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
697fi
698date
Note: See TracBrowser for help on using the repository browser.