source: trunk/libIGCM/AA_create_ts @ 58

Last change on this file since 58 was 58, checked in by sdipsl, 15 years ago

Headers for LoadLeveler? 3.5
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 25.8 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J TS                     # Nom du job
6#-Q- platine #BSUB -N                        # message a la fin du job
7#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
8#-Q- platine #BSUB -W 1:00                   # Limite temps
9#-Q- platine #BSUB -q post              # Passage en queue post
10#-Q- sx8brodie #######################
11#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
12#-Q- sx8brodie #######################
13#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
14#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
15#-Q- sx8brodie # Nom du travail LoadLeveler
16#-Q- sx8brodie # @ job_name   = Sortie   
17#-Q- sx8brodie # Fichier de sortie standard du travail       
18#-Q- sx8brodie # @ output     = $(job_name).$(jobid)
19#-Q- sx8brodie # Fichier de sortie d'erreur du travail
20#-Q- sx8brodie # @ error      =  $(job_name).$(jobid)
21#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
22#-Q- sx8brodie # @ notification = error
23#-Q- sx8brodie # @ environment  = COPY_ALL
24#-Q- sx8brodie # @ queue
25#-Q- sx8brodie #!/bin/ksh
26#-Q- sx8mercure #!/bin/ksh
27#-Q- sx8mercure ######################
28#-Q- sx8mercure ## SX8MERCURE   CEA ##
29#-Q- sx8mercure ######################
30#-Q- sx8mercure #PBS -N TS                   # Nom du job
31#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
32#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
33#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
34#-Q- sx8mercure #PBS -l cputim_job=2:00:00   # Limite temps a 2 heures
35#-Q- sx8mercure #PBS -q scalaire
36#-Q- default #!/bin/ksh
37#-Q- default ##################
38#-Q- default ## DEFAULT HOST ##
39#-Q- default ##################
40
41#set -vx
42
43date
44
45#-Q- sx8brodie export OMP_NUM_THREADS=1
46
47# $Date$
48# $Author$
49# $Revision$
50# IPSL (2006)
51#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
52
53########################################################################
54
55#D- Flag to determine if this job in a standalone mode
56#D- Default : value from AA_job if any
57StandAlone=${StandAlone:=true}
58
59#D- Flag to determine job's output directory
60#D- Default : value from libIGCM_post.ksh if any
61POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
62
63#D- Increased verbosity (1, 2, 3)
64#D- Default : value from AA_job if any
65Verbosity=${Verbosity:=3}
66
67#D- Low level debug : to bypass lib test checks and stack construction
68#D- Default : value from AA_job if any
69DEBUG_debug=${DEBUG_debug:=false}
70
71#D- Low level debug : to bypass lib test checks and stack construction
72#D- Default : value from AA_job if any
73libIGCM=${libIGCM:=/work/cont003/p86denv/PARA_SX8_ORCA2xLMD144142/Prelude/modipsl/libIGCM}
74
75#D- TEMPORARY Flag to determine atmospheric resolution
76#D- Default : value from atmospheric driver if any
77RESOL_ATM=${RESOL_ATM:=LMD144142}
78
79#D- TEMPORARY Flag to determine ocean resolution
80#D- Default : value from ocean driver if any
81RESOL_OCE=${RESOL_OCE:=ORCA2}
82
83#D- TEMPORARY Flag to determine ice resolution
84#D- Default : value from ice driver if any
85RESOL_ICE=${RESOL_ICE:=ORCA2}
86
87#D- Flag to determine surface resolution
88#D- Default : value from surface driver if any
89RESOL_SRF=${RESOL_SRF:=LMD144142}
90
91########################################################################
92
93. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh ;
94. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh     #; IGCM_debug_Check
95. ${libIGCM}/libIGCM_card/libIGCM_card.ksh   #; IGCM_card_Check
96. ${libIGCM}/libIGCM_date/libIGCM_date.ksh   #; IGCM_date_Check
97
98########################################################################
99
100#set -vx
101
102# Max number of files we use with a command (ex : ncrcat, ...)
103UNIX_MAX_LIMIT=120
104
105# Useful cleaning function
106MENAGE () 
107{ 
108    [ -f $3 ] & ( rm -f $1/$2 ;) 
109}
110
111########################################################################
112
113#set -vx
114
115if [ ${DEBUG_debug} = true ] ; then
116    IGCM_sys_MkdirWork ${RUN_DIR_PATH}
117    echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
118fi
119IGCM_sys_Cd ${RUN_DIR_PATH}
120
121if [ ${StandAlone} = true ] ; then
122    CARD_DIR=${SUBMIT_DIR}
123else
124    CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
125    IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
126fi
127
128#
129# First of all
130#
131IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
132typeset option
133for option in ${config_UserChoices[*]} ; do
134    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
135done
136#
137echo
138IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
139IGCM_debug_PrintVariables 3 config_UserChoices_JobName
140IGCM_debug_PrintVariables 3 config_UserChoices_LongName
141IGCM_debug_PrintVariables 3 config_UserChoices_TagName
142IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
143IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
144IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
145IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
146echo
147
148#==================================
149
150R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
151#
152IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card ListOfComponents
153#
154# Determine period for time series. Default : value from AA_job if any
155#
156DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
157#
158PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
159
160########################################################################
161#
162#      Depending on requested time series fill following variables :
163#
164#                           - LISTE_FILE_${comp}[*]
165#                           - LISTE_VARS_${file}[*]
166#                           - LISTE_AXIS_${file}[*]
167#                           - LISTE_PATCH_${file}[*]
168########################################################################
169
170#
171# For each selected component determine which files need post-processing
172#
173DoJob=false
174for comp in ${config_ListOfComponents[*]} ; do
175    #
176    ActiveComp=false
177    #
178    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
179    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
180    #
181    card=${CARD_DIR}/COMP/${compname}.card
182    IGCM_card_DefineArrayFromOption ${card} OutputFiles List
183    #
184    ListFilesName=${compname}_OutputFiles_List
185    eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
186    #
187    if [ X${FileName0} != X${NULL_STR} ] ; then
188        #
189        #IGCM_debug_Print 1 "Component      : ${compname}"
190        #
191        # INITIALISATION
192        #
193        eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
194        typeset i=0
195        #
196        until [ $i -eq $NbFiles ]; do
197            #
198            eval file_in_=\${${ListFilesName}[$i]} > /dev/null 2>&1
199            eval file_in=${file_in_}
200            let $(( i_ = i+2 ))
201            eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
202            eval flag_post=${flag_post_}
203            #
204            if [ X${flag_post} != XNONE ] ; then
205                #
206                # Initialize ${flag_post}_${compname}_${post_freq}
207                #
208                eval ${flag_post}_${compname}_TimeSeriesFrequency=true
209            fi
210            let $(( i=i+3 ))
211        done
212    fi
213
214    for element in $( set | grep Post_ | grep ${compname}_TimeSeriesFrequency | grep -i "=true" ) ; do
215        #
216        FILE=$( echo ${element} | awk -F "_${compname}" '{print $1}' | awk "-FPost_" '{print $2}' )
217        #
218        # For each comp and each file who needs post-processing
219        # determine which variables need time series.
220        #
221        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars
222        #
223        if [ $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars} ) = ${NULL_STR} ] ; then
224            #
225            IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
226            continue
227            #
228        fi
229        #
230        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
231        #
232        # So we need coordinates axis to properly fill produced time series
233        #
234        IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
235        #
236        if [ $( eval echo \$\{${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
237            #
238            IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
239            IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
240            continue
241            #
242        fi
243        #
244        # CREATE LIST
245        #
246        FlagDir=$( echo ${FILE} | awk -F "_" '{print $1}' )
247        case ${FlagDir} in
248            *Y)  TS_Dir=TS_YE  ;;
249            *M)  TS_Dir=TS_MO  ;;
250            *D)  TS_Dir=TS_DA  ;;
251            HF)  TS_Dir=TS_HF  ;;
252            INS) TS_Dir=TS_INS ;;
253        esac
254
255        count=0
256        for var in $( eval echo \$\{${compname}_Post_${FILE}_TimeSeriesVars[*]} ) ; do
257            #
258            DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
259            TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
260            #
261            if [ ! -f ${DIRECTORY}/${TestedFile} ] ; then
262                eval set +A LISTE_VARS_${FILE} \$\{LISTE_VARS_${FILE}[*]} ${var}
263                ActiveComp=true
264                (( count = count + 1 ))
265            fi
266        done
267
268        if [ ${count} -gt 0 ] ; then
269            #
270            eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
271            eval set +A LISTE_AXIS_${FILE} \$\{${compname}_Post_${FILE}_GatherWithInternal[*]}
272            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
273            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
274                for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
275                    if [ Xload${Patch} != X ] ; then
276                        . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
277                        eval load${Patch}=loaded
278                    fi
279                done
280            fi
281            #
282        fi
283    done
284
285    if [ ${ActiveComp} = true ] ; then
286        set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
287    fi
288
289done
290
291set -A config_ListOfComponents ${ActiveListOfComponents[*]}
292[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
293
294########################################################################
295#
296#                          DEFINE DATE FUNCTIONS
297#
298########################################################################
299
300# Give the end of the next $1 period from $2 date
301function create_ts_next_date {
302    IGCM_debug_PushStack "create_ts_next_date" $@
303    typeset year month Length
304    case ${config_UserChoices_PeriodLength} in
305        *Y|*y) 
306            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
307            IGCM_date_GetYearMonth ${1} year month
308            (( Length=0 ))
309            (( i=0 ))
310            until [ $i -ge $PeriodLengthInYears ] ; do
311                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) )) 
312                (( i=i+1 ))
313            done
314            ;;
315        *M|*m) 
316            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
317            IGCM_date_GetYearMonth ${1} year month
318            (( year0=year ))
319            if [ $(( month + 1 )) -lt 13 ] ; then
320                month0=$(( month + 1 ))
321            else
322                month0=$(( month + 1 - 12 ))
323                (( year = year0 + 1 ))
324            fi
325            (( Length=0 ))
326            (( i=0 ))
327            until [ $i -ge $PeriodLengthInMonths ] ; do
328                if [ $(( month0 + i )) -lt 13 ] ; then
329                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
330                else
331                    (( year = year0 + 1 ))
332                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
333                fi
334                (( i=i+1 ))
335            done
336            ;;
337        *D|*d)
338            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
339        *)
340            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
341            IGCM_debug_Verif_Exit_Post ;;
342    esac
343    echo ${Length}
344
345    IGCM_debug_PopStack "create_ts_next_date"
346}
347
348# Give the end of the actual $1 period from $2 date
349function create_ts_begin_date {
350    IGCM_debug_PushStack "create_ts_begin_date" $@
351    typeset year month Length
352    case ${config_UserChoices_PeriodLength} in
353        *Y|*y) 
354            PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | awk -F '[yY]' '{print $1}' )
355            IGCM_date_GetYearMonth ${1} year month
356            if [ X${2} = Xend ] ; then
357                (( year = year - PeriodLengthInYears + 1))
358            fi
359            (( Length=0 ))
360            (( i=0 ))
361            until [ $i -ge $PeriodLengthInYears ] ; do
362                (( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) )) 
363                (( i=i+1 ))
364            done
365            ;;
366        *M|*m) 
367            PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | awk -F '[mM]' '{print $1}' )
368            IGCM_date_GetYearMonth ${1} year month
369            if [ X${2} = Xend ] ; then
370                (( month = $month - $PeriodLengthInMonths + 1 ))
371            fi     
372            (( year0=$year ))
373            if [ $month -le 0 ] ; then
374                (( month = $month + 12 ))
375                year=$( printf "%04i\n" $(( year - 1 )) )
376            fi
377            month=$( printf "%02i\n" ${month} )
378            (( Length=0 ))
379            (( i=0 ))
380            until [ $i -ge $PeriodLengthInMonths ] ; do
381                if [ $(( month + i )) -lt 13 ] ; then
382                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
383                else
384                    (( year = year0 + 1 ))
385                    (( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
386                fi
387                (( i=i+1 ))
388            done
389            ;;
390        *D|*d)
391            Length=$( echo ${config_UserChoices_PeriodLength} | sed -e "s/[dD]//" ) ;;
392        *)
393            IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, *M, *D."
394            IGCM_debug_Verif_Exit_Post ;;
395    esac
396    echo ${Length}
397
398    IGCM_debug_PopStack "create_ts_begin_date"
399}
400
401########################################################################
402#
403#               IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
404#                 IF SO BRING THEM IN THE WORKING DIRECTORY
405#
406# PS : Keep in mind that IGCM_sys_Get here is a weak link :
407#      - especially for IDRiS
408#      - no special protection against "command too long" !
409#      - libIGCM_sys should handle that or modify following code part
410#
411########################################################################
412
413if [ ${StandAlone} != true ] ; then
414    Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
415       ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
416       echo \${run_PostProcessing_TimeSeriesRunning}" )
417
418    if [ X${Running_Flag} = Xy ] ; then
419        # Time Series Job allready running
420        IGCM_debug_Print 1 "Time Series Job allready running exit"
421        exit
422    fi
423    # Now run.card know we are running
424    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
425    # Get information from last execution
426    CompletedFlag=$( IGCM_sys_RshMaster \
427        "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
428        echo \${run_PostProcessing_TimeSeriesCompleted}" )
429fi
430
431# Is it the first submission or not ?
432if [ X${CompletedFlag} = X ] ; then
433    # First Time Series Submission
434    FIRST_PASS=TRUE
435    Length=$( create_ts_begin_date ${DateBegin} begin)
436    DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
437else
438    # Last Completed Time Series
439    FIRST_PASS=FALSE
440    DATE_FIN_JOB_B=${CompletedFlag}
441    date=${DateBegin}_${DATE_FIN_JOB_B}
442    for comp in ${config_ListOfComponents[*]} ; do
443        #
444        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
445        #
446        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
447        #
448        set -A liste_file
449        #
450        file_path=${R_SAVE}/${comp}/Analyse
451        #
452        i=0
453        file=${NULL_STR}
454        #
455        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
456            #
457            # Determine in which output can we find file
458            # IE : Analyse/TS or Analyse/DA2TS or ...
459            #
460            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
461            case ${FlagDir} in
462                *Y)  TS_Dir=TS_YE  ;;
463                *M)  TS_Dir=TS_MO  ;;
464                *D)  TS_Dir=TS_DA  ;;
465                HF)  TS_Dir=TS_HF  ;;
466                INS) TS_Dir=TS_INS ;;
467            esac
468            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
469                #
470                set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
471                #
472            done
473        done
474        if [ X${file} != X${NULL_STR} ] ; then
475            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
476            IGCM_debug_Verif_Exit_Post
477        fi
478    done
479    #
480    LEVEL=0
481    #
482fi
483
484########################################################################
485#
486#                          DEFINE LOOP PARAMETERS
487#
488########################################################################
489
490DATE_COUNT=${DATE_FIN_JOB_B}
491DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
492
493( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 || NBRE_FILE_TOT=1
494
495if [ ${DoJob} = true ] ; then
496    while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
497        (( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
498        Length=$( create_ts_next_date ${DATE_COUNT} )
499        DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
500    done
501else
502    NBRE_FILE_TOT=0
503fi
504
505if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
506    NBRE_TOUR=1
507elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
508    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT} 
509else
510    let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
511fi
512
513CURRENT_LOOP=1
514# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
515# WE HAVE 3 LEVELS
516#    - LEVEL 1 : JUST ONE LOOP TO COMPLETE
517#    - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
518#    - LEVEL 3 : WE ARE IN THE LAST LOOP
519
520while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] ; do
521     
522    if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then 
523        # - LEVEL 1 : JUST ONE LOOP TO COMPLETE
524        NBRE_FILE_LOOP=${NBRE_FILE_TOT}
525
526        if [ ${FIRST_PASS} = TRUE ] ; then
527            DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
528        else
529            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
530            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
531        fi
532
533        DATE_FIN=${PeriodDateEnd}
534        DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
535
536        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
537        [ -n "${LEVEL}" ] && FLAG_B="TRUE" || FLAG_B="FALSE"
538
539        LEVEL=1
540        DEPOT="TRUE"     
541    elif [ ${CURRENT_LOOP} -ne  ${NBRE_TOUR} ] ; then
542        # - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
543        NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
544
545        if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
546            Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
547            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
548        elif [ -z "${DATE_COURANTE}" ] ; then
549            DATE_COURANTE=${DATE_FIN_JOB_B}
550        else
551            Length=$( create_ts_next_date ${DATE_FIN} )
552            DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
553        fi
554         
555        (( TotLength = 0 ))
556        COMPTEUR=1
557        DATE_LOOP=${DATE_COURANTE}
558
559        while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
560            #
561            Length=$( create_ts_next_date ${DATE_LOOP} )
562            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
563            (( TotLength = TotLength + Length ))
564            (( COMPTEUR = COMPTEUR + 1 ))
565            #
566        done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
567        DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
568
569        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
570        [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" || [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" || FLAG_B="FALSE"
571        DEPOT="FALSE"
572        LEVEL=2
573    else
574        # - LEVEL 3 : WE ARE IN THE LAST LOOP
575        NBRE_FILE_LOOP=$( expr ${NBRE_FILE_TOT} % $(( UNIX_MAX_LIMIT * $(( CURRENT_LOOP - 1 )) )) )
576        [ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
577
578        Length=$( create_ts_next_date ${DATE_FIN} )
579        DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
580        DATE_FIN=${PeriodDateEnd}
581        [ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
582
583        LEVEL=3
584        DEPOT="TRUE"
585        FLAG_B="TRUE"
586    fi
587
588    # THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
589    for comp in ${config_ListOfComponents[*]} ; do
590        #
591        IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
592        #
593        IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
594        #
595        eval R_OUT_${comp}=${R_SAVE}/${comp}
596        #
597        i=0
598        #
599        for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
600            #
601            # Determine in which output can we find file
602            # IE : Output/MO or Output/DA or ...
603            #
604            FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
605            case ${FlagDir} in
606                *Y)    TS_Dir=TS_YE  ; FreqDir=YE  ;;
607                *M)    TS_Dir=TS_MO  ; FreqDir=MO  ;;
608                *D)    TS_Dir=TS_DA  ; FreqDir=DA  ;;
609                HF)    TS_Dir=TS_HF  ; FreqDir=HF  ;;
610                INS)   TS_Dir=TS_INS ; FreqDir=INS ;;
611                *) 
612                       IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
613                       IGCM_sys_Exit "Job create_ts" ;;
614            esac
615
616            # Initialize array
617            set -A liste_file
618            set -A liste_file_tmp
619
620            eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
621
622            COMPTEUR=1
623            DATE_LOOP=${DATE_COURANTE}
624
625            while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
626                #
627                Length1=$( create_ts_begin_date ${DATE_LOOP} end)
628                DATE_TAB=$(  IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
629
630                Length2=$( create_ts_next_date ${DATE_LOOP} end)
631                DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
632
633                set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
634                set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
635                (( COMPTEUR = COMPTEUR + 1 ))
636                #
637            done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
638            DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
639
640            # Get selected files
641            IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
642            eval ExitFlag=false
643            IGCM_debug_Verif_Exit_Post
644
645            # Apply IGCM_Patch if needed
646            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
647                for file_t in $( eval echo ${liste_file_tmp[*]} ); do           
648                    for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
649                        IGCM_${Patch} ${file_t}
650                    done
651                done
652            fi
653
654            # WE CAN CONCATENATE FILES AT THIS POINT
655            [ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
656            Length=$( create_ts_begin_date ${DATE_COURANTE} end)
657            DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
658            DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
659            DATE_BUILD_END=${DateBegin}_${DATE_FIN}
660
661            liste_coord=" "
662            for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
663              liste_coord=${liste_coord}${axis}","
664            done
665
666            # Time axis must be the last dimension
667            time_axis=${axis}
668
669            # SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
670            for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
671                #
672                IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
673                #
674#                 # Change time axis name to be CF compliant
675#                 if [ ${time_axis} != time_counter ]; then
676#                   IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
677#                 fi
678                #
679                if [ "${FLAG_B}" = "TRUE" ] ; then
680                    # WE CONCATENATE WITH EXISTING FILES
681                    file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
682                    file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
683                    file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
684                    IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
685                    #
686                    IGCM_sys_Rm ${file1} ${file2}
687                fi
688                #
689                if [ "${DEPOT}" = "TRUE" ] ; then
690                    # WE PUT FINAL FILE ON FILE SYSTEM
691                    [ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
692                    #${ncatted} -a modulo,time_counter,c,c," " ${file_out}
693                    file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
694                    eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
695                    [ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
696                    IGCM_sys_Rm ${file_out}
697                    #
698                fi
699                #
700                eval ExitFlag=false
701                IGCM_debug_Verif_Exit_Post
702            done     # for var in ${liste_var} ; do
703            IGCM_sys_Rm ${liste_file_tmp[*]}
704        done         # for file in ${liste_file_brut} do
705    done             # for comp in ${config_ListOfComponents[*]} ; do
706    eval ExitFlag=false
707    IGCM_debug_Verif_Exit_Post
708    # GO AHEAD IN DEEPEST LOOP
709    (( CURRENT_LOOP = CURRENT_LOOP + 1 ))
710done                 # while [ ${CURRENT_LOOP} -le  ${NBRE_TOUR} ] do
711# FINISH
712
713# DODS copy
714for comp in ${config_ListOfComponents[*]} ; do
715    for TS_Dir in $( ls ${R_SAVE}/${comp}/Analyse/ 2>&1 | grep TS_ ) ; do
716        [ -d ${R_SAVE}/${comp}/Analyse/${TS_Dir} ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
717    done
718done
719
720# clean environment variables and export usefull one :
721if [ X$( hostname -s ) = Xrhodes ] ; then
722    LISTE_ENV=$( env | grep -iv MYLANG | grep -iv HOST | grep -iv LOGIN | grep -iv config_ | grep -iv R_ | grep -iv libIGCM | grep -iv PERIOD \
723        | grep -iv DMFDIR | grep -iv AN_| grep -iv YEARS | grep -iv ex | grep -iv RESOL_ | grep -iv PATH | grep -iv WORKDIR | grep -iv USER   \
724        | grep -iv LOGNAME | grep -iv PROFILE | grep -iv ENV | grep -iv TMPDIR | grep -iv POST_DIR | grep -iv StandAlone | grep -iv DB | awk -F= '{print $1}' )
725 
726    for variables in ${LISTE_ENV} ; do
727        unset ${variables}
728    done
729fi
730
731export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
732export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF
733
734listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF"
735export listVarEnv
736
737#D-
738# --------------------------------------------------------------------
739#D- Test if all was right
740# --------------------------------------------------------------------
741IGCM_debug_Verif_Exit_Post
742# --------------------------------------------------------------------
743#D- Submit MONITORING
744# --------------------------------------------------------------------
745IGCM_sys_QsubPost monitoring
746
747#D-
748# --------------------------------------------------------------------
749#D- Update the run.card
750# --------------------------------------------------------------------
751if [ ${StandAlone} != true ] ; then
752    # Put in run.card end period of time series
753    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
754    # We have finish the job !
755    IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
756fi
757date
Note: See TracBrowser for help on using the repository browser.