source: trunk/libIGCM/AA_create_se @ 1603

Last change on this file since 1603 was 1603, checked in by aclsce, 7 months ago
  • Changed libIGCM behaviour in case of pb in pack_output.job : now, we exit from the job if there is any problem, whatever we are in PROD, DEVT, TEST mode.
  • Specification of pack frequency per file :

In order to reduce the number of inodes, it is possible to specify by file the frequency of packing. The syntax to do that is in the 4th column of OutputFiles? section of the component.card, for example as follows in lmdz.card :
[OutputFiles?]
List= (histmth.nc, ${R_OUT_ATM_O_M}/${PREFIX}_1M_histmth.nc, Post_1M_histmth, 100Y), \

(histday.nc, ${R_OUT_ATM_O_D}/${PREFIX}_1D_histday.nc, Post_1D_histday, 10Y), \

...

In this example, histmth files will be packed every 100 years and histday files will be packed every 10 years.
The pack frequency you defined in config.card is the frequency of pack by default, that means if a specific frequency of pack is specified for a file in a component.card, this file will be packed at the specific frequency whereas all other files will be packed at global pack frequency (specified in config.card) and in this case, the frequency pack (from the config.card) is the frequency the pack_output job will be launched at.
There is a constraint to use this fonctionality : the Packfrequency you defined in config.card must be greater or equal to the pack frequencies you specified for each type of file in component.card, otherwise the computing job will be stopped (with an explicit error message).
Surpack mode :
A surpack mode functionality is available through the use of pack_output.job. To enable this functionality, you have to put "surpack_mode=y" (default value is n). The way to use is similar to restart post-processing pack_output jobs, as indicated here : ​http://forge.ipsl.jussieu.fr/igcmg_doc/wiki/Doc/CheckDebug#RestartPack_output. You can either use a global pack frequency in config.card or specific pack frequency per file, as explained above.

  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 31.0 KB
Line 
1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
4#-Q- curie #MSUB -r SE             # Nom du job
5#-Q- curie #MSUB -o SE.out_%I
6#-Q- curie #MSUB -e SE.out_%I
7#-Q- curie #MSUB -n 1              # Reservation du processus
8#-Q- curie #MSUB -T 36000          # Limite de temps elapsed du job
9#-Q- curie #MSUB -q ::default_node::
10#-Q- curie #MSUB -c ::default_core::
11#-Q- curie #MSUB -Q normal
12#-Q- curie #MSUB -A ::default_project::
13#-Q- curie set +x
14#-Q- irene ######################
15#-Q- irene ## IRENE   TGCC/CEA ##
16#-Q- irene ######################
17#-Q- irene #MSUB -r SE            # Job name
18#-Q- irene #MSUB -o SE.out_%I
19#-Q- irene #MSUB -e SE.out_%I
20#-Q- irene #MSUB -n 1
21#-Q- irene #MSUB -T 36000         # Maximum elapsed time
22#-Q- irene #MSUB -q ::default_node::
23#-Q- irene #MSUB -c ::default_core::
24#-Q- irene #MSUB -Q normal
25#-Q- irene #MSUB -A ::default_post_project::
26#-Q- irene #MSUB -m store,work,scratch
27#-Q- irene set +x
28#-Q- irene-amd ######################
29#-Q- irene-amd ## IRENE   TGCC/CEA ##
30#-Q- irene-amd ######################
31#-Q- irene-amd #MSUB -r SE     # Job name
32#-Q- irene-amd #MSUB -o SE.out_%I
33#-Q- irene-amd #MSUB -e SE.out_%I
34#-Q- irene-amd #MSUB -n 1              # Number of cores
35#-Q- irene-amd #MSUB -T 36000          # Maximum elapsed time
36#-Q- irene-amd #MSUB -q ::default_node::
37#-Q- irene-amd #MSUB -c ::default_core::
38#-Q- irene-amd #MSUB -Q normal
39#-Q- irene-amd #MSUB -A ::default_post_project::
40#-Q- irene-amd #MSUB -m store,work,scratch
41#-Q- irene-amd set +x
42#-Q- jeanzay #!/bin/ksh
43#-Q- jeanzay ######################
44#-Q- jeanzay ## JEANZAY    IDRIS ##
45#-Q- jeanzay ######################
46#-Q- jeanzay #SBATCH --job-name=SE         # Job Name
47#-Q- jeanzay #SBATCH --output=SE.out_%I    # standard output
48#-Q- jeanzay #SBATCH --error=SE.out_%I     # error output
49#-Q- jeanzay #SBATCH -N  1                        # Number of core
50#-Q- jeanzay #SBATCH --partition=prepost          # Post-processing partition
51#-Q- jeanzay #SBATCH --time=10:00:00       # Wall clock limit (seconds)
52#-Q- jeanzay #SBATCH --account ::default_project::@cpu
53#-Q- jeanzay set +x
54#-Q- ada #!/bin/ksh
55#-Q- ada #######################
56#-Q- ada ## ADA         IDRIS ##
57#-Q- ada #######################
58#-Q- ada # @ job_type = mpich
59#-Q- ada # @ requirements = (Feature == "prepost")
60#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
61#-Q- ada # @ wall_clock_limit = 10:00:00
62#-Q- ada # Memory required for ncrcat
63#-Q- ada # @ as_limit = 30Gb
64#-Q- ada # Nom du travail LoadLeveler
65#-Q- ada # @ job_name   = SE
66#-Q- ada # Fichier de sortie standard du travail
67#-Q- ada # @ output     = $(job_name).$(jobid)
68#-Q- ada # Fichier de sortie d'erreur du travail
69#-Q- ada # @ error      =  $(job_name).$(jobid)
70#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
71#-Q- ada # @ notification = error
72#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $POST_DIR ; $Script_Post_Output ; $MASTER ; $DateBegin ; $PeriodDateEnd ; $StandAlone ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; wall_clock_limit=$(wall_clock_limit)
73#-Q- ada # @ queue
74#-Q- obelix ######################
75#-Q- obelix ## OBELIX      LSCE ##
76#-Q- obelix ######################
77#-Q- obelix #PBS -N SE
78#-Q- obelix #PBS -m a
79#-Q- obelix #PBS -j oe
80#-Q- obelix #PBS -q medium
81#-Q- obelix #PBS -o SE.$$
82#-Q- obelix #PBS -S /bin/ksh
83#-Q- mesoipsl #!/bin/ksh
84#-Q- mesoipsl ######################
85#-Q- mesoipsl ## MESO ESPRI IPSL  ##
86#-Q- mesoipsl ######################
87#-Q- mesoipsl #SBATCH --job-name=TS         # Job Name
88#-Q- mesoipsl #SBATCH --output=TS.out_%I    # standard output
89#-Q- mesoipsl #SBATCH --error=TS.out_%I     # error output
90#-Q- mesoipsl #SBATCH -N  1                        # Number of core
91#-Q- mesoipsl #SBATCH --time=10:00:00       # Wall clock limit (seconds)
92#-Q- ifort_CICLAD ######################
93#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
94#-Q- ifort_CICLAD ######################
95#-Q- ifort_CICLAD #PBS -N SE
96#-Q- ifort_CICLAD #PBS -m a
97#-Q- ifort_CICLAD #PBS -j oe
98#-Q- ifort_CICLAD ###PBS -q h12       # Queue for 12 hours at ciclad only
99#-Q- ifort_CICLAD #PBS -S /bin/ksh
100#-Q- default #!/bin/ksh
101#-Q- default ##################
102#-Q- default ## DEFAULT HOST ##
103#-Q- default ##################
104
105#**************************************************************
106# Author: Sebastien Denvil
107# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
108# $Revision::                                          $ Revision of last commit
109# $Author::                                            $ Author of last commit
110# $Date::                                              $ Date of last commit
111# IPSL (2006)
112#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
113#
114#**************************************************************
115
116#set -eu
117#set -vx
118
119date
120
121#-Q- ada export OMP_NUM_THREADS=1
122
123#D- Task type DO NOT CHANGE (computing, post-processing or checking)
124TaskType=post-processing
125
126########################################################################
127
128#D- Flag to determine if this job in a standalone mode
129#D- Default : value from AA_job if any
130StandAlone=${StandAlone:=true}
131
132#D- Path to libIGCM
133#D- Default : value from AA_job if any
134libIGCM=${libIGCM:=::modipsl::/libIGCM}
135
136#D- Flag to determine if rebuild process has been performed asynchronously
137#D- Possible value true or false.
138#D- If true raw files has allready been patched by rebuild job
139#D- If false create_ts will patch the raw files
140#D- Default : Value from AA_job if any. Usefull in StandAlone case.
141RebuildFrequency=${RebuildFrequency:=true}
142
143#D- Flag to determine atlas job's output directory
144#D- Default : value from libIGCM_post.ksh if any
145POST_DIR=${POST_DIR:=${PBS_O_WORKDIR:=$(pwd)}}
146
147#D- Increased verbosity (1, 2, 3)
148#D- Default : value from AA_job if any
149Verbosity=${Verbosity:=3}
150
151#D- Low level debug : to bypass lib test checks and stack construction
152#D- Default : value from AA_job if any
153DEBUG_debug=${DEBUG_debug:=false}
154
155#D- TEMPORARY Flag to determine atmospheric resolution
156#D- Default : value from atmospheric driver if any
157RESOL_ATM=${RESOL_ATM:=ALL}
158
159#D- Flag to determine surface resolution
160#D- Default : value from surface driver if any
161RESOL_SRF=ALL
162
163#D- Flag to determine surface resolution
164#D- Default : value from surface driver if any
165RESOL_SBG=ALL
166
167#D- TEMPORARY Flag to determine ocean resolution
168#D- Default : value from ocean driver if any
169RESOL_OCE=${RESOL_OCE:=ORCA2}
170
171#D- TEMPORARY Flag to determine ice resolution
172#D- Default : value from ice driver if any
173RESOL_ICE=${RESOL_OCE:=ORCA2}
174
175#D- TEMPORARY Flag to determine marine biogeochemistry resolution
176#D- Default : value from ice driver if any
177RESOL_MBG=${RESOL_OCE:=ORCA2}
178
179########################################################################
180
181. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
182. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
183. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
184#-------
185. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
186. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
187#-------
188IGCM_sys_MkdirWork ${RUN_DIR_PATH}
189IGCM_sys_Cd ${RUN_DIR_PATH}
190#-------
191( ${DEBUG_debug} ) && IGCM_debug_Check
192( ${DEBUG_debug} ) && IGCM_card_Check
193( ${DEBUG_debug} ) && IGCM_date_Check
194
195########################################################################
196
197#set -vx
198
199# ------------------------------------------------------------------
200# Test if all was right before proceeding further
201# ------------------------------------------------------------------
202IGCM_debug_Verif_Exit
203
204if [ ${StandAlone} = true ] ; then
205    CARD_DIR=${SUBMIT_DIR}
206else
207    CARD_DIR=${RUN_DIR_PATH}
208    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
209    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
210    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
211    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
212fi
213
214#==================================
215# First of all
216#
217# Read libIGCM compatibility version in config.card
218# Read UserChoices section
219# Read Ensemble section
220# Read Post section
221# Define all netcdf output directories
222#==================================
223IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
224
225# ------------------------------------------------------------------
226# Activate BigBrother so as to supervise this job
227# ------------------------------------------------------------------
228IGCM_debug_BigBro_Initialize
229
230#==================================
231# Define default value to keep compatibility with previous card: means before changes due to TGCC
232if [ X${PackDefault} = Xtrue ] ; then
233  [ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
234else
235  config_Post_PackFrequency=NONE
236fi
237
238#==================================
239# If pack is active then PackFrequency overule the config_UserChoices_PeriodLength
240if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
241  config_UserChoices_PeriodLength=${config_Post_PackFrequency}
242fi
243
244#==================================
245# Read ListOfComponents section:
246IGCM_card_DefineArrayFromSection   ${CARD_DIR}/config.card ListOfComponents
247
248#==================================
249# Read SeasonalFrequency:
250IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card Post SeasonalFrequency
251
252#
253# Determine period for seasonnal average. Default : value from AA_job if any
254#
255DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
256#
257PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
258#
259freq=$( echo ${config_Post_SeasonalFrequency} |  sed -e "s/[yY]//" )
260#
261an_fin=$(( $( IGCM_date_ConvertGregorianDateToJulian ${PeriodDateEnd} ) / 1000 ))
262an_deb=$( IGCM_date_YearDigit $(( an_fin - freq + 1 )) )
263an_fin=$( IGCM_date_YearDigit ${an_fin} )
264#
265# For each component selected determine which files need post-processing
266#
267for comp in ${config_ListOfComponents[*]} ; do
268  #
269  IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
270  eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
271  #
272  card=${CARD_DIR}/COMP/${compname}.card
273  IGCM_card_DefineArrayFromOption ${card} OutputFiles List
274  #
275  ListFilesName=${compname}_OutputFiles_List
276  eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
277  #
278  if [ X${FileName0} != X${NULL_STR} ] ; then
279    #
280    # INITIALISATION
281    #
282    eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
283    typeset i=2
284    #
285    until [ $i -gt $NbFiles ]; do
286      #
287      eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
288      #
289      if [ X${flag_post} != XNONE ] ; then
290        #
291        # Check if seasonal is required for this file
292        #
293        IGCM_card_DefineArrayFromSection ${card} ${flag_post}
294        #
295        # Seasonal case : If option Seasonal is not found (old cards) put SEASONAL ON by default
296        #
297        # variable option allready typeset above
298        for option in $( eval echo \${${compname}_${flag_post}[*]} ) ; do
299          if [ ${option} = Seasonal ] ; then
300            FoundSeasonal=true
301            IGCM_card_DefineVariableFromOption ${card} ${flag_post} Seasonal
302          fi
303        done
304        #
305        if [ ! X${FoundSeasonal} = Xtrue ] ; then
306            eval ${compname}_${flag_post}_Seasonal=ON
307        fi
308        #
309        if [ ! X$( eval echo \${${compname}_${flag_post}_Seasonal} ) = XON ] ; then
310            (( i_ = i+1 ))
311            eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
312            eval Testvar=${Testvar_}
313            if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
314                (( i = i+4 ))
315            else
316          (( i=i+3 ))
317            fi
318          continue
319        fi
320        #
321        # CREATE LIST
322        #
323        FILE=$( echo ${flag_post} | awk "-FPost_" '{print $2}' )
324        #
325        DIRECTORY=${R_SAVE}/${comp}/Analyse/SE
326        #
327        TestedFile=${config_UserChoices_JobName}_SE_${an_deb}_${an_fin}_${FILE}.nc
328        #
329        IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
330        if [ ! $? = 0 ] ; then
331          eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
332          (( i_ = i+1 ))
333          eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
334          eval Testvar=${Testvar_}
335          if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
336              eval PeriodPack_FileType=${Testvar}
337          else
338              PeriodPack_FileType=${config_Post_PackFrequency}
339          fi
340          eval set +A LISTE_PACKFREQUENCY_${comp} \${LISTE_PACKFREQUENCY_${comp}[*]} ${PeriodPack_FileType}
341          if [ ! ${RebuildFrequency} = true ] ; then
342            IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
343            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
344            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
345              for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
346                if [ Xload${Patch} != X ] ; then
347                  . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
348                  eval load${Patch}=loaded
349                fi
350              done
351            fi
352          fi
353        fi
354      fi
355      (( i_ = i+1 ))
356      eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
357      eval Testvar=${Testvar_}
358      if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
359          (( i = i+4 ))
360      else
361      (( i=i+3 ))
362      fi
363    done # until [ $i -eq $NbFiles ]
364  fi
365done         # for comp
366
367for comp in ${config_ListOfComponents[*]} ; do
368  #
369  eval R_OUT_${comp}=${R_SAVE}/${comp}
370  #
371  i=0
372  for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
373  config_UserChoices_PeriodLength=$( eval echo \${LISTE_PACKFREQUENCY_${comp}[$i]} ) 
374    #
375    # Determine in which output can we find file
376    # IE : Output/MO or Output/DA or ...
377    #
378    FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
379    case ${FlagDir} in
380      1Y)    FreqDir=YE  ;;
381      1M)    FreqDir=MO  ;;
382      5D|1D) FreqDir=DA  ;;
383      HF)    FreqDir=HF  ;;
384      3H)    FreqDir=HF  ;;
385      INS)   FreqDir=INS ;;
386    esac
387    #
388    RESULT_SE=${config_UserChoices_JobName}_SE_${an_deb}_${an_fin}_${file}.nc
389    # Do not redoo an existing SE
390    IGCM_sys_TestFileArchive ${R_SAVE}/${comp}/Analyse/SE/${RESULT_SE}
391    if [ $? = 0 ] ; then
392      continue
393    fi
394    #
395    eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
396    #
397    case ${config_UserChoices_PeriodLength} in
398      # For Yearly simulation :
399      *Y|*y)
400        PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[yY]//' )
401        (( an = an_deb ))
402        # Initialize array
403        unset liste_file_se
404        unset liste_file_se_tmp
405        while [ ${an} -le ${an_fin} ] ; do
406          #
407          an4=$(printf "%04i\n" ${an} )
408          #
409          diy=$( IGCM_date_DaysInCurrentPeriod ${an4}0101 ${config_UserChoices_PeriodLength} )
410          afin=$( IGCM_date_AddDaysToGregorianDate ${an4}0101 $(( diy - 1 )) )
411          #
412          date=${an4}0101_${afin}
413          #
414          liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
415          liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
416          (( an = an + ${PeriodLengthInYears} ))
417        done
418        #
419        IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
420        IGCM_debug_Verif_Exit
421
422        # Apply IGCM_Patch if needed
423        if [ ! ${RebuildFrequency} = true ] ; then
424          if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
425            for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
426              for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
427                IGCM_${Patch} ${file_t}
428              done
429            done
430          fi
431        fi
432
433        # Search of common list of variables to be treated
434        nbfile=0
435        for file_common_var in ${liste_file_se_tmp[*]} ; do
436          ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
437          (( nbfile = nbfile + 1 ))
438        done
439        list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
440        IGCM_sys_Rm tmpfile_ncra
441
442        # detect time counter : time or time_counter ie unlimited variable
443        var_unlim=$(ncdump -h ${liste_file_se_tmp[0]}|grep UNLIMITED|awk '{print $1}')
444
445        for mois in  01 02 03 04 05 06 07 08 09 10 11 12 ; do
446          #
447          (( nb = ${mois} - 1 ))
448          #
449          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
450          IGCM_sys_ncra -O -v ${list_var_final_ncra} -d ${var_unlim},${nb},,12 ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
451        done
452        IGCM_sys_Rm ${liste_file_se_tmp[*]}
453        ;;
454      # For Monthly simulation :
455      1M|1m)
456        for mois in 01 02 03 04 05 06 07 08 09 10 11 12 ; do
457          #
458          (( nb = ${mois} - 1 ))
459          #
460          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
461          #
462          (( an = an_deb ))
463          # Initialize array
464          unset liste_file_se
465          unset liste_file_se_tmp
466          while [ ${an} -le ${an_fin} ] ; do
467            moisL=$( IGCM_date_DaysInMonth ${an} ${mois} )
468            #
469            an4=$(printf "%04i\n" ${an} )
470            #
471            date=${an4}${mois}01_${an4}${mois}${moisL}
472            #
473            liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
474            liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
475
476            (( an = an + 1 ))
477          done
478          #
479          IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
480          IGCM_debug_Verif_Exit
481
482          # Apply IGCM_Patch if needed
483          if [ ! ${RebuildFrequency} = true ] ; then
484            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
485              for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
486                for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
487                  IGCM_${Patch} ${file_t}
488                done
489              done
490            fi
491          fi
492
493          # Search of common list of variables to be treated
494          nbfile=0
495          for file_common_var in ${liste_file_se_tmp[*]} ; do
496            ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
497            (( nbfile = nbfile + 1 ))
498          done
499            list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
500          IGCM_sys_Rm tmpfile_ncra
501
502          IGCM_sys_ncra -O -v ${list_var_final_ncra} ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
503
504          IGCM_sys_Rm ${liste_file_se_tmp[*]}
505        done
506        ;;
507      *D|*d)
508        for mois in 01 02 03 04 05 06 07 08 09 10 11 12 ; do
509          #
510          (( nb = ${mois} - 1 ))
511          #
512          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
513          #
514          (( an = an_deb ))
515          # Initialize array
516          unset liste_file_se
517          unset liste_file_se_tmp
518          while [ ${an} -le ${an_fin} ] ; do
519            moisL=$( IGCM_date_DaysInMonth ${an} ${mois} )
520            #
521            an4=$(printf "%04i\n" ${an} )
522              #
523            length_d=$(( ${config_UserChoices_PeriodLength} - 1 ))
524            # for one day
525            if [ ${length_d} -eq 1 ] ; then
526              (( length_d = 0 ))
527            fi
528            (( day  = 1 ))
529            while [ ${day} -le ${moisL} ] ; do
530              day2=$(printf "%02i\n" ${day} )
531              day_beg=${an4}${mois}${day2}
532              day_end=$( IGCM_date_AddDaysToGregorianDate ${day_beg} ${length_d} )
533              date=${an4}${mois}${day_beg}_${an4}${mois}${day_end}
534              #
535              liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
536              liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
537              (( day = day + 1 ))
538            done
539            (( an = an + 1 ))
540          done
541          #
542          IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
543          IGCM_debug_Verif_Exit
544
545          # Apply IGCM_Patch if needed
546          if [ ! ${RebuildFrequency} = true ] ; then
547            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
548              for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
549                for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
550                  IGCM_${Patch} ${file_t}
551                done
552              done
553            fi
554          fi
555
556          # Search of common list of variables to be treated
557          nbfile=0
558          for file_common_var in ${liste_file_se_tmp[*]} ; do
559            ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
560            (( nbfile = nbfile + 1 ))
561          done
562            list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
563          IGCM_sys_Rm tmpfile_ncra
564
565          IGCM_sys_ncra -O -v ${list_var_final_ncra} ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
566          IGCM_sys_Rm ${liste_file_se_tmp[*]}
567        done
568        ;;
569      esac
570      #
571
572      # Search of common list of variables to be treated
573      nbfile=0
574      for file_common_var in ${OUT_SE[*]} ; do
575        ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncrcat
576        (( nbfile = nbfile + 1 ))
577      done
578      list_var_final_ncrcat=`cat tmpfile_ncrcat | sort | uniq -c | grep -v ^time_ | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
579      IGCM_sys_Rm tmpfile_ncrcat
580
581    IGCM_sys_ncrcat -O -v ${list_var_final_ncrcat} ${OUT_SE[*]} ${RESULT_SE}
582    #
583    IGCM_sys_Rm ${OUT_SE[*]}
584    #
585    # detect time counter : time or time_counter ie unlimited variable
586    var_unlim=$( ncdump -h ${RESULT_SE} | grep UNLIMITED |  cut -d ' ' -f 1 | sed -e 's/^[[:space:]]*//' )
587
588    # Put correct climatological axis following calendar
589    case ${config_UserChoices_CalendarType} in
590      360d|360_day)
591      # 360 days
592        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15, 45, 75, 105, 135, 165, 195, 225, 255, 285, 315, 345.} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 30, 30, 60, 60, 90, 90, 120, 120, 150, 150, 180, 180, 210, 210, 240, 240, 270, 270, 300, 300, 330, 330, 360.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"360_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
593        ;;
594      noleap|365_day)
595      # 365 days
596        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45, 74.5, 105, 135.5, 166, 196.5, 227.5, 258, 288.5, 319, 349.5} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 59, 59, 90, 90, 120, 120, 151, 151, 181, 181, 212, 212, 243, 243, 273, 273, 304, 304, 334, 334, 365.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"365_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
597        ;;
598      all_leap|366_day)
599      # 366 days
600        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45.5, 75.5, 106, 136.5, 167, 197.5, 228.5, 259, 289.5, 320, 350.5} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 60, 60, 91, 91, 121, 121, 152, 152, 182, 182, 213, 213, 244, 244, 274, 274, 305, 305, 335, 335, 366.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"366_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
601        ;;
602      leap|gregorian|standard)
603      # 365.2425 days
604        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45.12125, 74.7425, 105.2425, 135.7425, 166.2425, 196.7425, 227.7425, 258.2425, 288.7425, 319.2425, 349.7425} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 59.2425, 59.2425, 90.2425, 90.2425, 120.2425, 120.2425, 151.2425, 151.2425, 181.2425, 181.2425, 212.2425, 212.2425, 243.2425, 243.2425, 273.2425, 273.2425, 304.2425, 304.2425, 334.2425, 334.2425, 365.2425} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"gregorian\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
605        ;;
606    esac
607    eval IGCM_sys_ncap2 -Oh -s $ncap2cmd ${RESULT_SE} tmp.nc
608    IGCM_sys_Rm ${RESULT_SE}
609    IGCM_sys_Mv tmp.nc ${RESULT_SE}
610    #
611    eval IGCM_sys_Put_Out ${RESULT_SE} \${R_OUT_${comp}}/Analyse/SE/${RESULT_SE}
612    [ -f ${RESULT_SE} ] && IGCM_sys_Rm ${RESULT_SE}
613    #
614    (( i = i + 1 ))
615  done
616done
617
618# DODS copy
619if [ X${config_Post_DodsCopy} = XTRUE ]  ; then
620  for comp in ${config_ListOfComponents[*]} ; do
621    IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/SE
622    [ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/SE
623  done
624fi
625
626# Temporal coverage id for atlas
627YEARS=${an_deb}_${an_fin}
628DIR_OUT=SE
629
630export DEBUG_debug; export BigBrother ; export libIGCM; export SUBMIT_DIR; export POST_DIR; export Script_Output ;
631export StandAlone ; export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ;
632export RESOL_SBG ; export YEARS; export DIR_OUT
633listVarEnv="DEBUG_debug,BigBrother,libIGCM,SUBMIT_DIR,POST_DIR,Script_Post_Output,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG,YEARS,DIR_OUT"
634export listVarEnv;
635
636#D-
637# --------------------------------------------------------------------
638#D- Test if all was right before ATLAS
639# --------------------------------------------------------------------
640IGCM_debug_Verif_Exit
641
642if [ ${config_UserChoices_TagName} = "IPSLCM4_v1_OASIS3" ] || \
643   [ ${config_UserChoices_TagName} = "IPSLCM4_v2" ]        || \
644   [ ${config_UserChoices_TagName} = "IPSL_ESM_V1" ] ; then
645  # ATLAS
646  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
647    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
648    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
649    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
650    IGCM_sys_QsubPost atlas_LMDZ                              ;
651    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
652    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
653    export Script_Post_Output=additionnal.${PeriodDateEnd}    ;
654    IGCM_sys_QsubPost additionnal                             ;
655  fi
656  #
657elif [ ${config_UserChoices_TagName} = "IPSLCM5" ]         || \
658     [ ${config_UserChoices_TagName} = "IPSLCM5A" ]        || \
659     [ ${config_UserChoices_TagName} = "IPSLCM5B" ]        || \
660     [ ${config_UserChoices_TagName} = "IPSLCM5A-MR" ]     || \
661     [ ${config_UserChoices_TagName} = "IPSLCM5A2" ]       || \
662     [ ${config_UserChoices_TagName} = "IPSLCM6" ] ; then
663  # METRICS
664  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
665    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
666    IGCM_sys_QsubPost metrics_LMDZ                            ;
667    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
668    IGCM_sys_QsubPost metrics_NEMO                            ;
669  fi
670  # ATLAS
671  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
672    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
673    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
674    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
675    IGCM_sys_QsubPost atlas_LMDZ                              ;
676    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
677    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
678    export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
679    IGCM_sys_QsubPost atlas_PISCES                            ;
680  fi
681elif [ ${config_UserChoices_TagName} = "LMDZ4OR" ]         || \
682     [ ${config_UserChoices_TagName} = "LMDZOR" ]          || \
683     [ ${config_UserChoices_TagName} = "LMDZ4OR_v2" ] ; then
684  # METRICS
685  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
686    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
687    IGCM_sys_QsubPost metrics_LMDZ                            ;
688  fi
689  # ATLAS
690  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
691    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
692    IGCM_sys_QsubPost atlas_LMDZ                              ;
693    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
694    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
695  fi
696elif [ ${config_UserChoices_TagName} = "LMDZ" ] ; then
697  # METRICS
698  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
699    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
700    IGCM_sys_QsubPost metrics_LMDZ                            ;
701  fi
702  # ATLAS
703  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
704    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
705    IGCM_sys_QsubPost atlas_LMDZ                              ;
706  fi
707elif [ ${config_UserChoices_TagName} = "OL" ]              || \
708     [ ${config_UserChoices_TagName} = "OL2" ] ; then
709  # METRICS
710  # ATLAS
711  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
712    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
713    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
714  fi
715elif [ ${config_UserChoices_TagName} = "ORCA2_LIM" ]       || \
716     [ ${config_UserChoices_TagName} = "ORCA2_LIM2" ]      || \
717     [ ${config_UserChoices_TagName} = "ORCA2_LIM3" ]      || \
718     [ ${config_UserChoices_TagName} = "ORCA1_LIM2" ]      || \
719     [ ${config_UserChoices_TagName} = "ORCA1_LIM3" ] ; then
720  # METRICS
721  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
722    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
723    IGCM_sys_QsubPost metrics_NEMO                            ;
724  fi
725  # ATLAS
726  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
727    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
728    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
729  fi
730elif [ ${config_UserChoices_TagName} = "ORCA2_LIM2_PISCES" ] || \
731     [ ${config_UserChoices_TagName} = "ORCA2_LIM3_PISCES" ] || \
732     [ ${config_UserChoices_TagName} = "ORCA1_LIM2_PISCES" ] || \
733     [ ${config_UserChoices_TagName} = "ORCA1_LIM3_PISCES" ] ; then
734  # METRICS
735  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
736    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
737    IGCM_sys_QsubPost metrics_NEMO                            ;
738  fi
739  # ATLAS
740  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
741    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
742    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
743    export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
744    IGCM_sys_QsubPost atlas_PISCES                            ;
745  fi
746elif [ ${config_UserChoices_TagName} = "ORCA2_OFF_PISCES" ] ; then
747  # METRICS
748  # ATLAS
749  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
750  export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
751  IGCM_sys_QsubPost atlas_PISCES                            ;
752  fi
753fi
754
755# Execute C-ESM-EP driver (which launches jobs)
756if [ $PostCesmep = SE ] ; then
757    ${SUBMIT_DIR}/cesmep_lite/libIGCM_post.sh $DateBegin $PeriodDateEnd
758fi
759
760# Clean RUN_DIR_PATH (necessary for cesium and titane only)
761IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
762
763# ------------------------------------------------------------------
764# Finalize BigBrother to inform that the jobs end
765# ------------------------------------------------------------------
766IGCM_debug_BigBro_Finalize
Note: See TracBrowser for help on using the repository browser.