source: trunk/libIGCM/libIGCM_config/libIGCM_config.ksh @ 1057

Last change on this file since 1057 was 1057, checked in by sdipsl, 10 years ago
  • refactor archive and storage path definition. They can be defined using UserChoices? from config.card. Very easy now to decide where to store everything. There is a single place to look at : IGCM_sys_defineArchives. see #221
  • clean up IGCM_debug_PushStack arguments
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 51.6 KB
RevLine 
[2]1#!/bin/ksh
2
3#**************************************************************
4# Author: Sebastien Denvil, Martial Mancip
[373]5# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr Martial.Mancip__at__ipsl.jussieu.fr
6# $Revision::                                          $ Revision of last commit
7# $Author::                                            $ Author of last commit
8# $Date::                                              $ Date of last commit
[2]9# IPSL (2006)
10#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11#
12#**************************************************************
13
14#===================================
[726]15function IGCM_config_CommonConfiguration
16{
17  IGCM_debug_PushStack "IGCM_config_CommonConfiguration" $@
18
19  # Debug Print :
[769]20  [ ${Verbosity} -gt 0 ] && echo
[726]21  IGCM_debug_Print 1 "IGCM_config_CommonConfiguration" $@
22
23  # config.card path
24  configPath=$1
25
26  #==================================
27  typeset option auxprint CompatibilityTag
28
29  #==================================
30  # Read libIGCM compatibility version in config.card
31  IGCM_card_DefineVariableFromOption ${configPath} Compatibility libIGCM
[939]32  CompatibilityTag=${config_Compatibility_libIGCM}
[726]33
34  if [ ! "${CompatibilityTag}" = "${libIGCM_CurrentTag}" ] ; then
35    IGCM_debug_Exit "config.card is not compatible with libIGCM version ${libIGCM_CurrentTag} see libIGCM FAQ http://wiki.ipsl.jussieu.fr/wiki_ipsl/IGCMG/libIGCM/DocUtilisateur/FAQ ."
36  fi
37
38  #==================================
39  # Read UserChoices section:
[769]40  [ ${Verbosity} -gt 0 ] && echo
[726]41  IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
42
43  IGCM_card_DefineArrayFromSection ${configPath} UserChoices
44  for option in ${config_UserChoices[*]} ; do
45    IGCM_card_DefineVariableFromOption ${configPath} UserChoices ${option}
46    eval auxprint=\${config_UserChoices_${option}}
47    IGCM_debug_Print 3 "${option} : ${auxprint}"
48  done
49
50  #==================================
51  # Read Ensemble section:
[769]52  [ ${Verbosity} -gt 0 ] && echo
[726]53  IGCM_debug_Print 1 "DefineArrayFromOption  : config_Ensemble"
54
55  IGCM_card_DefineArrayFromSection ${configPath} Ensemble
56  for option in ${config_Ensemble[*]} ; do
57    IGCM_card_DefineVariableFromOption ${configPath} Ensemble ${option}
58    eval auxprint=\${config_Ensemble_${option}}
59    IGCM_debug_Print 3 "${option} : ${auxprint}"
60  done
61
62  #==================================
63  # Read Post section:
[769]64  [ ${Verbosity} -gt 0 ] && echo
[726]65  IGCM_debug_Print 1 "DefineArrayFromOption : config_Post"
66
67  IGCM_card_DefineArrayFromSection ${configPath} Post
68  for option in ${config_Post[*]} ; do
69    IGCM_card_DefineVariableFromOption ${configPath} Post ${option}
70    eval auxprint=\${config_Post_${option}}
71    IGCM_debug_Print 3 "${option} : ${auxprint}"
72  done
[769]73  [ ${Verbosity} -gt 0 ] && echo
[726]74
[728]75  #==================================
76  # Define default value to keep compatibility with previous card: means before changes due to TGCC
77  # Apply some overrules to ensure proper usage of computing centres resources
[804]78  #
[728]79  if [ X${PackDefault} = Xtrue ] ; then
80    if [ X${config_UserChoices_SpaceName} = XTEST ]; then
81      # TEST simulations will not be packed and will stay on SCRATCHDIR filesystem
82      IGCM_debug_Print 1 "SpaceName=TEST. OVERRULE PackFrequency to NONE"
83      config_Post_PackFrequency=NONE
84    else
85      # Default to RebuildFrequency if nothing has been set up related to PackFrequency
86      [ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
87    fi
88  else
89    # If we DO NOT apply pack in this computing center
90    config_Post_PackFrequency=NONE
91  fi
92
[726]93  #====================================================
[1057]94  # Define ARCHIVE : Dedicated to large files
95  # Define STORAGE : Dedicated to small/medium files
96  # Define R_OUT   : Output tree located on ARCHIVE
97  # Define R_FIG   : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
98  # Define R_BUF   : USELESS and DEPRECATED output tree.
99  IGCM_sys_defineArchives
100
101  #====================================================
[726]102  # R_SAVE : Job output directory
103  # R_BUFR : Job output buffered directory
104
105  if ( [ ! X${config_UserChoices_SpaceName} = X ] && [ ! X${config_UserChoices_ExperimentName} = X ] ) ; then
106    FreeName=$( echo ${config_UserChoices_JobName} | sed 's/.*_//' )
107    if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
108      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
109      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
110      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
111      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
112    else
113      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
114      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
115      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
116      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
117    fi
118  else
119    if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
120      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
121      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
122      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
123      R_DODS=${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
124    else
125      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
126      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
127      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
128      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_JobName}
129    fi
130  fi
131
[782]132  #====================================================
[1031]133  # Define REBUILD_DIR : where we store files needing rebuild process 
134  REBUILD_DIR=${R_BUFR}/REBUILD
135  if [ ! X${TaskType} = Xchecking ] ; then
136    IGCM_sys_MkdirWork ${REBUILD_DIR}
[782]137  fi
138
[903]139  #====================================================
140  # DodsCopy : apply default value if not defined
141  if ( [ X${config_Post_DodsCopy} = X${NULL_STR} ] || [ X${config_Post_DodsCopy} = X ] ) ; then
142    config_Post_DodsCopy=TRUE
143  fi
144
[726]145  IGCM_debug_PopStack "IGCM_config_CommonConfiguration"
146}
147
148#===================================
[2]149function IGCM_config_Initialize
150{
[544]151  IGCM_debug_PushStack "IGCM_config_Initialize"
[2]152
[544]153  # Debug Print :
154  echo
155  IGCM_debug_Print 1 "IGCM_config_Initialize"
[2]156
[544]157  # Test modipsl tree existence.
158  IGCM_sys_TestDir ${MODIPSL}
159  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
160  IGCM_sys_TestDir ${libIGCM}
161  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
162  IGCM_sys_TestDir ${R_EXE}
163  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
164  IGCM_sys_TestDir ${SUBMIT_DIR}
165  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
[2]166
[544]167  if ( $DEBUG_debug ) ; then
168    echo "Keep trace of inital SUBMIT_DIR : "
169    ls -lta ${SUBMIT_DIR}
170  fi
[144]171
[544]172  #==================================
[619]173  # Read ListOfComponents section:
[726]174  echo
175  IGCM_debug_Print 1 "DefineArrayFromSection : ListOfComponents"
176
[544]177  IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/config.card ListOfComponents
178  for comp in ${config_ListOfComponents[*]} ; do
179    IGCM_card_DefineArrayFromOption ${SUBMIT_DIR}/config.card ListOfComponents ${comp}
180  done
181  IGCM_debug_Print 3 ${config_ListOfComponents[*]}
[2]182
[544]183  #==================================
[726]184  # Read Executable section:
[544]185  IGCM_card_DefineArrayFromSection   ${SUBMIT_DIR}/config.card Executable
[619]186
[544]187  #==================================
[726]188  # Read Restarts section:
189  # Restarts : Gerneral rule or local for each component.
190  echo
191  IGCM_debug_Print 1 "DefineArrayFromOption : config_Restarts"
192
193  IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/config.card Restarts
194  for option in ${config_Restarts[*]} ; do
195    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/config.card Restarts ${option}
196    eval auxprint=\${config_Restarts_${option}}
197    IGCM_debug_Print 3 "${option} : ${auxprint}"
198  done
199
200  #==================================
201  # Define Job Outputs Name
202  echo
[544]203  IGCM_debug_Print 2 "Define Script_Output_Prefix and Exe_Output"
[939]204  Script_Output_Prefix=${config_UserChoices_Script_Output_Prefix:='Script_Output'}
[544]205  IGCM_debug_Print 3 "Script_Output_Prefix = ${Script_Output_Prefix}"
[939]206  Exe_Output=out_execution
[544]207  IGCM_debug_Print 3 "Exe_Output           = ${Exe_Output}"
[2]208
[544]209  #===================================================================#
210  # Prepare variables available for ${COMP}.card and ${COMP}.driver   #
211  #             But available to any son functions                    #
212  #===================================================================#
[2]213
[544]214  # Convert yyyy-mm-dd date to gregorian yyyymmdd
215  DateBegin=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )
[619]216  DateEnd=$(   IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd}   )
[2]217
[544]218  # Period Length In Days between DateBegin and DateEnd
219  (( ExperienceLengthInDays=$( IGCM_date_DaysBetweenGregorianDate ${DateEnd} ${DateBegin} )  + 1 ))
220  if [ ${ExperienceLengthInDays} -lt 0 ] ; then
221    IGCM_debug_Print 1 "Problem with dates in config.card : ${DateEnd} < ${DateBegin} ! You must check that."
222    IGCM_debug_Exit "IGCM_config_Initialize" " Wrong Dates."
[619]223    IGCM_debug_Verif_Exit
[544]224  fi
[2]225
[544]226  # Day and Year of Initial State (Given in julian format)
227  InitDay=$(( $( IGCM_date_ConvertGregorianDateToJulian $DateBegin ) % 1000 ))
228  InitYear=$(( $( IGCM_date_ConvertGregorianDateToJulian $DateBegin ) / 1000 ))
[2]229
[544]230  #================================================================#
231  #                  Test and Prepare directories                  #
232  #================================================================#
[2]233
[544]234  # ==> 4 kinds of input files :
235  #     1) R_INIT  : Initial State Files   (Etat0, carteveg)
236  #     2) R_BC    : Boundary Conditions   (Forcages, lai)
237  #     3) Parameters files (allready define through ${SUBMIT_DIR})
238  #     4) Restarts files   (allready define in IGCM_config_Initialize)
[2]239
[544]240  # Here we offer the possibility to redefine R_INIT, R_BC
241  # and PeriodNb through config.card
242  R_INIT=${config_UserChoices_R_INIT:=${R_IN}/INIT}
[731]243  echo
[544]244  IGCM_debug_Print 2 "(Re)Define R_INIT, R_BC and PeriodNb"
245  IGCM_debug_Print 3 "R_INIT=${R_INIT}"
246  R_BC=${config_UserChoices_R_BC:=${R_IN}/BC}
247  IGCM_debug_Print 3  "R_BC=${R_BC}"
248  PeriodNb=${config_UserChoices_PeriodNb:=${PeriodNb}}
249  IGCM_debug_Print 3  "Loop in main Job with ${PeriodNb} period(s)"
[2]250
[544]251  # Test Archive input/output.
[726]252  echo
[544]253  IGCM_sys_TestDirArchive ${ARCHIVE}
254  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
[2]255
[771]256  # SD ADA SPECIFIC #
257  #      TO FIX     #
258  #IGCM_sys_TestDirArchive ${R_IN}
259  #[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
260
[619]261  if ( ${FirstInitialize} ) ; then
[726]262    IGCM_sys_MkdirArchive   ${R_SAVE}
[805]263    [ ! ${config_Post_PackFrequency} = NONE ] && IGCM_sys_Mkdir ${R_BUFR}
[544]264  else
265    IGCM_sys_TestDirArchive ${R_SAVE}
266    [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive ${R_SAVE}"
[2]267
[805]268    if [ ! ${config_Post_PackFrequency} = NONE ] ; then
269      IGCM_sys_TestDir ${R_BUFR}
270      [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir ${R_BUFR}"
271    fi
[545]272
[544]273    #Test state of run in run.card
274    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodState
[619]275
[544]276    if ( [ ${run_Configuration_PeriodState} != "Start" ] && [ ${run_Configuration_PeriodState} != "Running" ] && [ ${run_Configuration_PeriodState} != "OnQueue" ] && [ ${run_Configuration_PeriodState} != "Continue" ] ) ; then
[619]277      echo
[544]278      IGCM_debug_Print 1 "!! Error in run.card with PeriodState : " ${run_Configuration_PeriodState} "!!"
[985]279      IGCM_debug_Print 1 "Check post-processing jobs carefully by running ${libIGCM}/RunChecker.job -p ${SUBMIT_DIR}"
280      IGCM_debug_Print 1 "Then try running ${libIGCM}/clean_month.job to rerun one period"
281      IGCM_debug_Print 1 "Then try running ${libIGCM}/clean_year.job to rerun more."
[544]282      IGCM_debug_Exit
283      IGCM_debug_Verif_Exit
[2]284    fi
[544]285  fi
[2]286
[544]287  #====================================================
[619]288  # Experience type : DEB(ug), DEV(elopment), RUN
[544]289  if [ X${JobType} != XRUN ] ; then
290    echo
291    echo "===================================================="
292    echo "libIGCM JOB is NOT in RUN type mode."
293    echo "!! OUTPUT files will NOT be PROTECTED !!"
294    echo "Be carefull : you can ERASE the result of this job !"
[128]295
[544]296    case ${JobType} in
297    DEB)
298      echo "DEBUG mode : activation of 'set -vx' mode."
299      echo "DEBUG mode : no protection for output files."
300      echo "DEBUG mode : if active force asynchronous rebuild frequency to PeriodLength frequency."
301      ;;
302    DEV)
303      echo "DEVelopment mode : no protection for output files."
304      echo "DEVelopment mode : if active force asynchronous rebuild frequency to PeriodLength frequency."
305      ;;
306    esac
[166]307
[544]308    if ( [ X${config_Post_RebuildFrequency} != XNONE ] && [ ${DRYRUN} -eq 0 ] ) ; then
309      if [ X${config_Post_RebuildFrequency} != X${config_UserChoices_PeriodLength} ] ; then
[619]310        echo "------------"
311        echo "WARNING : Job is NOT in RUN mode then we will force REBUILD Frequency"
[786]312        echo "          to PeriodLength : ${config_UserChoices_PeriodLength}"
[619]313        echo "------------"
314        config_Post_RebuildFrequency=${config_UserChoices_PeriodLength}
[544]315      fi
[128]316    fi
[544]317    echo "===================================================="
318    echo
319  fi
[128]320
[544]321  #====================================================
322  #R_OUT_KSH : Storage place for job output
323  #R_OUT_EXE : Storage place for binary used during simulation
324  R_OUT_KSH=${R_SAVE}/Out
325  R_OUT_EXE=${R_SAVE}/Exe
[166]326
[544]327  #====================================================
328  #R_BUF_KSH : Buffer place for job output
329  #R_BUF_EXE : Buffer place for binary used during simulation
330  R_BUF_KSH=${R_BUFR}/Out
331  R_BUF_EXE=${R_BUFR}/Exe
332
333  IGCM_debug_PopStack "IGCM_config_Initialize"
[2]334}
335
336#===================================
[890]337function IGCM_config_DaysInPeriodLength
[118]338{
[890]339  IGCM_debug_PushStack "IGCM_config_DaysInPeriodLength"
[118]340
[544]341  typeset i
[124]342
[544]343  # Determine number of day(s) in PeriodLength :
344  case ${config_UserChoices_PeriodLength} in
345  *Y|*y)
[619]346    PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[yY]//' )
[731]347    echo
[544]348    IGCM_debug_Print 2 "Number of years for PeriodLength : ${PeriodLengthInYears}"
349    PeriodLengthInDays=0
350    i=0
[619]351    until [ $i -ge $PeriodLengthInYears ] ; do
352      (( PeriodLengthInDays = PeriodLengthInDays + $( IGCM_date_DaysInYear $(( year + i )) ) ))
353      (( i=i+1 ))
354    done
[544]355    ;;
356  *M|*m)
[619]357    PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[mM]//' )
[731]358    echo
[544]359    IGCM_debug_Print 2 "Number of months for PeriodLength : ${PeriodLengthInMonths}"
360    PeriodLengthInDays=0
361    i=0
[619]362    until [ $i -ge $PeriodLengthInMonths ] ; do
[829]363      if [ $(( 10#${month} + ${i} )) -lt 13 ] ; then
364        (( PeriodLengthInDays  = PeriodLengthInDays + $( IGCM_date_DaysInMonth $year $(( 10#${month} + ${i} )) ) ))
[619]365      else
[829]366        (( PeriodLengthInDays  = PeriodLengthInDays + $( IGCM_date_DaysInMonth $year $(( 10#${month} + ${i} - 12 )) ) ))
[619]367      fi
368      (( i=i+1 ))
369    done
[544]370    ;;
[619]371  *D|*d)
[544]372    PeriodLengthInMonths=0
[731]373    PeriodLengthInDays=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[dD]//' )
374    echo
375    IGCM_debug_Print 2 "Number of days for PeriodLength : ${PeriodLengthInDays}";;
[619]376  *)
[890]377    IGCM_debug_Exit "IGCM_config_DaysInPeriodLength " ${config_UserChoices_PeriodLength} " invalid period length : choose in *Y, *M, *D."
[544]378    IGCM_debug_Verif_Exit ;;
379  esac
[118]380
[890]381  IGCM_debug_PopStack "IGCM_config_DaysInPeriodLength"
[380]382}
[118]383
[380]384#===================================
[890]385function IGCM_config_DateCoherency
[891]386{
[890]387  IGCM_debug_PushStack "IGCM_config_DateCoherency"
388
389  echo
390  IGCM_debug_Print 1 "IGCM_config_DateCoherency"
391  echo
392
393  typeset Length VerifiedPeriodDateBegin VerifiedPeriodDateEnd
394
395  # check coherency between (PeriodDateBegin, PeriodDateEnd) and (DateBegin, CumulPeriod, PeriodLength)
396  # DateBegin + CumulPeriod*PeriodLength = PeriodDateBegin
397  echo
[922]398
[890]399  case ${config_UserChoices_PeriodLength} in
400  *Y|*y)
401    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInYears} ))Y )
402    ;;
403  *M|*m)
404    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInMonths} ))M )
405    ;;
406  *D|*d)
407    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInDays} ))D )
408    ;;
409  esac
[892]410  VerifiedPeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} ${Length}-1 )
[890]411
[892]412  if [ ${VerifiedPeriodDateEnd} != ${PeriodDateEnd} ] ; then
413    IGCM_debug_Print 1 "From run.card PeriodDateEnd is not consistent with DateBegin and CumulPeriod."
414    IGCM_debug_Print 1 "We have DateBegin = ${DateBegin}"
415    IGCM_debug_Print 1 "We have CumulPeriod = ${CumulPeriod}"
416    IGCM_debug_Print 1 "We have PeriodDateEnd = ${PeriodDateEnd}"
417    IGCM_debug_Print 1 "We have VerifiedPeriodDateEnd = ${VerifiedPeriodDateEnd}"
418    IGCM_debug_Print 1 "You must have change run.card in an inconsistent way."
419
420    IGCM_debug_Exit "STOP here to avoid further issues."
421  fi
422
[890]423  # PeriodDateBegin + PeriodLength = PeriodDateEnd
424  VerifiedPeriodDateBegin=$( IGCM_date_AddDaysToGregorianDate ${VerifiedPeriodDateEnd} $(( ${PeriodLengthInDays} * -1 )) )
425
426  IGCM_debug_PopStack "IGCM_config_DateCoherency"
427}
428
429
430#===================================
[380]431function IGCM_config_Check
432{
[544]433  IGCM_debug_PushStack "IGCM_config_Check"
[380]434
[544]435  # If one of the following modulo is not zero :
436  # we will issue an error then explain and exit in
437  # AA_job IGCM_debug_Verif_Exit call before binary submission
[380]438
[544]439  echo
440  IGCM_debug_Print 1 "IGCM_config_Check"
441  echo
442
443  typeset i
444
[554]445  # Check RebuildFrequency against key frequencies : PeriodLength ; PackFrequency ; TimeSeriesFrequency ; SeasonalFrequency
[545]446  if ( [ ! X${config_Post_RebuildFrequency} = X${NULL_STR} ] && [ ! X${config_Post_RebuildFrequency} = XNONE ] ) ; then
[544]447    AsynchronousRebuild=true
448    IGCM_debug_Print 1 "Asynchronous rebuild has been activated."
[380]449    echo
[544]450    # modulo (RebuildFrequency and PeriodLength/TimeSeriesFrequency/SeasonalFrequency) must be zero
451    IGCM_debug_Print 1 "Check coherence between RebuildFrequency and PeriodLength"
452    IGCM_post_CheckModuloFrequency config_Post_RebuildFrequency config_UserChoices_PeriodLength
[554]453    IGCM_debug_Print 1 "Check coherence between PackFrequency and RebuildFrequency"
454    IGCM_post_CheckModuloFrequency config_Post_PackFrequency config_Post_RebuildFrequency
[544]455    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and RebuildFrequency"
456    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_Post_RebuildFrequency
457    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and RebuildFrequency"
458    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency config_Post_RebuildFrequency
459  else
460    AsynchronousRebuild=false
461    IGCM_debug_Print 1 "Asynchronous rebuild has not been activated"
462    IGCM_debug_Print 1 "Proceed with standard post-treatment pathway"
[380]463    echo
[554]464    #modulo (PeriodLength and TimeSeriesFrequency/SeasonalFrequency) must be zero
[544]465    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and PeriodLength"
466    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_UserChoices_PeriodLength
467    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and PeriodLength"
468    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency   config_UserChoices_PeriodLength
469  fi
[380]470
[619]471  # Check PackFrequency against other key frequencies
[554]472  # Modulo (PackFrequency and TimeSeriesFrequency/SeasonalFrequency and PeriodLenght) must be zero
473  if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
474    Pack=true
475    #
476    IGCM_debug_Print 1 "Check coherence between PackFrequency and PeriodLength"
477    IGCM_post_CheckModuloFrequency config_Post_PackFrequency config_UserChoices_PeriodLength
478    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and PackFrequency"
479    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_Post_PackFrequency
480    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and PackFrequency"
481    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency config_Post_PackFrequency
482  else
483    Pack=false
484  fi
485
[890]486  # modulo (TimeSeriesFrequency and all Chunck2D) must be zero
[544]487  NbJob=${#CHUNCK2D_SIZE[@]}
488  i=0
489  until [ $i -ge $NbJob ]; do
490    value=${CHUNCK2D_SIZE[${i}]}
491    IGCM_debug_Print 1 "Check coherence between All Chunck2D frequency and TimeSeriesFrequency"
492    IGCM_post_CheckModuloFrequency value config_Post_TimeSeriesFrequency
[619]493    case ${value} in
494    *Y|*y) ;;
[786]495    *)
496      IGCM_debug_Print 1 "All ChunckJob2D frequency must be expressed in year *Y|*y in comp.card"
[544]497      IGCM_debug_Exit "This will stop the job" ;;
498    esac
499    (( i=i+1 ))
500  done
[387]501
[890]502  # modulo (TimeSeriesFrequency and all Chunck3D) must be zero
[544]503  NbJob=${#CHUNCK3D_SIZE[@]}
504  i=0
505  until [ $i -ge $NbJob ]; do
506    value=${CHUNCK3D_SIZE[${i}]}
507    IGCM_debug_Print 1 "Check coherence between All Chunck3D frequency and TimeSeriesFrequency"
508    IGCM_post_CheckModuloFrequency value config_Post_TimeSeriesFrequency
[619]509    case ${value} in
510    *Y|*y) ;;
[786]511    *)
512      IGCM_debug_Print 1 "All ChunckJob3D frequency must be expressed in year *Y|*y in comp.card"
[544]513      IGCM_debug_Exit "This will stop the job" ;;
514    esac
515    (( i=i+1 ))
516  done
[380]517
[890]518  # check to be sure there is enough space on temporary filesystems to run
[731]519  echo
[651]520  IGCM_debug_Print 1 "Check if there is enough space on temporary filesystem"
521  IGCM_sys_check_quota
522
[933]523  # check to be sure that RUN_DIR_PATH, that will be removed is not pointing to an important directory
524  echo
525  IGCM_debug_Print 1 "Check where RUN_DIR_PATH variable is pointing to"
526  IGCM_sys_check_path
527
528
[544]529  IGCM_debug_PopStack "IGCM_config_Check"
[118]530}
531
532#===================================
[2]533function IGCM_config_PeriodStart
534{
[544]535  IGCM_debug_PushStack "IGCM_config_PeriodStart"
[2]536
[544]537  echo
538  IGCM_debug_Print 1 "IGCM_config_PeriodStart"
539  echo
[2]540
[544]541  if ( ${FirstInitialize} ) ; then
542    #================================================#
543    #         Initialize date/period information     #
544    #================================================#
[2]545
[890]546    IGCM_date_GetYearMonthDay ${DateBegin} year month day
547    IGCM_config_DaysInPeriodLength
[2]548
[939]549    PeriodDateBegin=${DateBegin}
[544]550    PeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( ${PeriodLengthInDays} - 1 )) )
[939]551    CumulPeriod=1
[2]552
[544]553    #=================================================#
554    #              Write updated run.card             #
555    #=================================================#
[2]556
[544]557    #Correct run.card Configuration for this period
558    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin ${PeriodDateBegin}
559    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd ${PeriodDateEnd}
560    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod ${CumulPeriod}
561    if [ X$( grep "SubmitPath" ${SUBMIT_DIR}/run.card ) != X ] ; then
562      IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration SubmitPath ${SUBMIT_DIR}
563    fi
[427]564
[544]565    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Running"
[619]566
[544]567  else
568    #================================================#
569    #         The file run.card allready exist       #
570    #================================================#
[2]571
[544]572    #Test state of run in run.card
573    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodState
574    if ( [ ${run_Configuration_PeriodState} != "Running" ] && [ ${run_Configuration_PeriodState} != "OnQueue" ] && [ ${run_Configuration_PeriodState} != "Continue" ] ) ; then
[619]575      echo
[544]576      IGCM_debug_Print 1 "!! Error in run.card with PeriodState : " ${run_Configuration_PeriodState} "!!"
577      IGCM_debug_Print 1 "Try running ${libIGCM}/clean_month.job to fix this"
578      echo
579      IGCM_debug_Exit
580      IGCM_debug_Verif_Exit
581    fi
[2]582
[544]583    #===================================#
[786]584    #        Read updated run.card      #
[544]585    #===================================#
[2]586
[544]587    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin
588    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd
589    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod
[2]590
[544]591    PeriodDateBegin=$( IGCM_date_ConvertFormatToGregorian ${run_Configuration_PeriodDateBegin} )
592    PeriodDateEnd=$( IGCM_date_ConvertFormatToGregorian ${run_Configuration_PeriodDateEnd} )
[939]593    CumulPeriod=${run_Configuration_CumulPeriod}
[2]594
[1028]595    LastPeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate $( IGCM_date_ConvertFormatToGregorian ${PeriodDateBegin} ) -1 )
[471]596
[544]597    if [ ${Period} = 1 ]; then
[910]598      # save last Job output and current run.card
599      typeset Potential
600      IGCM_sys_Cd ${SUBMIT_DIR}
601      #
602      IGCM_debug_Print 2 "Save previous ksh job output"
603      for Potential in $( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.[0-9][0-9][0-9][0-9][0-9][0-9] ) ; do
604        if [ X${Pack} = Xtrue ] ; then
[911]605          ( IGCM_sys_TestFileBuffer  ${R_BUF_KSH}/${Potential} ) || IGCM_sys_PutBuffer_Out ${Potential} ${R_BUF_KSH}/${Potential}.$$ rw
[910]606        else
[911]607          ( IGCM_sys_TestFileArchive ${R_OUT_KSH}/${Potential} ) || IGCM_sys_Put_Out ${Potential} ${R_OUT_KSH}/${Potential}.$$ rw
[910]608        fi
609      done
610      #
611      IGCM_debug_Print 2 "Save current run.card"
[544]612      IGCM_card_CheckConflict run.card
[634]613      if [ X${Pack} = Xtrue ] ; then
[910]614        IGCM_sys_PutBuffer_Out ${SUBMIT_DIR}/run.card ${R_BUF_KSH}/run.card rw
[634]615      else
[910]616        IGCM_sys_Put_Out ${SUBMIT_DIR}/run.card ${R_OUT_KSH}/run.card rw
[634]617      fi
[910]618      #
[544]619      IGCM_sys_Cd ${RUN_DIR}
620    else
621      unset FileToBeDeleted
622    fi
[2]623
[892]624    # Determine number of day(s) in PeriodLength
[544]625    IGCM_date_GetYearMonthDay $PeriodDateBegin year month day
[890]626    IGCM_config_DaysInPeriodLength
[2]627
[892]628    # Check coherency between (PeriodDateBegin, PeriodDateEnd) and (DateBegin, CumulPeriod, PeriodLength)
629    IGCM_config_DateCoherency
630    # And EXIT if not OK
631    IGCM_debug_Verif_Exit
632
[544]633    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Running"
[890]634  fi
[2]635
[544]636  # BEGIN: SHOULD GO IN A FUNCTION FROM libIGCM_date.ksh
637  # Compute year_m1 and year_p1 (year minus 1Y and year plus 1Y)
638  year_m1=$(( year - 1 ))
[619]639  year_p1=$(( year + 1 ))
[544]640  # Compute month_m1 (month minus 1M)
641  # Compute yyyymm_m1 (yyyymm minus 1M)
[829]642  month_m1=$(( 10#${month} - 1 ))
[544]643  if [ ${month_m1} = 0 ]; then
644    month_m1=12
645    yyyymm_m1=${year_m1}12
646  elif [ ${month_m1} -le 9 ]; then
647    month_m1=0${month_m1}
648    yyyymm_m1=${year}${month_m1}
649  else
650    yyyymm_m1=${year}${month_m1}
651  fi
652  # Compute month_p1 (month plus 1M)
653  # Compute yyyymm_p1 (yyyymm plus 1M)
[829]654  month_p1=$(( 10#${month} + 1 ))
[544]655  if [ ${month_p1} = 13 ]; then
656    month_p1=01
657    yyyymm_p1=${year_p1}01
658  elif [ ${month_p1} -le 9 ]; then
659    month_p1=0${month_p1}
660    yyyymm_p1=${year}${month_p1}
661  else
662    yyyymm_p1=${year}${month_p1}
663  fi
664  #IGCM_debug_Print 1 "jg 1 month_m1 = ${month_m1} month_p1 = ${month_p1} "
665  #IGCM_debug_Print 1 "jg 1 calculate yyyymm_m1 = ${yyyymm_m1} "
666  #IGCM_debug_Print 1 "jg 1 calculate yyyymm_p1 = ${yyyymm_p1} "
[522]667
[799]668  #===================================================================#
669  # Calculate CyclicYear to be used for looping over a given forcing  #
670  # period. Add CyclicBegin and CyclicEnd in config.card UserChoices. #
671  #===================================================================#
672
673  # To use the variable CyclicYear, one must add in config.card CyclicBegin and CyclicEnd.
674  # CyclicBegin is the first year in the cycle. CyclicEnd is the last year included in the cycle.
675  if ( [ ! X${config_UserChoices_CyclicBegin} = X ] && [ ! X${config_UserChoices_CyclicEnd} = X ] ) ; then
676    CycleNb=$(( ${config_UserChoices_CyclicEnd} - ${config_UserChoices_CyclicBegin} + 1 ))
[819]677    CyclicYear_p1=NOTDEFINED
678
679    # For current year
680    yeartmp=$year
681    diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
682    while [ $diffy -lt 0 ] ; do
683      yeartmp=$(( ${yeartmp} + ${CycleNb} ))
684      diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
685    done
686    CyclicYear=$(( ( ${diffy} % ${CycleNb} ) + ${config_UserChoices_CyclicBegin} ))
687
688    # For next coming year
689    yeartmp=$(( $year + 1 ))
690    diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
691    while [ $diffy -lt 0 ] ; do
692      yeartmp=$(( ${yeartmp} + ${CycleNb} ))
693      diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
694    done
695    CyclicYear_p1=$(( ( ${diffy} % ${CycleNb} ) + ${config_UserChoices_CyclicBegin} ))
696
[812]697    IGCM_debug_Print 1 "CyclicYear   = ${CyclicYear}, CyclicYear_p1 = ${CyclicYear_p1}, current year=$year"
[799]698  else
699    CyclicYear="ERROR_CyclicYear_Variable_Not_Defined"
[812]700    CyclicYear_p1="ERROR_CyclicYear_p1_Variable_Not_Defined"
[819]701    IGCM_debug_Print 1 "CyclicYear wont be use without adding CyclicBegin and CyclicEnd in config.card"
[799]702  fi
703
[544]704  # END: SHOULD GO IN A FUNCTION FROM libIGCM_date.ksh
[522]705
[544]706  #===================================================================#
707  # Prepare variables available for ${COMP}.card and ${COMP}.driver   #
[786]708  #             But available to any son functions                    #
[544]709  #===================================================================#
[2]710
[544]711  # Period Length In Days between DateBegin and DateCurrent (at end of period == PeriodDateEnd !)
712  (( SimulationLengthInDays = $( IGCM_date_DaysBetweenGregorianDate ${PeriodDateEnd} ${DateBegin} ) + 1 ))
[2]713
[544]714  # Debug Print :
[731]715  echo
[544]716  IGCM_debug_Print 1 "IGCM_config_PeriodStart : Before Execution"
717  IGCM_debug_Print 1 "Year of simulation      : ${year}"
718  IGCM_debug_Print 1 "Month of simulation     : ${month}"
719  IGCM_debug_Print 1 "PeriodLengthInDays      : ${PeriodLengthInDays}"
720  IGCM_debug_Print 1 "PeriodDateBegin         : ${PeriodDateBegin}"
721  IGCM_debug_Print 1 "PeriodDateEnd           : ${PeriodDateEnd}"
722  IGCM_debug_Print 1 "SimulationLengthInDays  : ${SimulationLengthInDays}"
723  IGCM_debug_Print 1 "ExperienceLengthInDays  : ${ExperienceLengthInDays}"
[2]724
[544]725  #================================================================#
726  #         Prepare variables available for comp_finalyze          #
727  #================================================================#
[2]728
[544]729  # Period for save files
[939]730  DatesPeriod=${PeriodDateBegin}_${PeriodDateEnd}
[2]731
[544]732  # Prefix for save files of this period
[939]733  PREFIX=${config_UserChoices_JobName}_${DatesPeriod}
[2]734
[544]735  # List of files that will be deleted in RUN_DIR after run
736  [ -f stack ] && FileToBeDeleted[0]="stack"
[2]737
[544]738  # Test if the same run as already been saved :
739  if [ X${JobType} = XRUN ] ; then
740    if [ ${DRYRUN} -le 0 ] ; then
741      if ( IGCM_sys_TestFileBuffer ${R_BUF_KSH}/${PREFIX}_${Exe_Output} ) ; then
[619]742        IGCM_debug_Exit "IGCM_config_PeriodStart" "You are currently RErunning an old job."
743        IGCM_debug_Print 1 "Because of readonly permissions, you can't RErun a job when saved files"
744        IGCM_debug_Print 1 " are still in the ARCHIVE directory. You must deleted those files, or "
745        IGCM_debug_Print 1 " the whole ${R_SAVE} tree. See clean_month.job in ${libIGCM} directory."
746        IGCM_debug_Print 1 " This exit has been initiated because at least ${R_BUF_KSH}/${PREFIX}_${Exe_Output} exists."
747        IGCM_debug_Verif_Exit
[544]748      fi
[2]749    fi
[544]750  else
751    if ( IGCM_sys_TestFileBuffer ${R_BUF_KSH}/${PREFIX}_${Exe_Output} ) ; then
752      IGCM_debug_Print 1 "IGCM_config_PeriodStart" "RErun an old job. Allowed in DEBUG or DEV mode."
753    fi
754  fi
[128]755
[939]756  #================================================================#
757  #       Prepare variables available for binary execution         #
[1009]758  #       Call function for creation of run script                 #
759  #       Only done once per job                                   #
[939]760  #================================================================#
[616]761
[1009]762  if [ ${Period} -eq 1 ]; then
763    typeset ExeNameIn ExeNameFirst CompNameFirst comp i j
764    typeset tempvar tempvarMPI tempvarNOD NbElts NbExec
[616]765
[1009]766    PROCESSUS_NUMBER=0
767    NbExec=0
768    i=0
[939]769
[1009]770    OK_PARA_MPI=false
771    OK_PARA_OMP=false
772    OK_PARA_NOD=false
773    OK_PARA_MPMD=false
[616]774
[1009]775    for comp in ${config_ListOfComponents[*]} ; do
[616]776
[1009]777      IGCM_debug_Print 1 ${comp}
[616]778
[1009]779      eval ExeNameIn=\${config_Executable_${comp}[0]}
[616]780
[1009]781      # NO order in config.card for parallelized values !
782      # just use suffix : MPI , OMP and NOD (for number of NODes.)
[616]783
[1009]784      # NOD is the number of NODes allocated
785      eval ${comp}_PROC_NOD=0
[616]786
[1009]787      # MPI is the number of MPI processus per nodes
788      eval ${comp}_PROC_MPI=0
[616]789
[1009]790      # OMP is the number of OpenMP threads per MPI processus
791      eval ${comp}_PROC_OMP=0
[619]792
[1009]793      # Only if we really have an executable for the component :
794      if ( [ "X${ExeNameIn}" != X\"\" ] && [ "X${ExeNameIn}" != "Xinca.dat" ] ) ; then
[616]795
[1009]796        # Keep the first executable found and the first CompName
797        ExeNameFirst=${ExeNameIn}
798        CompNameFirst=${comp}
[616]799
[1009]800        # Are we a second executable?
801        (( NbExec = NbExec + 1 ))
[616]802
[1009]803        # set 1 MPI task, 1 OpenMP thread and 1 node as default
804        eval ${comp}_PROC_MPI=1
805        eval ${comp}_PROC_OMP=1
806        eval ${comp}_PROC_NOD=1
[616]807
[1009]808        eval NbElts=\${#config_Executable_${comp}[@]}
[616]809
[1009]810        if [ ${NbElts} -ge 2 ] ; then
811          (( j = 2 ))
812          while [ $j -lt ${NbElts} ] ; do
813            eval tempvar=\${config_Executable_${comp}[${j}]}
814            IGCM_debug_Print 2 ${tempvar}
[616]815
[1009]816            if [ X${tempvar} = X ] ; then
817              IGCM_debug_Print 2 "Error reading MPI/OMP parameters !!!"
818              IGCM_debug_Exit "Check your config.card. Exit now"
819              IGCM_debug_Verif_Exit
820            fi
[616]821
[1009]822            case ${tempvar} in
823            *[mM][pP][iI]*)
824              # Read MPI parameter for composante
825              eval ${comp}_PROC_MPI=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/MPI//" )
826              OK_PARA_MPI=true;;
827            *[oO][mM][pP]*)
828              # Read OMP parameter for composante
829              eval ${comp}_PROC_OMP=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/OMP//" )
830              OK_PARA_OMP=true;;
831            *[nN][oO][dD]*)
832              # Read NOD (NumBer of Nodes) parameter for composante
833              eval ${comp}_PROC_NOD=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/NOD//" )
834              OK_PARA_NOD=true
835              OK_PARA_MPI=true;;
836            esac
837            (( j = j + 1 ))
838          done
839        fi
840        eval tempvarMPI=\${${comp}_PROC_MPI}
841        eval tempvarNOD=\${${comp}_PROC_NOD}
842        eval tempvarOMP=\${${comp}_PROC_OMP}
843
844        (( PROCESSUS_NUMBER = PROCESSUS_NUMBER + tempvarMPI * tempvarNOD * tempvarOMP ))
[616]845      fi
[1009]846      (( i=i+1 ))
847    done
[616]848
[1009]849    # set MPMD mode if more than 2 executable names.
850    [ ${NbExec} -ge 2 ] && OK_PARA_MPMD=true
[616]851
[1009]852    # Verification of BATCH_NUM_PROC_TOT total number of processors set in job header.
853    if [ X${BATCH_NUM_PROC_TOT} != X ] ; then
854      # BATCH_NUM_PROC_TOT is set
855      if ( ${OK_PARA_MPI} ) ; then
856        IGCM_debug_Print 1 "MPI/OMP/NOD found into config.card and BATCH_NUM_PROC_TOT = ${BATCH_NUM_PROC_TOT} "
857      else
858        # with previous method.
859        if [ ${BATCH_NUM_PROC_TOT} -gt 1 ] ; then
860          # with more than 1 proc
861          if ( ${OK_PARA_MPMD} ) ; then
862            # with MPMD ie CPL/oasis method
863            IGCM_debug_Print 2 "Use default number of MPI tasks for this machine : "
864            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_OCE} for OCE"
865            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_CPL} for CPL"
866            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_ATM} for ATM"
867            OK_PARA_MPI=true
868            CPL_PROC_MPI=${DEFAULT_NUM_PROC_CPL}
869            OCE_PROC_MPI=${DEFAULT_NUM_PROC_OCE}
870            ATM_PROC_MPI=${DEFAULT_NUM_PROC_ATM}
871            PROCESSUS_NUMBER=${DEFAULT_NUM_PROC_TOTAL}
872          else
873            # with have only one executable
874            IGCM_debug_Print 2 "Use ${BATCH_NUM_PROC_TOT} MPI tasks for ${CompNameFirst} : ${ExeNameFirst} "
875            OK_PARA_MPI=true
876            eval ${CompNameFirst}_PROC_MPI=${BATCH_NUM_PROC_TOT}
877            PROCESSUS_NUMBER=${BATCH_NUM_PROC_TOT}
878          fi
[616]879        else
[1009]880          PROCESSUS_NUMBER=1
[616]881        fi
[1009]882      fi
883      # Verification with PBS parameter
884      if [ ${BATCH_NUM_PROC_TOT} -ne ${PROCESSUS_NUMBER} ] ; then
885        IGCM_debug_Exit "ERROR with parallelization parameters !"
886        IGCM_debug_Print 2 "Job header variable BATCH_NUM_PROC_TOT = ${BATCH_NUM_PROC_TOT} "
887        IGCM_debug_Print 2 "is the total number of _processors_ reserved."
888        IGCM_debug_Print 2 "It is not equal to the sum of _processus_  = ${PROCESSUS_NUMBER}."
889        IGCM_debug_Verif_Exit
890      fi
891      NUM_PROC_CPL=${CPL_PROC_MPI} # for backward compatibility
892      NUM_PROC_OCE=${OCE_PROC_MPI} # for backward compatibility
893      NUM_PROC_ATM=${ATM_PROC_MPI} # for backward compatibility
894    else # BATCH_NUM_PROC_TOT=""
895      if ( ${OK_PARA_MPI} ) ; then
896        IGCM_debug_Exit "ERROR : missing value for ${BATCH_NUM_PROC_TOT} processors,"
897        IGCM_debug_Print 2 "You have parallel parameters in config->Executable->list."
898        IGCM_debug_Print 2 "Please add BATCH_NUM_PROC_TOT variable in job header as well."
899        IGCM_debug_Exit "Exit now."
900        IGCM_debug_Verif_Exit
[619]901      else
[1009]902        # sequential case !
903        if [ ${PROCESSUS_NUMBER} -eq 0 ] ; then
904          (( PROCESSUS_NUMBER = 1 ))
905          IGCM_debug_Print 2 "PROCESSUS_NUMBER is all 0 (sequential use of old definition in config->Executable->list)."
906          IGCM_debug_Print 2 "We set it to 1."
907        fi
[616]908      fi
909    fi
910
[1009]911    IGCM_debug_Print 1 "MPI/OMP treatment PROCESSUS_NUMBER = ${PROCESSUS_NUMBER}"
[616]912
[1009]913    IGCM_sys_build_execution_scripts
914  fi
[616]915
[544]916  ExecutionFail=false
[473]917
[544]918  IGCM_debug_PopStack "IGCM_config_PeriodStart"
[2]919}
920
921#===================================
[523]922function IGCM_config_SaveSourceModifications
923{
[544]924  IGCM_debug_PushStack "IGCM_config_SaveSourceModifications"
[523]925
[544]926  typeset ExeOutDateMax listVarEnv
927  ExeOutDateMax=$1
[523]928
[544]929  listVarEnv="ExeOutDateMax,R_OUT_EXE,PREFIX,SUBMIT_DIR"
930  IGCM_sys_RshMaster "\
[545]931    . ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh; \
[544]932    export ExeOutDateMax=${ExeOutDateMax};\
[545]933    export R_OUT_EXE=${R_OUT_EXE};\
934    export PREFIX=${PREFIX};\
935    export SUBMIT_DIR=${SUBMIT_DIR};\
936    export listVarEnv=${listVarEnv};\
937    Script_Output=out_SaveSourceModifications;\
938    IGCM_sys_Qsub ${libIGCM}/SaveSourceModifications.job ${ExeOutDateMax} ${R_OUT_EXE} ${PREFIX} ${SUBMIT_DIR}"
[523]939
[544]940  IGCM_debug_PopStack "IGCM_config_SaveSourceModifications"
[523]941}
942
943#===================================
[2]944function IGCM_config_PeriodEnd
945{
[544]946  IGCM_debug_PushStack "IGCM_config_PeriodEnd"
[2]947
[544]948  echo
949  IGCM_debug_Print 1 "IGCM_config_PeriodEnd"
950  echo
[2]951
[622]952  if [ ${DRYRUN} -le 1 ] ; then
[457]953
[622]954    IGCM_debug_Print 1 "Check components binary : size and creation date"
[2]955
[622]956    typeset LS_comp LS_bin ExeDate ExeCpuLog NextExeSize LastCompExeSize
957    typeset comp i
958    typeset ExeNameIn ExeNameOut UpdateExe ExeSecDateMax
[2]959
[622]960    #==================================#
961    #        Get last Exe Size         #
962    #==================================#
[2]963
[622]964    (( i=0 ))
965    if ( ${FirstInitialize} ) ; then
966      run_Log_LastExeSize=""
967      for comp in ${config_ListOfComponents[*]} ; do
968        run_Log_LastExeSize[$i]=0
969        (( i=i+1 ))
970      done
971    else
972      IGCM_card_DefineArrayFromOption ${SUBMIT_DIR}/run.card Log LastExeSize
973    fi
974    #==================================#
975    #         And Build ExeDate        #
976    #==================================#
[2]977
[622]978    # ExeDate = ATM_Jun_12_09:34-SRF_Jun_12_09:34-OCE_Jun_12_09:34-ICE_Jun_12_09:34-CPL_Jun_12_09:33
979    # Would be nice to have next line but no way to format ls output (need to ls -l --time-style "+%Y-%m-%dT%H:%M")
980    # ExeDate = ATM_2009-06-12T09:34+SRF_2009-06-12T09:34+OCE_2009-06-12T09:34+ICE_2009-06-12T09:34+CPL_2009-06-12T09:34
981    ExeDate=""
982    NextExeSize="( "
983    (( i=0 ))
984    UpdateExe=false
985    (( ExeSecDateMax = 0 ))
986    for comp in ${config_ListOfComponents[*]} ; do
[2]987
[622]988      IGCM_debug_Print 3 ${comp}
[2]989
[622]990      eval ExeNameIn=\${config_Executable_${comp}[0]}
991      eval ExeNameOut=\${config_Executable_${comp}[1]}
992      # Only if we really have an executable for the component :
993      if [ X${ExeNameIn} = X\"\" ] ; then
994        # If there is no exe file for this component
995        (( ExeSize=0 ))
[619]996      else
[939]997        LS_bin=${R_EXE}/${ExeNameIn}
[622]998        IGCM_sys_FileSize ${LS_bin} ExeSize
999
1000        set +A LS_comp -- $( LC_TIME=en_US ls -l ${LS_bin} )
1001        if [ X${ExeDate} = X ] ; then
1002          # First component exe date
1003          ExeDate=${comp}_${LS_comp[5]}_${LS_comp[6]}
1004        else
1005          ExeDate=${ExeDate}-${comp}_${LS_comp[5]}_${LS_comp[6]}
1006        fi
1007        ExeDate=${ExeDate}_${LS_comp[7]}
[619]1008      fi
[2]1009
[622]1010      if [ ${i} -eq 0 ] ; then
1011        # First component
1012        NextExeSize="( "${ExeSize}
[619]1013      else
[622]1014        NextExeSize=${NextExeSize}", "${ExeSize}
[619]1015      fi
[939]1016      LastCompExeSize=${run_Log_LastExeSize[$i]}
[622]1017      (( i=i+1 ))
[523]1018
[622]1019      if [ ${ExeSize} -ne ${LastCompExeSize} ] ; then
1020        if ( ${FirstInitialize} ) ; then
1021          IGCM_debug_Print 1 "Save first ${ExeNameIn} in ${R_OUT_EXE} !"
1022        else
1023          IGCM_debug_Print 1 "${ExeNameIn} has changed in ${R_EXE} !"
1024          IGCM_debug_Print 1 "Save latest ${ExeNameIn} in ${R_OUT_EXE} !"
[939]1025          FileToBeDeleted[${#FileToBeDeleted[@]}]=${ExeNameOut}
[622]1026        fi
[939]1027        IGCM_sys_Put_Out ${ExeNameOut} ${R_OUT_EXE}/${PREFIX}_${ExeNameIn} rw
[622]1028        UpdateExe=true
1029
[691]1030        # SD : switch off for now
1031        #IGCM_sys_GetDate_FichWork ${LS_bin} ExeSecDate
1032        #if [ $ExeSecDateMax -lt $ExeSecDate ] ; then
1033        #  ExeSecDateMax=$ExeSecDate
1034        #fi
[619]1035      fi
[622]1036    done
[2]1037
[622]1038    # SD : switch off for now
1039    #if ( ${UpdateExe} ) ; then
1040    #  echo "Launch SaveSourceModifications."
1041    #  IGCM_config_SaveSourceModifications ${ExeSecDateMax}
1042    #fi
[523]1043
[622]1044    NextExeSize=${NextExeSize}" )"
1045    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Log LastExeSize "${NextExeSize}"
[457]1046
[622]1047    if [ ${DRYRUN} -le 1 ] ; then
1048      tail -1500 ${Exe_Output} > ${Exe_Output}_tail.txt
1049      ExeCpuLog=$( gawk -f ${libIGCM}/libIGCM_sys/IGCM_add_out.awk ${Exe_Output}_tail.txt )
1050      RET=$?
1051      if [ $RET -eq 0 ] ; then
1052        # ExeCpuLog variable contents 5 fields
1053        echo "${CumulPeriod} ${PeriodDateBegin} ${PeriodDateEnd} ${ExeCpuLog} ${ExeDate}" |   \
1054          gawk '{printf("# %11d | %15s | %15s | %19s | %19s | %15.5f | %15.5f | %15.5f | %s\n", \
[544]1055            $1,$2,$3,$4,$5,$6,$7,$8,$9)}' >> ${SUBMIT_DIR}/run.card
[622]1056      fi
[939]1057      FileToBeDeleted[${#FileToBeDeleted[@]}]=${Exe_Output}_tail.txt
[619]1058    fi
[622]1059
[619]1060  fi
[2]1061
[1041]1062  #==================================#
1063  #         Save Job output          #
1064  #==================================#
1065  if [ X${Pack} = Xtrue ] ; then
1066    IGCM_sys_PutBuffer_Out ${Exe_Output} ${R_BUF_KSH}/${PREFIX}_${Exe_Output}
1067  else
1068    IGCM_sys_Put_Out ${Exe_Output} ${R_OUT_KSH}/${PREFIX}_${Exe_Output}
1069  fi
1070  FileToBeDeleted[${#FileToBeDeleted[@]}]=${Exe_Output}
1071
[622]1072  # All was right ? no ? then we stop.
[619]1073  IGCM_debug_Verif_Exit
[2]1074
[622]1075  # If all was OK, we can delete all files not necessary for next Job
[619]1076  echo
1077  IGCM_debug_Print 1 "Files that will be deleted before next period-run : "
[2]1078
[619]1079  if [ ${DRYRUN} -le 2 ] ; then
1080    for f in ${FileToBeDeleted[@]} ; do [ -f ${f} ] && ls -la $f ; [ -f ${f} ] && rm -f $f ; done
1081  else
1082    echo ${FileToBeDeleted[@]}
1083  fi
[2]1084
[922]1085  # Send some accounting element to the user if CumulPeriod=3
1086  if [ ${CumulPeriod} -eq 3 ] ; then
1087    echo
1088    IGCM_debug_Print 1 "Send email containing some accounting information : "
1089
1090    RealCpuTime=$( echo ${ExeCpuLog} | gawk '{print $3}' )
1091
1092    consumeHoursPerPeriod=$( echo "scale=6;${RealCpuTime}*${PROCESSUS_NUMBER}/3600" | bc )
1093
1094    consumeHoursPerWholeSimulation=$( echo "scale=6;${consumeHoursPerPeriod}/${PeriodLengthInDays}*${ExperienceLengthInDays}" | bc )
1095
1096    recommendedPeriodNb=$( echo "scale=6;24/${consumeHoursPerPeriod}*${PROCESSUS_NUMBER}" | bc )
1097
1098    IGCM_sys_SendMail Accounting
1099  fi
1100
[622]1101  #=================================================#
1102  #         Modification of libIGCM behaviour       #
1103  #=================================================#
[119]1104
[622]1105  # To use this function, one must copy libIGCM.card from ${libIGCM} directory
1106  # and put it in ${SUBMIT_DIR} directory. After modifications of ${SUBMIT_DIR}/libIGCM.card,
1107  # variables define inside [UserChanges] will be modified for next Period of libIGCM main loop.
[619]1108  if [ -f ${SUBMIT_DIR}/libIGCM.card ] ; then
1109    echo
1110    echo "########################################################################"
1111    echo "!!!                 Modification of libIGCM behaviour                !!!"
1112    echo
[119]1113
[619]1114    IGCM_debug_Print 1 "DefineArrayFromOption  : libIGCM_UserChanges in libIGCM.card"
1115    IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/libIGCM.card UserChanges
1116    IGCM_debug_Print 2 "libIGCM_UserChanges" ${libIGCM_UserChanges[*]}
[119]1117
[622]1118    # Special treatments for libIGCM internals
[619]1119    for option in ${libIGCM_UserChanges[*]} ; do
1120      IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/libIGCM.card UserChanges ${option}
[119]1121
[619]1122      echo "We will change : ${option}."
1123      eval echo "Previous value : " \${${option}}
1124      eval echo "Change to : " \${libIGCM_UserChanges_${option}}
[119]1125
[619]1126      eval ${option}=\${libIGCM_UserChanges_${option}}
[119]1127
[619]1128      case ${option} in
1129      config_UserChoices_DateEnd)
1130        IGCM_debug_PrintVariables 1 config_UserChoices_DateEnd
1131        DateEnd=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )
1132
[622]1133        # Period Length In Days between DateBegin and DateEnd
[619]1134        (( ExperienceLengthInDays=$( IGCM_date_DaysBetweenGregorianDate ${DateEnd} ${DateBegin} )  + 1 ))
1135        if [ ${ExperienceLengthInDays} -lt 0 ] ; then
1136          IGCM_debug_Print 1 "Problem with dates in libIGCM.card : ${DateEnd} < ${DateBegin} ! You must check that."
[939]1137          IGCM_debug_Exit "IGCM_PeriodEnd have wrong dates."
[619]1138          IGCM_debug_Verif_Exit
1139        fi
1140        ;;
1141      config_UserChoices_PeriodLength)
1142        IGCM_debug_Print 1  "Change config_UserChoices_PeriodLength=${config_UserChoices_PeriodLength}"
1143        ;;
1144      PeriodNb)
1145        IGCM_debug_Print 1  "Loop in main Job with ${PeriodNb} period(s)"
1146        ;;
1147      config_Post_RebuildFrequency)
1148        IGCM_debug_Print 1  "Change config_Post_RebuildFrequency=${config_Post_RebuildFrequency} : IGCM_post_Configure"
1149        IGCM_post_Configure
1150        ;;
1151      config_Post_TimeSeriesFrequency)
[653]1152        IGCM_debug_Print 1  "Change config_Post_TimeSeriesFrequency = ${config_Post_TimeSeriesFrequency} : IGCM_post_Configure"
[619]1153        IGCM_post_Configure
1154        ;;
1155      config_Post_SeasonalFrequency)
[653]1156        IGCM_debug_Print 1  "Change config_Post_SeasonalFrequency = ${config_Post_SeasonalFrequency} : IGCM_post_Configure"
[619]1157        IGCM_post_Configure
1158        ;;
1159      esac
1160    done
[119]1161
[619]1162    echo
1163    echo "########################################################################"
1164    echo
[544]1165  fi
[119]1166
[544]1167  #=================================================#
1168  #         Determine next computed period          #
1169  #=================================================#
[2]1170
[544]1171  PeriodDateBegin=$( IGCM_date_AddDaysToGregorianDate ${PeriodDateEnd} 1 )
1172  IGCM_date_GetYearMonthDay $PeriodDateBegin year month day
1173  year_m1=$(( year - 1 ))
1174  year_p1=$(( year + 1 ))
[890]1175  IGCM_config_DaysInPeriodLength
[544]1176  PeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${PeriodDateBegin} $(( ${PeriodLengthInDays} - 1 )) )
[2]1177
[544]1178  # Debug Print :
1179  echo
1180  IGCM_debug_Print 1 "IGCM_config_PeriodEnd : Preparing Next Execution"
1181  IGCM_debug_Print 1 "PeriodDateBegin       : ${PeriodDateBegin}"
1182  IGCM_debug_Print 1 "PeriodDateEnd         : ${PeriodDateEnd}"
1183  IGCM_debug_Print 1 "PeriodLengthInDays    : ${PeriodLengthInDays}"
[2]1184
[544]1185  PeriodDateBegin=$( IGCM_date_ConvertFormatToHuman ${PeriodDateBegin} )
1186  PeriodDateEnd=$( IGCM_date_ConvertFormatToHuman ${PeriodDateEnd} )
[2]1187
[544]1188  (( CumulPeriod = CumulPeriod + 1 ))
[2]1189
[544]1190  # Debug Print :
1191  echo
1192  IGCM_debug_Print 3 "PeriodDateBegin Human : ${PeriodDateBegin}"
1193  IGCM_debug_Print 3 "PeriodDateEnd Human   : ${PeriodDateEnd}"
1194  IGCM_debug_Print 3 "CumulPeriod           : ${CumulPeriod}"
[2]1195
[544]1196  #=================================================#
[786]1197  #             Write updated run.card              #
[544]1198  #=================================================#
[2]1199
[544]1200  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin ${PeriodDateBegin}
1201  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd ${PeriodDateEnd}
1202  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod ${CumulPeriod}
[2]1203
[544]1204  if ( ${FirstInitialize} ) ; then
1205    # It's no more the first time
1206    FirstInitialize=false
1207  fi
[446]1208
[544]1209  IGCM_debug_PopStack "IGCM_config_PeriodEnd"
[2]1210}
1211
1212#===================================
1213function IGCM_config_Finalize
1214{
[544]1215  IGCM_debug_PushStack "IGCM_config_Finalize"
[2]1216
[544]1217  echo
1218  IGCM_debug_Print 1 "IGCM_config_Finalize"
1219  echo
[2]1220
[873]1221  if ( $DEBUG_debug ) ; then
1222    # Inform the rabbitMQ queue
1223    if [ X${ActivateBigBro} = Xtrue ] ; then
[920]1224      # RabbitMQ message code
[873]1225      code=1100
[920]1226      # RabbitMQ message body
[1056]1227      Body=$( echo "{${genericSimulationID},\"msgCode\":\"${code}\",\"msgUID\":\"$(uuidgen)\",\"msgTimestamp\":\"$( date +"%Y-%m-%dT%H:%M:%S.%N%z" )\"}" )
[920]1228      # Fill the rabbitMQ queue
1229      IGCM_debug_sendAMQP
[873]1230    fi
1231  fi
1232
[544]1233  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
1234    #==========================#
1235    # End of entire simulation #
1236    #==========================#
[2]1237
[544]1238    # Mail notification
1239    IGCM_sys_SendMail
1240    #
1241    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Completed"
[920]1242    #
[544]1243    IGCM_debug_Print 1 "Normal End of computation."
[873]1244
[544]1245    if ( $DEBUG_debug ) ; then
[873]1246      if [ X${ActivateBigBro} = Xtrue ] ; then
[920]1247        # RabbitMQ message code
[873]1248        code=0100
[920]1249        # RabbitMQ message body
[1056]1250        Body=$( echo "{${genericSimulationID},\"msgCode\":\"${code}\",\"msgUID\":\"$(uuidgen)\",\"msgTimestamp\":\"$( date +"%Y-%m-%dT%H:%M:%S.%N%z" )\"}" )
[920]1251        # Fill the rabbitMQ queue
1252        IGCM_debug_sendAMQP
[873]1253      fi
[544]1254      echo
1255      IGCM_debug_Print 1 "Your files on ${R_SAVE} :"
1256      IGCM_sys_Tree ${R_SAVE}
[2]1257
[544]1258      IGCM_debug_Print 1 "Your files on ${R_BUFR} :"
1259      IGCM_sys_Tree ${R_BUFR}
1260    fi
1261  else
1262    #=================#
1263    # Submit next job #
1264    #=================#
[2]1265
[544]1266    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "OnQueue"
[2]1267
[544]1268    # Name of next Ksh Script output :
[939]1269    Script_Output=${Script_Output_Prefix}_${config_UserChoices_JobName}.$( printf "%06d" ${CumulPeriod} )
[2]1270
[544]1271    IGCM_debug_Print 1 "Submit next job"
[922]1272    # SUBMIT NEXT JOB from SUBMIT_DIR and come back in RUN_DIR
1273    IGCM_sys_Cd ${SUBMIT_DIR}
[544]1274    # Keep only the 5 latest ${Script_Output_Prefix}_${config_UserChoices_JobName}
1275    ScriptTot=$( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.?????? 2>/dev/null | wc -l )
1276    [ ${ScriptTot} -gt 5 ] && rm -f $( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.?????? | head -$(( ${ScriptTot} - 5 )) )
[861]1277    # Submit next job and come back
[544]1278    IGCM_sys_Qsub ${SUBMIT_DIR}/Job_${config_UserChoices_JobName}
[861]1279    IGCM_sys_Cd -
[544]1280  fi
[154]1281
[544]1282  # Supress Non Deleted Boundary files
[804]1283  #if [ ${DRYRUN} -le 2 ] ; then
1284  #  IGCM_comp_DelFixeBoundaryFiles
1285  #  ls -la
1286  #fi
[619]1287
[804]1288  # Clean ${RUN_DIR}=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
[544]1289  # Only for production run (No clean up in DEV or DEB mode)
1290  # and command sent from .. directory.
1291  IGCM_sys_Cd ..
[943]1292  [ X${JobType} = XRUN ] && IGCM_sys_RmRunDir -rf ${RUN_DIR_PATH}
[544]1293
[997]1294  # To be sure we flush message buffer before exiting the job
1295  FlushAMQP=TRUE
1296
[544]1297  IGCM_debug_PopStack "IGCM_config_Finalize"
[2]1298}
1299
1300#===================================
Note: See TracBrowser for help on using the repository browser.