Context Navigation

source: trunk/libIGCM/AA_create_ts @ 240

Last change on this file since 240 was 237, checked in by sdipsl, 14 years ago
-Add some default behavior to reach the backward compatibility
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 31.3 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 20:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- titane #!/bin/ksh
65	#-Q- titane ######################
66	#-Q- titane ## TITANE CEA ##
67	#-Q- titane ######################
68	#-Q- titane #MSUB -r TS # Nom du job
69	#-Q- titane #MSUB -N 1 # Reservation du noeud
70	#-Q- titane #MSUB -n 1 # Reservation du processus
71	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
72	#-Q- titane #MSUB -E "-j o"
73	#-Q- titane #MSUB -E "-S /bin/ksh"
74	#-Q- titane ##MSUB -e nco.out # Sortie standard
75	#-Q- titane ##MSUB -o nco.out # Sortie standard
76	#-Q- default #!/bin/ksh
77	#-Q- default ##################
78	#-Q- default ## DEFAULT HOST ##
79	#-Q- default ##################
80
81	# $Date$
82	# $Author$
83	# $Revision$
84	# IPSL (2006)
85	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
86
87	#set -eu
88	#set -vx
89
90	date
91
92	#-Q- sx8brodie export OMP_NUM_THREADS=1
93	#-Q- aix6 export OMP_NUM_THREADS=1
94
95	########################################################################
96
97	#D- Flag to determine if this job in a standalone mode
98	#D- Default : value from AA_job if any
99	StandAlone=${StandAlone:=true}
100
101	#D- Low level debug : to bypass lib test checks and stack construction
102	#D- Default : value from AA_job if any
103	libIGCM=${libIGCM:=::modipsl::/libIGCM}
104	# WARNING for StandAlone used : To run this script on some machine,
105	# you must check MirrorlibIGCM variable in sys library.
106	# If this variable is true, you must use libIGCM_POST path instead
107	# of your running libIGCM directory.
108
109	#D- Flag to determine allready produced time series. Empty if you start from the beginning
110	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
111	CompletedFlag=${CompletedFlag:=}
112
113	#D- Flag to determine task type this script will perform.
114	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
115	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
116	TsTask=${TsTask:=2D}
117
118	#D- Flag to determine if rebuild process has been performed asynchronously
119	#D- Possible value true or false.
120	#D- If true raw files has allready been patched by rebuild job
121	#D- If false create_ts will patch the raw files
122	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
123	RebuildFrequency=${RebuildFrequency:=true}
124
125	#D- Flag to determine job's output directory
126	#D- Default : value from libIGCM_post.ksh if any
127	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
128
129	#D- Increased verbosity (1, 2, 3)
130	#D- Default : value from AA_job if any
131	Verbosity=${Verbosity:=3}
132
133	#D- Low level debug : to bypass lib test checks and stack construction
134	#D- Default : value from AA_job if any
135	DEBUG_debug=${DEBUG_debug:=false}
136
137	#D- TEMPORARY Flag to determine atmospheric resolution
138	#D- Default : value from atmospheric driver if any
139	RESOL_ATM=${RESOL_ATM:=ALL}
140
141	#D- TEMPORARY Flag to determine ocean resolution
142	#D- Default : value from ocean driver if any
143	RESOL_OCE=${RESOL_OCE:=ORCA2}
144
145	#D- TEMPORARY Flag to determine ice resolution
146	#D- Default : value from ice driver if any
147	RESOL_ICE=${RESOL_ICE:=ORCA2}
148
149	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
150	#D- Default : value from ice driver if any
151	RESOL_MBG=${RESOL_MBG:=ORCA2}
152
153	#D- Flag to determine surface resolution
154	#D- Default : value from surface driver if any
155	RESOL_SRF=ALL
156
157	#D- Flag to determine surface resolution
158	#D- Default : value from surface driver if any
159	RESOL_SBG=ALL
160
161	########################################################################
162
163	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
164	( ${DEBUG_debug} ) && IGCM_debug_Check
165	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
166	( ${DEBUG_debug} ) && IGCM_card_Check
167	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
168	( ${DEBUG_debug} ) && IGCM_date_Check
169	#-------
170	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
171
172	########################################################################
173
174	#set -vx
175
176	# Useful cleaning function
177	MENAGE ()
178	{
179	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
180	}
181
182	########################################################################
183
184	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
185	IGCM_sys_Cd ${RUN_DIR_PATH}
186
187	if [ ${StandAlone} = true ] ; then
188	CARD_DIR=${SUBMIT_DIR}
189	else
190	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
191	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
192	fi
193
194	#
195	# First of all
196	#
197	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
198	typeset option
199	for option in ${config_UserChoices[*]} ; do
200	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
201	done
202	#
203	echo
204	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
205	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
206	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
207	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
208	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
209	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
210	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
211	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
212	echo
213
214	#==================================
215	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
216	#
217	# Determine component to take care of depending on kind of task create_ts will perform
218	#
219	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
220	#
221	# Determine period for time series. Default : value from AA_job or from launch_TS if any
222	#
223	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
224	#
225	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
226	#
227	# Determine Dimension of the produced time series : 2D or 3D
228	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
229	Dimension=2D
230	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
231	Dimension=3D
232	else
233	Dimension=""
234	fi
235
236
237	########################################################################
238	#
239	# Depending on requested time series fill following variables :
240	#
241	# - LISTE_FILE_${comp}[*]
242	# - LISTE_VARS_${file}[*]
243	# - LISTE_AXIS_${file}[*]
244	# - LISTE_PATCH_${file}[*]
245	########################################################################
246
247	#
248	# For each selected component determine which files need post-processing
249	#
250	DoJob=false
251	for comp in ${config_ListOfComponents[*]} ; do
252	#
253	ActiveComp=false
254	#
255	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
256	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
257	#
258	card=${CARD_DIR}/COMP/${compname}.card
259
260	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
261	#
262	ListFilesName=${compname}_OutputFiles_List
263	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
264	#
265	if [ X${FileName0} != X${NULL_STR} ] ; then
266	#
267	#IGCM_debug_Print 1 "Component : ${compname}"
268	#
269	# INITIALISATION
270	#
271	typeset i
272	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
273	((NbFiles=${FlagToRead}))
274	((i=${NbFiles}-2))
275	else
276	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
277	i=0
278	fi
279	#
280	until [ $i -ge $NbFiles ]; do
281	#
282	(( i_ = i+2 ))
283	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
284	eval flag_post=${flag_post_}
285	#
286	if [ X${flag_post} != XNONE ] ; then
287	#
288	# For each comp and each file who needs post-processing
289	# determine which variables need time series.
290	#
291	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
292	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
293	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
294	#
295	# If TimeSeriesVars list is empty we skip
296	#
297	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
298	#
299	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
300	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
301	(( i=i+3 ))
302	continue
303	#
304	fi
305	#
306	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
307	#
308	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
309	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
310	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
311	(( i=i+3 ))
312	continue
313	fi
314	#
315	# If we are not a Chunck type task, we skip files/variables handled by chunck task
316	#
317	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
318	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
319	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
320	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
321	(( i=i+3 ))
322	continue
323	fi
324	fi
325	#
326	# Now on we know we have things to do
327	#
328
329	#
330	# We need coordinates axis to properly fill produced time series
331	#
332	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
333	#
334	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
335	#
336	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
337	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
338	(( i=i+3 ))
339	continue
340	#
341	fi
342	#
343	# We need LIST of variables not allready produced (useful for standalone mode)
344	#
345	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
346	case ${FlagDir} in
347	*Y) TS_Dir=TS_YE ;;
348	*M) TS_Dir=TS_MO ;;
349	*D) TS_Dir=TS_DA ;;
350	HF) TS_Dir=TS_HF ;;
351	INS) TS_Dir=TS_INS ;;
352	esac
353	#
354	count=0
355	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
356	#
357	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
358	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
359	#
360	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
361	if [ ! $? = 0 ] ; then
362	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
363	ActiveComp=true
364	(( count = count + 1 ))
365	fi
366	done
367	#
368	if [ ${count} -gt 0 ] ; then
369	#
370	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
371	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
372	#
373	if [ ! X${RebuildFrequency} = Xtrue ] ; then
374	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
375	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
376	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
377	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
378	if [ Xload${Patch} != X ] ; then
379	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
380	eval load${Patch}=loaded
381	fi
382	done
383	fi
384	fi
385	fi
386	fi
387	(( i=i+3 ))
388	done
389	fi
390	#
391	if [ ${ActiveComp} = true ] ; then
392	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
393	fi
394	done
395
396	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
397	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
398
399	########################################################################
400	#
401	# DEFINE DATE FUNCTIONS
402	#
403	########################################################################
404
405	# Give the end of the next $1 period from $2 date
406	function create_ts_next_date {
407	IGCM_debug_PushStack "create_ts_next_date" $@
408	typeset Length
409	case ${config_UserChoices_PeriodLength} in
410	Y\|y)
411	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
412	IGCM_date_GetYearMonth ${1} year month
413	(( Length=0 ))
414	(( i=0 ))
415	until [ $i -ge $PeriodLengthInYears ] ; do
416	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
417	(( i=i+1 ))
418	done
419	;;
420	M\|m)
421	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
422	IGCM_date_GetYearMonth ${1} year month
423	(( year0 = year ))
424	if [ $(( month + 1 )) -lt 13 ] ; then
425	month0=$(( month + 1 ))
426	else
427	month0=$(( month + 1 - 12 ))
428	(( year = year0 + 1 ))
429	fi
430	(( Length=0 ))
431	(( i=0 ))
432	until [ $i -ge $PeriodLengthInMonths ] ; do
433	if [ $(( month0 + i )) -lt 13 ] ; then
434	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
435	else
436	(( year = year0 + 1 ))
437	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
438	fi
439	(( i=i+1 ))
440	done
441	;;
442	D\|d)
443	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
444	*)
445	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
446	IGCM_debug_Verif_Exit_Post ;;
447	esac
448	echo ${Length}
449
450	IGCM_debug_PopStack "create_ts_next_date"
451	}
452
453	# Give the end of the actual $1 period from $2 date
454	function create_ts_begin_date {
455	IGCM_debug_PushStack "create_ts_begin_date" $@
456	typeset Length
457	case ${config_UserChoices_PeriodLength} in
458	Y\|y)
459	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
460	IGCM_date_GetYearMonth ${1} year month
461	if [ X${2} = Xend ] ; then
462	(( year = year - PeriodLengthInYears + 1))
463	fi
464	(( Length=0 ))
465	(( i=0 ))
466	until [ $i -ge $PeriodLengthInYears ] ; do
467	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
468	(( i=i+1 ))
469	done
470	;;
471	M\|m)
472	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
473	IGCM_date_GetYearMonth ${1} year month
474	if [ X${2} = Xend ] ; then
475	(( month = month - PeriodLengthInMonths + 1 ))
476	fi
477	(( year0 = year ))
478	if [ $month -le 0 ] ; then
479	(( month = month + 12 ))
480	year=$( printf "%04i\n" $(( year - 1 )) )
481	fi
482	month=$( printf "%02i\n" ${month} )
483	(( Length=0 ))
484	(( i=0 ))
485	until [ $i -ge $PeriodLengthInMonths ] ; do
486	if [ $(( month + i )) -lt 13 ] ; then
487	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
488	else
489	(( year = year0 + 1 ))
490	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
491	fi
492	(( i=i+1 ))
493	done
494	;;
495	D\|d)
496	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
497	*)
498	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
499	IGCM_debug_Verif_Exit_Post ;;
500	esac
501	echo ${Length}
502
503	IGCM_debug_PopStack "create_ts_begin_date"
504	}
505
506	########################################################################
507	#
508	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
509	# IF SO BRING THEM IN THE WORKING DIRECTORY
510	#
511	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
512	# - especially for IDRiS
513	# - no special protection against "command too long" !
514	# - libIGCM_sys should handle that or modify following code part
515	#
516	########################################################################
517
518	if [ ${StandAlone} != true ] ; then
519	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
520	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
521	echo \${run_PostProcessing_TimeSeriesRunning}" )
522
523	#if [ X${Running_Flag} = Xy ] ; then
524	# # Time Series Job allready running
525	# IGCM_debug_Print 1 "Time Series Job allready running exit"
526	# exit
527	#fi
528	# Now run.card know we are running
529
530
531
532
533	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
534
535
536
537
538
539	# Get information from last execution
540	#CompletedFlag=$( IGCM_sys_RshMaster \
541	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
542	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
543	fi
544
545	# Is it the first submission or not ?
546
547	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
548	# First Time Series Submission
549	FIRST_PASS=TRUE
550	Length=$( create_ts_begin_date ${DateBegin} begin)
551	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
552	else
553	# Last Completed Time Series
554	FIRST_PASS=FALSE
555	DATE_FIN_JOB_B=${CompletedFlag}
556	date=${DateBegin}_${DATE_FIN_JOB_B}
557	for comp in ${config_ListOfComponents[*]} ; do
558	#
559	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
560	#
561	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
562	#
563	#set -A liste_file
564	unset liste_file
565	#
566	file_path=${R_SAVE}/${comp}/Analyse
567	#
568	i=0
569	file=${NULL_STR}
570	#
571	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
572	#
573	# Determine in which output can we find file
574	# IE : Analyse/TS or Analyse/DA2TS or ...
575	#
576	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
577	case ${FlagDir} in
578	*Y) TS_Dir=TS_YE ;;
579	*M) TS_Dir=TS_MO ;;
580	*D) TS_Dir=TS_DA ;;
581	HF) TS_Dir=TS_HF ;;
582	INS) TS_Dir=TS_INS ;;
583	esac
584	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
585	#
586	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
587	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
588	#
589	done
590	done
591	if [ X${file} != X${NULL_STR} ] ; then
592	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
593	ExitFlag=false
594	IGCM_debug_Verif_Exit_Post
595	fi
596	done
597	#
598	LEVEL=0
599	#
600	fi
601
602	########################################################################
603	#
604	# DEFINE LOOP PARAMETERS
605	#
606	########################################################################
607
608	DATE_COUNT=${DATE_FIN_JOB_B}
609	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
610
611	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
612
613	if [ ${DoJob} = true ] ; then
614	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
615	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
616	Length=$( create_ts_next_date ${DATE_COUNT} )
617	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
618	done
619	else
620	NBRE_FILE_TOT=0
621	fi
622
623	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
624	NBRE_TOUR=1
625	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
626	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
627	else
628	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
629	fi
630
631	CURRENT_LOOP=1
632	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
633	# WE HAVE 3 LEVELS
634	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
635	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
636	# - LEVEL 3 : WE ARE IN THE LAST LOOP
637
638	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
639
640	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
641	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
642	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
643
644	if [ ${FIRST_PASS} = TRUE ] ; then
645	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
646	else
647	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
648	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
649	fi
650
651	DATE_FIN=${PeriodDateEnd}
652	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
653
654	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
655	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
656
657	LEVEL=1
658	DEPOT="TRUE"
659	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
660	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
661	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
662
663	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
664	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
665	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
666	elif [ -z "${DATE_COURANTE}" ] ; then
667	DATE_COURANTE=${DATE_FIN_JOB_B}
668	else
669	Length=$( create_ts_next_date ${DATE_FIN} )
670	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
671	fi
672
673	(( TotLength = 0 ))
674	COMPTEUR=1
675	DATE_LOOP=${DATE_COURANTE}
676
677	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
678	#
679	Length=$( create_ts_next_date ${DATE_LOOP} )
680	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
681	(( TotLength = TotLength + Length ))
682	(( COMPTEUR = COMPTEUR + 1 ))
683	#
684	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
685	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
686
687	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
688	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
689	DEPOT="FALSE"
690	LEVEL=2
691	else
692	# - LEVEL 3 : WE ARE IN THE LAST LOOP
693	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
694	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
695
696	Length=$( create_ts_next_date ${DATE_FIN} )
697	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
698	DATE_FIN=${PeriodDateEnd}
699	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
700
701	LEVEL=3
702	DEPOT="TRUE"
703	FLAG_B="TRUE"
704	fi
705
706	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
707	for comp in ${config_ListOfComponents[*]} ; do
708	#
709	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
710	#
711	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
712	#
713	eval R_OUT_${comp}=${R_SAVE}/${comp}
714	#
715	i=0
716	#
717	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
718	#
719	# Determine in which output can we find file
720	# IE : Output/MO or Output/DA or ...
721	#
722	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
723	case ${FlagDir} in
724	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
725	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
726	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
727	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
728	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
729	*)
730	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
731	IGCM_sys_Exit "Job create_ts" ;;
732	esac
733
734	# Initialize array
735	#set -A liste_file
736	#set -A liste_file_tmp
737	unset liste_file
738	unset liste_file_tmp
739	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
740
741	COMPTEUR=1
742	DATE_LOOP=${DATE_COURANTE}
743
744	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
745	#
746	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
747	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
748
749	Length2=$( create_ts_next_date ${DATE_LOOP} end)
750	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
751
752	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
753	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
754	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
755	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
756	(( COMPTEUR = COMPTEUR + 1 ))
757	#
758	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
759	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
760
761	# Get selected files
762	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
763	ExitFlag=false
764	IGCM_debug_Verif_Exit_Post
765
766	# Apply IGCM_Patch if needed
767	if [ ! X${RebuildFrequency} = Xtrue ] ; then
768	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
769	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
770	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
771	IGCM_${Patch} ${file_t}
772	done
773	done
774	fi
775	fi
776
777	# WE CAN CONCATENATE FILES AT THIS POINT
778	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
779	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
780	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
781	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
782	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
783
784	liste_coord=" "
785	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
786	liste_coord=${liste_coord}${axis}","
787	done
788
789	# Time axis must be the last dimension
790	time_axis=${axis}
791
792	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
793	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
794	#
795	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
796	#
797	# # Change time axis name to be CF compliant
798	# if [ ${time_axis} != time_counter ]; then
799	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
800	# fi
801	#
802	if [ "${FLAG_B}" = "TRUE" ] ; then
803	# WE CONCATENATE WITH EXISTING FILES
804	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
805	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
806	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
807	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
808	#
809	IGCM_sys_Rm ${file1} ${file2}
810	fi
811	#
812	if [ "${DEPOT}" = "TRUE" ] ; then
813	# WE PUT FINAL FILE ON FILE SYSTEM
814	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
815	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
816	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
817	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
818	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
819	IGCM_sys_Rm ${file_out}
820	#
821	fi
822	#
823	ExitFlag=false
824	IGCM_debug_Verif_Exit_Post
825	done # for var in ${liste_var} ; do
826	IGCM_sys_Rm ${liste_file_tmp[*]}
827	done # for file in ${liste_file_brut} do
828	done # for comp in ${config_ListOfComponents[*]} ; do
829	ExitFlag=false
830	IGCM_debug_Verif_Exit_Post
831	# GO AHEAD IN DEEPEST LOOP
832	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
833	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
834	# FINISH
835
836	# DODS copy
837	for comp in ${config_ListOfComponents[*]} ; do
838	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
839	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
840	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
841	done
842	done
843
844	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
845	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
846
847	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
848	export listVarEnv
849
850	#D-
851	# --------------------------------------------------------------------
852	#D- Test if all was right
853	# --------------------------------------------------------------------
854	IGCM_debug_Verif_Exit_Post
855	# --------------------------------------------------------------------
856	#D- Submit MONITORING
857	# --------------------------------------------------------------------
858	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
859
860	#D-
861	# --------------------------------------------------------------------
862	#D- Update the run.card
863	# --------------------------------------------------------------------
864	if [ ${StandAlone} != true ] ; then
865	# Put in run.card end period of time series
866	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
867	# We have finish the job !
868	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
869	fi
870
871	# Clean RUN_DIR_PATH (necessary for cesium)
872	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
873
874	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: