Context Navigation

source: trunk/libIGCM/AA_create_ts @ 278

Last change on this file since 278 was 267, checked in by mmaipsl, 14 years ago
A bug occur on linux hosts when we pass an environment variable that contains empty string through process. Then I have changed TsTask? for "empty" string if Dimension is empty for backward compatibility and come back to the real empty string in AA_create_ts after setting default value.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 31.9 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 20:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #PBS -N TS # Nom du job
57	#-Q- sx9mercure #PBS -j o # regroupement des stdout et stderr
58	#-Q- sx9mercure #PBS -S /usr/bin/ksh # shell de soumission
59	#-Q- sx9mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
60	#-Q- sx9mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
61	#-Q- sx9mercure #PBS -q scal-new
62	#-Q- titane #!/bin/ksh
63	#-Q- titane ######################
64	#-Q- titane ## TITANE CEA ##
65	#-Q- titane ######################
66	#-Q- titane #MSUB -r TS # Nom du job
67	#-Q- titane #MSUB -N 1 # Reservation du noeud
68	#-Q- titane #MSUB -n 1 # Reservation du processus
69	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
70	#-Q- titane #MSUB -E "-j o"
71	#-Q- titane #MSUB -E "-S /bin/ksh"
72	#-Q- titane ##MSUB -e nco.out # Sortie standard
73	#-Q- titane ##MSUB -o nco.out # Sortie standard
74	#-Q- default #!/bin/ksh
75	#-Q- default ##################
76	#-Q- default ## DEFAULT HOST ##
77	#-Q- default ##################
78
79	# $Date$
80	# $Author$
81	# $Revision$
82	# IPSL (2006)
83	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
84
85	#set -eu
86	#set -vx
87
88	date
89
90	#-Q- sx8brodie export OMP_NUM_THREADS=1
91	#-Q- aix6 export OMP_NUM_THREADS=1
92
93	########################################################################
94
95	#D- Flag to determine if this job in a standalone mode
96	#D- Default : value from AA_job if any
97	StandAlone=${StandAlone:=true}
98
99	#D- Low level debug : to bypass lib test checks and stack construction
100	#D- Default : value from AA_job if any
101	libIGCM=${libIGCM:=::modipsl::/libIGCM}
102	# WARNING for StandAlone used : To run this script on some machine,
103	# you must check MirrorlibIGCM variable in sys library.
104	# If this variable is true, you must use libIGCM_POST path instead
105	# of your running libIGCM directory.
106
107	#D- Flag to determine allready produced time series. Empty if you start from the beginning
108	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
109	CompletedFlag=${CompletedFlag:=}
110
111	#D- Flag to determine task type this script will perform.
112	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
113	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
114	TsTask=${TsTask:=2D}
115	if [ X${TsTask} = Xempty ] ; then
116	TsTask=""
117	fi
118
119	#D- Flag to determine if rebuild process has been performed asynchronously
120	#D- Possible value true or false.
121	#D- If true raw files has allready been patched by rebuild job
122	#D- If false create_ts will patch the raw files
123	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
124	RebuildFrequency=${RebuildFrequency:=true}
125
126	#D- Flag to determine job's output directory
127	#D- Default : value from libIGCM_post.ksh if any
128	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
129
130	#D- Increased verbosity (1, 2, 3)
131	#D- Default : value from AA_job if any
132	Verbosity=${Verbosity:=3}
133
134	#D- Low level debug : to bypass lib test checks and stack construction
135	#D- Default : value from AA_job if any
136	DEBUG_debug=${DEBUG_debug:=false}
137
138	#D- TEMPORARY Flag to determine atmospheric resolution
139	#D- Default : value from atmospheric driver if any
140	RESOL_ATM=ALL
141
142	#D- Flag to determine surface resolution
143	#D- Default : value from surface driver if any
144	RESOL_SRF=ALL
145
146	#D- Flag to determine surface resolution
147	#D- Default : value from surface driver if any
148	RESOL_SBG=ALL
149
150	#D- TEMPORARY Flag to determine ocean resolution
151	#D- Default : value from ocean driver if any
152	RESOL_OCE=${RESOL_OCE:=ORCA2}
153
154	#D- TEMPORARY Flag to determine ice resolution
155	#D- Default : value from ice driver if any
156	RESOL_ICE=${RESOL_ICE:=ORCA2}
157
158	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
159	#D- Default : value from ice driver if any
160	RESOL_MBG=${RESOL_MBG:=ORCA2}
161
162	########################################################################
163
164	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
165	( ${DEBUG_debug} ) && IGCM_debug_Check
166	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
167	( ${DEBUG_debug} ) && IGCM_card_Check
168	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
169	( ${DEBUG_debug} ) && IGCM_date_Check
170	#-------
171	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
172
173	########################################################################
174
175	#set -vx
176
177	# Useful cleaning function
178	MENAGE ()
179	{
180	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
181	}
182
183	########################################################################
184
185	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
186	IGCM_sys_Cd ${RUN_DIR_PATH}
187
188	if [ ${StandAlone} = true ] ; then
189	CARD_DIR=${SUBMIT_DIR}
190	else
191	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
192	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
193	fi
194
195	#
196	# First of all
197	#
198	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
199	typeset option
200	for option in ${config_UserChoices[*]} ; do
201	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
202	done
203	#
204	echo
205	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
206	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
207	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
208	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
209	IGCM_debug_PrintVariables 3 config_UserChoices_SpaceName
210	IGCM_debug_PrintVariables 3 config_UserChoices_ExperimentName
211	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
212	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
213	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
214	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
215	echo
216
217	#==================================
218	#R_SAVE : Job output directory
219	if ( [ ! X${config_UserChoices_SpaceName} = X ] && [ ! X${config_UserChoices_ExperimentName} = X ] ) ; then
220	FreeName=$( echo ${config_UserChoices_JobName} \| sed 's/.*_//' )
221	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
222	R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
223	else
224	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
225	R_DODS=${config_UserChoices_TagName}/${config_UserChoices_JobName}
226	fi
227	#
228	# Determine component to take care of depending on kind of task create_ts will perform
229	#
230	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
231	#
232	# Determine period for time series. Default : value from AA_job or from launch_TS if any
233	#
234	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
235	#
236	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
237	#
238	# Determine Dimension of the produced time series : 2D or 3D
239	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
240	Dimension=2D
241	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
242	Dimension=3D
243	else
244	Dimension=""
245	fi
246
247
248	########################################################################
249	#
250	# Depending on requested time series fill following variables :
251	#
252	# - LISTE_FILE_${comp}[*]
253	# - LISTE_VARS_${file}[*]
254	# - LISTE_AXIS_${file}[*]
255	# - LISTE_PATCH_${file}[*]
256	########################################################################
257
258	#
259	# For each selected component determine which files need post-processing
260	#
261	DoJob=false
262	for comp in ${config_ListOfComponents[*]} ; do
263	#
264	ActiveComp=false
265	#
266	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
267	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
268	#
269	card=${CARD_DIR}/COMP/${compname}.card
270
271	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
272	#
273	ListFilesName=${compname}_OutputFiles_List
274	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
275	#
276	if [ X${FileName0} != X${NULL_STR} ] ; then
277	#
278	#IGCM_debug_Print 1 "Component : ${compname}"
279	#
280	# INITIALISATION
281	#
282	typeset i
283	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
284	((NbFiles=${FlagToRead}))
285	((i=${NbFiles}))
286	else
287	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
288	i=2
289	fi
290	#
291	until [ $i -gt $NbFiles ]; do
292	#
293	eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
294	#
295	if [ X${flag_post} != XNONE ] ; then
296	#
297	# For each comp and each file who needs post-processing
298	# determine which variables need time series.
299	#
300	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
301	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
302	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
303	#
304	# If TimeSeriesVars list is empty we skip
305	#
306	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
307	#
308	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
309	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
310	(( i=i+3 ))
311	continue
312	#
313	fi
314	#
315	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
316	#
317	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
318	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
319	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
320	(( i=i+3 ))
321	continue
322	fi
323	#
324	# If we are not a Chunck type task, we skip files/variables handled by chunck task
325	#
326	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
327	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
328	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
329	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
330	(( i=i+3 ))
331	continue
332	fi
333	fi
334	#
335	# Now on we know we have things to do
336	#
337
338	#
339	# We need coordinates axis to properly fill produced time series
340	#
341	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
342	#
343	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
344	#
345	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
346	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
347	(( i=i+3 ))
348	continue
349	#
350	fi
351	#
352	# We need LIST of variables not allready produced (useful for standalone mode)
353	#
354	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
355	case ${FlagDir} in
356	*Y) TS_Dir=TS_YE ;;
357	*M) TS_Dir=TS_MO ;;
358	*D) TS_Dir=TS_DA ;;
359	HF) TS_Dir=TS_HF ;;
360	3H) TS_Dir=TS_HF ;;
361	INS) TS_Dir=TS_INS ;;
362	esac
363	#
364	count=0
365	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
366	#
367	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
368	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
369	#
370	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
371	if [ ! $? = 0 ] ; then
372	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
373	ActiveComp=true
374	(( count = count + 1 ))
375	fi
376	done
377	#
378	if [ ${count} -gt 0 ] ; then
379	#
380	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
381	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
382	#
383	if [ ! X${RebuildFrequency} = Xtrue ] ; then
384	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
385	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
386	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
387	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
388	if [ Xload${Patch} != X ] ; then
389	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
390	eval load${Patch}=loaded
391	fi
392	done
393	fi
394	fi
395	fi
396	fi
397	(( i=i+3 ))
398	done
399	fi
400	#
401	if [ ${ActiveComp} = true ] ; then
402	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
403	fi
404	done
405
406	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
407	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
408
409	########################################################################
410	#
411	# DEFINE DATE FUNCTIONS
412	#
413	########################################################################
414
415	# Give the end of the next $1 period from $2 date
416	function create_ts_next_date {
417	IGCM_debug_PushStack "create_ts_next_date" $@
418	typeset Length
419	case ${config_UserChoices_PeriodLength} in
420	Y\|y)
421	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
422	IGCM_date_GetYearMonth ${1} year month
423	(( Length=0 ))
424	(( i=0 ))
425	until [ $i -ge $PeriodLengthInYears ] ; do
426	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
427	(( i=i+1 ))
428	done
429	;;
430	M\|m)
431	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
432	IGCM_date_GetYearMonth ${1} year month
433	(( year0 = year ))
434	if [ $(( month + 1 )) -lt 13 ] ; then
435	month0=$(( month + 1 ))
436	else
437	month0=$(( month + 1 - 12 ))
438	(( year = year0 + 1 ))
439	fi
440	(( Length=0 ))
441	(( i=0 ))
442	until [ $i -ge $PeriodLengthInMonths ] ; do
443	if [ $(( month0 + i )) -lt 13 ] ; then
444	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
445	else
446	(( year = year0 + 1 ))
447	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
448	fi
449	(( i=i+1 ))
450	done
451	;;
452	D\|d)
453	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
454	*)
455	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
456	IGCM_debug_Verif_Exit_Post ;;
457	esac
458	echo ${Length}
459
460	IGCM_debug_PopStack "create_ts_next_date"
461	}
462
463	# Give the end of the actual $1 period from $2 date
464	function create_ts_begin_date {
465	IGCM_debug_PushStack "create_ts_begin_date" $@
466	typeset Length
467	case ${config_UserChoices_PeriodLength} in
468	Y\|y)
469	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
470	IGCM_date_GetYearMonth ${1} year month
471	if [ X${2} = Xend ] ; then
472	(( year = year - PeriodLengthInYears + 1))
473	fi
474	(( Length=0 ))
475	(( i=0 ))
476	until [ $i -ge $PeriodLengthInYears ] ; do
477	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
478	(( i=i+1 ))
479	done
480	;;
481	M\|m)
482	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
483	IGCM_date_GetYearMonth ${1} year month
484	if [ X${2} = Xend ] ; then
485	(( month = month - PeriodLengthInMonths + 1 ))
486	fi
487	(( year0 = year ))
488	if [ $month -le 0 ] ; then
489	(( month = month + 12 ))
490	year=$( printf "%04i\n" $(( year - 1 )) )
491	fi
492	month=$( printf "%02i\n" ${month} )
493	(( Length=0 ))
494	(( i=0 ))
495	until [ $i -ge $PeriodLengthInMonths ] ; do
496	if [ $(( month + i )) -lt 13 ] ; then
497	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
498	else
499	(( year = year0 + 1 ))
500	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
501	fi
502	(( i=i+1 ))
503	done
504	;;
505	D\|d)
506	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
507	*)
508	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
509	IGCM_debug_Verif_Exit_Post ;;
510	esac
511	echo ${Length}
512
513	IGCM_debug_PopStack "create_ts_begin_date"
514	}
515
516	########################################################################
517	#
518	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
519	# IF SO BRING THEM IN THE WORKING DIRECTORY
520	#
521	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
522	# - especially for IDRiS
523	# - no special protection against "command too long" !
524	# - libIGCM_sys should handle that or modify following code part
525	#
526	########################################################################
527
528	if [ ${StandAlone} != true ] ; then
529	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
530	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
531	echo \${run_PostProcessing_TimeSeriesRunning}" )
532
533	#if [ X${Running_Flag} = Xy ] ; then
534	# # Time Series Job allready running
535	# IGCM_debug_Print 1 "Time Series Job allready running exit"
536	# exit
537	#fi
538	# Now run.card know we are running
539
540
541
542
543	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
544
545
546
547
548
549	# Get information from last execution
550	#CompletedFlag=$( IGCM_sys_RshMaster \
551	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
552	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
553	fi
554
555	# Is it the first submission or not ?
556
557	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
558	# First Time Series Submission
559	FIRST_PASS=TRUE
560	Length=$( create_ts_begin_date ${DateBegin} begin)
561	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
562	else
563	# Last Completed Time Series
564	FIRST_PASS=FALSE
565	DATE_FIN_JOB_B=${CompletedFlag}
566	date=${DateBegin}_${DATE_FIN_JOB_B}
567	for comp in ${config_ListOfComponents[*]} ; do
568	#
569	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
570	#
571	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
572	#
573	#set -A liste_file
574	unset liste_file
575	#
576	file_path=${R_SAVE}/${comp}/Analyse
577	#
578	i=0
579	file=${NULL_STR}
580	#
581	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
582	#
583	# Determine in which output can we find file
584	# IE : Analyse/TS or Analyse/DA2TS or ...
585	#
586	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
587	case ${FlagDir} in
588	*Y) TS_Dir=TS_YE ;;
589	*M) TS_Dir=TS_MO ;;
590	*D) TS_Dir=TS_DA ;;
591	HF) TS_Dir=TS_HF ;;
592	3H) TS_Dir=TS_HF ;;
593	INS) TS_Dir=TS_INS ;;
594	esac
595	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
596	#
597	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
598	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
599	#
600	done
601	done
602	if [ X${file} != X${NULL_STR} ] ; then
603	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
604	ExitFlag=false
605	IGCM_debug_Verif_Exit_Post
606	fi
607	done
608	#
609	LEVEL=0
610	#
611	fi
612
613	########################################################################
614	#
615	# DEFINE LOOP PARAMETERS
616	#
617	########################################################################
618
619	DATE_COUNT=${DATE_FIN_JOB_B}
620	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
621
622	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
623
624	if [ ${DoJob} = true ] ; then
625	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
626	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
627	Length=$( create_ts_next_date ${DATE_COUNT} )
628	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
629	done
630	else
631	NBRE_FILE_TOT=0
632	fi
633
634	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
635	NBRE_TOUR=1
636	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
637	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
638	else
639	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
640	fi
641
642	CURRENT_LOOP=1
643	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
644	# WE HAVE 3 LEVELS
645	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
646	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
647	# - LEVEL 3 : WE ARE IN THE LAST LOOP
648
649	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
650
651	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
652	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
653	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
654
655	if [ ${FIRST_PASS} = TRUE ] ; then
656	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
657	else
658	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
659	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
660	fi
661
662	DATE_FIN=${PeriodDateEnd}
663	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
664
665	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
666	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
667
668	LEVEL=1
669	DEPOT="TRUE"
670	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
671	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
672	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
673
674	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
675	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
676	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
677	elif [ -z "${DATE_COURANTE}" ] ; then
678	DATE_COURANTE=${DATE_FIN_JOB_B}
679	else
680	Length=$( create_ts_next_date ${DATE_FIN} )
681	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
682	fi
683
684	(( TotLength = 0 ))
685	COMPTEUR=1
686	DATE_LOOP=${DATE_COURANTE}
687
688	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
689	#
690	Length=$( create_ts_next_date ${DATE_LOOP} )
691	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
692	(( TotLength = TotLength + Length ))
693	(( COMPTEUR = COMPTEUR + 1 ))
694	#
695	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
696	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
697
698	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
699	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
700	DEPOT="FALSE"
701	LEVEL=2
702	else
703	# - LEVEL 3 : WE ARE IN THE LAST LOOP
704	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
705	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
706
707	Length=$( create_ts_next_date ${DATE_FIN} )
708	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
709	DATE_FIN=${PeriodDateEnd}
710	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
711
712	LEVEL=3
713	DEPOT="TRUE"
714	FLAG_B="TRUE"
715	fi
716
717	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
718	for comp in ${config_ListOfComponents[*]} ; do
719	#
720	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
721	#
722	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
723	#
724	eval R_OUT_${comp}=${R_SAVE}/${comp}
725	#
726	i=0
727	#
728	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
729	#
730	# Determine in which output can we find file
731	# IE : Output/MO or Output/DA or ...
732	#
733	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
734	case ${FlagDir} in
735	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
736	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
737	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
738	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
739	3H) TS_Dir=TS_HF ; FreqDir=HF ;;
740	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
741	*)
742	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
743	IGCM_sys_Exit "Job create_ts" ;;
744	esac
745
746	# Initialize array
747	#set -A liste_file
748	#set -A liste_file_tmp
749	unset liste_file
750	unset liste_file_tmp
751	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
752
753	COMPTEUR=1
754	DATE_LOOP=${DATE_COURANTE}
755
756	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
757	#
758	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
759	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
760
761	Length2=$( create_ts_next_date ${DATE_LOOP} end)
762	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
763
764	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
765	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
766	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
767	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
768	(( COMPTEUR = COMPTEUR + 1 ))
769	#
770	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
771	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
772
773	# Get selected files
774	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
775	ExitFlag=false
776	IGCM_debug_Verif_Exit_Post
777
778	# Apply IGCM_Patch if needed
779	if [ ! X${RebuildFrequency} = Xtrue ] ; then
780	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
781	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
782	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
783	IGCM_${Patch} ${file_t}
784	done
785	done
786	fi
787	fi
788
789	# WE CAN CONCATENATE FILES AT THIS POINT
790	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
791	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
792	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
793	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
794	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
795
796	liste_coord=" "
797	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
798	liste_coord=${liste_coord}${axis}","
799	done
800
801	# Time axis must be the last dimension
802	time_axis=${axis}
803
804	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
805	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
806	#
807	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
808	#
809	# # Change time axis name to be CF compliant
810	# if [ ${time_axis} != time_counter ]; then
811	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
812	# fi
813	#
814	if [ "${FLAG_B}" = "TRUE" ] ; then
815	# WE CONCATENATE WITH EXISTING FILES
816	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
817	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
818	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
819	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
820	#
821	IGCM_sys_Rm ${file1} ${file2}
822	fi
823	#
824	if [ "${DEPOT}" = "TRUE" ] ; then
825	# WE PUT FINAL FILE ON FILE SYSTEM
826	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
827	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
828	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
829	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
830	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
831	IGCM_sys_Rm ${file_out}
832	#
833	fi
834	#
835	ExitFlag=false
836	IGCM_debug_Verif_Exit_Post
837	done # for var in ${liste_var} ; do
838	IGCM_sys_Rm ${liste_file_tmp[*]}
839	done # for file in ${liste_file_brut} do
840	done # for comp in ${config_ListOfComponents[*]} ; do
841	ExitFlag=false
842	IGCM_debug_Verif_Exit_Post
843	# GO AHEAD IN DEEPEST LOOP
844	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
845	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
846	# FINISH
847
848	# DODS copy
849	for comp in ${config_ListOfComponents[*]} ; do
850	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
851	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
852	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
853	done
854	done
855
856	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
857	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
858
859	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
860	export listVarEnv
861
862	#D-
863	# --------------------------------------------------------------------
864	#D- Test if all was right
865	# --------------------------------------------------------------------
866	IGCM_debug_Verif_Exit_Post
867	# --------------------------------------------------------------------
868	#D- Submit MONITORING
869	# --------------------------------------------------------------------
870	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
871
872	#D-
873	# --------------------------------------------------------------------
874	#D- Update the run.card
875	# --------------------------------------------------------------------
876	if [ ${StandAlone} != true ] ; then
877	# Put in run.card end period of time series
878	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
879	# We have finish the job !
880	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
881	fi
882
883	# Clean RUN_DIR_PATH (necessary for cesium)
884	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
885
886	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: