Context Navigation

source: tags/libIGCM_v1_4/AA_create_ts

Last change on this file was 189, checked in by sdipsl, 15 years ago
Bugfix of the bugfix (badly placed bracket) Ask more time to the scheduler to anticipate CMIP5 time series
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 29.3 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ##################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ##################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #PBS -N TS # Nom du job
57	#-Q- sx9mercure #PBS -j o # regroupement des stdout et stderr
58	#-Q- sx9mercure #PBS -S /usr/bin/ksh # shell de soumission
59	#-Q- sx9mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
60	#-Q- sx9mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
61	#-Q- sx9mercure #PBS -q scalaire
62	#-Q- default #!/bin/ksh
63	#-Q- default ##################
64	#-Q- default ## DEFAULT HOST ##
65	#-Q- default ##################
66
67	# $Date$
68	# $Author$
69	# $Revision$
70	# IPSL (2006)
71	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
72
73	#set -eu
74	#set -vx
75
76	date
77
78	#-Q- sx8brodie export OMP_NUM_THREADS=1
79	#-Q- aix6 export OMP_NUM_THREADS=1
80
81	########################################################################
82
83	#D- Flag to determine if this job in a standalone mode
84	#D- Default : value from AA_job if any
85	StandAlone=${StandAlone:=true}
86
87	#D- Flag to determine allready produced time series.
88	#D- Default : Commented. Uncomment for StandAlone case if you want to continue a time series
89	#CompletedFlag=
90
91	#D- Flag to determine job's output directory
92	#D- Default : value from libIGCM_post.ksh if any
93	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
94
95	#D- Increased verbosity (1, 2, 3)
96	#D- Default : value from AA_job if any
97	Verbosity=${Verbosity:=3}
98
99	#D- Low level debug : to bypass lib test checks and stack construction
100	#D- Default : value from AA_job if any
101	DEBUG_debug=${DEBUG_debug:=false}
102
103	#D- Low level debug : to bypass lib test checks and stack construction
104	#D- Default : value from AA_job if any
105	libIGCM=${libIGCM:=/path/to/your/libIGCM}
106
107	#D- TEMPORARY Flag to determine atmospheric resolution
108	#D- Default : value from atmospheric driver if any
109	RESOL_ATM=ALL
110
111	#D- TEMPORARY Flag to determine ocean resolution
112	#D- Default : value from ocean driver if any
113	RESOL_OCE=${RESOL_OCE:=ORCA2}
114
115	#D- TEMPORARY Flag to determine ice resolution
116	#D- Default : value from ice driver if any
117	RESOL_ICE=${RESOL_ICE:=ORCA2}
118
119	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
120	#D- Default : value from ice driver if any
121	RESOL_MBG=${RESOL_MBG:=ORCA2}
122
123	#D- Flag to determine surface resolution
124	#D- Default : value from surface driver if any
125	RESOL_SRF=ALL
126
127	#D- Flag to determine surface resolution
128	#D- Default : value from surface driver if any
129	RESOL_BIOS=ALL
130
131	########################################################################
132
133	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
134	( ${DEBUG_debug} ) && IGCM_debug_Check
135	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
136	( ${DEBUG_debug} ) && IGCM_card_Check
137	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
138	( ${DEBUG_debug} ) && IGCM_date_Check
139	#-------
140	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
141
142	########################################################################
143
144	#set -vx
145
146	# Useful cleaning function
147	MENAGE ()
148	{
149	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
150	}
151
152	########################################################################
153
154	if [ ${DEBUG_debug} = true ] ; then
155	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
156	echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
157	fi
158	IGCM_sys_Cd ${RUN_DIR_PATH}
159
160	if [ ${StandAlone} = true ] ; then
161	CARD_DIR=${SUBMIT_DIR}
162	else
163	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
164	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
165	fi
166
167	#
168	# First of all
169	#
170	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
171	typeset option
172	for option in ${config_UserChoices[*]} ; do
173	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
174	done
175	#
176	echo
177	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
178	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
179	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
180	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
181	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
182	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
183	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
184	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
185	echo
186
187	#==================================
188	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
189	#
190	# Determine component to take care of depending on kind of task create_ts will perform
191	#
192	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
193	#
194	# Determine period for time series. Default : value from AA_job or from launch_TS if any
195	#
196	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
197	#
198	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
199	#
200	# Determine Dimension of the produced time series : 2D or 3D
201	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
202	Dimension=2D
203	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
204	Dimension=3D
205	else
206	Dimension=""
207	fi
208
209
210	########################################################################
211	#
212	# Depending on requested time series fill following variables :
213	#
214	# - LISTE_FILE_${comp}[*]
215	# - LISTE_VARS_${file}[*]
216	# - LISTE_AXIS_${file}[*]
217	# - LISTE_PATCH_${file}[*]
218	########################################################################
219
220	#
221	# For each selected component determine which files need post-processing
222	#
223	DoJob=false
224	for comp in ${config_ListOfComponents[*]} ; do
225	#
226	ActiveComp=false
227	#
228	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
229	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
230	#
231	card=${CARD_DIR}/COMP/${compname}.card
232
233	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
234	#
235	ListFilesName=${compname}_OutputFiles_List
236	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
237	#
238	if [ X${FileName0} != X${NULL_STR} ] ; then
239	#
240	#IGCM_debug_Print 1 "Component : ${compname}"
241	#
242	# INITIALISATION
243	#
244	typeset i
245	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
246	((NbFiles=${FlagToRead}))
247	((i=${NbFiles}-2))
248	else
249	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
250	i=0
251	fi
252	#
253	until [ $i -ge $NbFiles ]; do
254	#
255	(( i_ = i+2 ))
256	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
257	eval flag_post=${flag_post_}
258	#
259	if [ X${flag_post} != XNONE ] ; then
260	#
261	# For each comp and each file who needs post-processing
262	# determine which variables need time series.
263	#
264	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
265	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
266	#
267	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
268	#
269	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
270	(( i=i+3 ))
271	continue
272	#
273	fi
274	#
275	# If we are not a Chunck type task, skip files/variables handled by chunck task
276	#
277	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
278	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
279	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
280	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
281	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
282	(( i=i+3 ))
283	continue
284	fi
285	fi
286	#
287	# We need coordinates axis to properly fill produced time series
288	#
289	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
290	#
291	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
292	#
293	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
294	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
295	(( i=i+3 ))
296	continue
297	#
298	fi
299	#
300	# We need LIST of variables not allready produced (useful for standalone mode)
301	#
302	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
303	case ${FlagDir} in
304	*Y) TS_Dir=TS_YE ;;
305	*M) TS_Dir=TS_MO ;;
306	*D) TS_Dir=TS_DA ;;
307	HF) TS_Dir=TS_HF ;;
308	INS) TS_Dir=TS_INS ;;
309	esac
310	#
311	count=0
312	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
313	#
314	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
315	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
316	#
317	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
318	if [ ! $? = 0 ] ; then
319	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
320	ActiveComp=true
321	(( count = count + 1 ))
322	fi
323	done
324	#
325	if [ ${count} -gt 0 ] ; then
326	#
327	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
328	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
329	#
330	if [ ! X${RebuildFrequency} = Xtrue ] ; then
331	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
332	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
333	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
334	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
335	if [ Xload${Patch} != X ] ; then
336	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
337	eval load${Patch}=loaded
338	fi
339	done
340	fi
341	fi
342	fi
343	fi
344	(( i=i+3 ))
345	done
346	fi
347	#
348	if [ ${ActiveComp} = true ] ; then
349	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
350	fi
351	done
352
353	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
354	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
355
356	########################################################################
357	#
358	# DEFINE DATE FUNCTIONS
359	#
360	########################################################################
361
362	# Give the end of the next $1 period from $2 date
363	function create_ts_next_date {
364	IGCM_debug_PushStack "create_ts_next_date" $@
365	typeset Length
366	case ${config_UserChoices_PeriodLength} in
367	Y\|y)
368	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
369	IGCM_date_GetYearMonth ${1} year month
370	(( Length=0 ))
371	(( i=0 ))
372	until [ $i -ge $PeriodLengthInYears ] ; do
373	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
374	(( i=i+1 ))
375	done
376	;;
377	M\|m)
378	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
379	IGCM_date_GetYearMonth ${1} year month
380	(( year0 = year ))
381	if [ $(( month + 1 )) -lt 13 ] ; then
382	month0=$(( month + 1 ))
383	else
384	month0=$(( month + 1 - 12 ))
385	(( year = year0 + 1 ))
386	fi
387	(( Length=0 ))
388	(( i=0 ))
389	until [ $i -ge $PeriodLengthInMonths ] ; do
390	if [ $(( month0 + i )) -lt 13 ] ; then
391	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
392	else
393	(( year = year0 + 1 ))
394	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
395	fi
396	(( i=i+1 ))
397	done
398	;;
399	D\|d)
400	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
401	*)
402	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
403	IGCM_debug_Verif_Exit_Post ;;
404	esac
405	echo ${Length}
406
407	IGCM_debug_PopStack "create_ts_next_date"
408	}
409
410	# Give the end of the actual $1 period from $2 date
411	function create_ts_begin_date {
412	IGCM_debug_PushStack "create_ts_begin_date" $@
413	typeset Length
414	case ${config_UserChoices_PeriodLength} in
415	Y\|y)
416	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
417	IGCM_date_GetYearMonth ${1} year month
418	if [ X${2} = Xend ] ; then
419	(( year = year - PeriodLengthInYears + 1))
420	fi
421	(( Length=0 ))
422	(( i=0 ))
423	until [ $i -ge $PeriodLengthInYears ] ; do
424	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
425	(( i=i+1 ))
426	done
427	;;
428	M\|m)
429	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
430	IGCM_date_GetYearMonth ${1} year month
431	if [ X${2} = Xend ] ; then
432	(( month = month - PeriodLengthInMonths + 1 ))
433	fi
434	(( year0 = year ))
435	if [ $month -le 0 ] ; then
436	(( month = month + 12 ))
437	year=$( printf "%04i\n" $(( year - 1 )) )
438	fi
439	month=$( printf "%02i\n" ${month} )
440	(( Length=0 ))
441	(( i=0 ))
442	until [ $i -ge $PeriodLengthInMonths ] ; do
443	if [ $(( month + i )) -lt 13 ] ; then
444	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
445	else
446	(( year = year0 + 1 ))
447	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
448	fi
449	(( i=i+1 ))
450	done
451	;;
452	D\|d)
453	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
454	*)
455	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
456	IGCM_debug_Verif_Exit_Post ;;
457	esac
458	echo ${Length}
459
460	IGCM_debug_PopStack "create_ts_begin_date"
461	}
462
463	########################################################################
464	#
465	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
466	# IF SO BRING THEM IN THE WORKING DIRECTORY
467	#
468	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
469	# - especially for IDRiS
470	# - no special protection against "command too long" !
471	# - libIGCM_sys should handle that or modify following code part
472	#
473	########################################################################
474
475	if [ ${StandAlone} != true ] ; then
476	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
477	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
478	echo \${run_PostProcessing_TimeSeriesRunning}" )
479
480	#if [ X${Running_Flag} = Xy ] ; then
481	# # Time Series Job allready running
482	# IGCM_debug_Print 1 "Time Series Job allready running exit"
483	# exit
484	#fi
485	# Now run.card know we are running
486
487
488
489
490	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
491
492
493
494
495
496	# Get information from last execution
497	#CompletedFlag=$( IGCM_sys_RshMaster \
498	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
499	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
500	fi
501
502	# Is it the first submission or not ?
503
504	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
505	# First Time Series Submission
506	FIRST_PASS=TRUE
507	Length=$( create_ts_begin_date ${DateBegin} begin)
508	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
509	else
510	# Last Completed Time Series
511	FIRST_PASS=FALSE
512	DATE_FIN_JOB_B=${CompletedFlag}
513	date=${DateBegin}_${DATE_FIN_JOB_B}
514	for comp in ${config_ListOfComponents[*]} ; do
515	#
516	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
517	#
518	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
519	#
520	#set -A liste_file
521	unset liste_file
522	#
523	file_path=${R_SAVE}/${comp}/Analyse
524	#
525	i=0
526	file=${NULL_STR}
527	#
528	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
529	#
530	# Determine in which output can we find file
531	# IE : Analyse/TS or Analyse/DA2TS or ...
532	#
533	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
534	case ${FlagDir} in
535	*Y) TS_Dir=TS_YE ;;
536	*M) TS_Dir=TS_MO ;;
537	*D) TS_Dir=TS_DA ;;
538	HF) TS_Dir=TS_HF ;;
539	INS) TS_Dir=TS_INS ;;
540	esac
541	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
542	#
543	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
544	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
545	#
546	done
547	done
548	if [ X${file} != X${NULL_STR} ] ; then
549	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
550	ExitFlag=false
551	IGCM_debug_Verif_Exit_Post
552	fi
553	done
554	#
555	LEVEL=0
556	#
557	fi
558
559	########################################################################
560	#
561	# DEFINE LOOP PARAMETERS
562	#
563	########################################################################
564
565	DATE_COUNT=${DATE_FIN_JOB_B}
566	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
567
568	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
569
570	if [ ${DoJob} = true ] ; then
571	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
572	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
573	Length=$( create_ts_next_date ${DATE_COUNT} )
574	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
575	done
576	else
577	NBRE_FILE_TOT=0
578	fi
579
580	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
581	NBRE_TOUR=1
582	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
583	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
584	else
585	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
586	fi
587
588	CURRENT_LOOP=1
589	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
590	# WE HAVE 3 LEVELS
591	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
592	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
593	# - LEVEL 3 : WE ARE IN THE LAST LOOP
594
595	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
596
597	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
598	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
599	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
600
601	if [ ${FIRST_PASS} = TRUE ] ; then
602	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
603	else
604	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
605	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
606	fi
607
608	DATE_FIN=${PeriodDateEnd}
609	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
610
611	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
612	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
613
614	LEVEL=1
615	DEPOT="TRUE"
616	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
617	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
618	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
619
620	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
621	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
622	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
623	elif [ -z "${DATE_COURANTE}" ] ; then
624	DATE_COURANTE=${DATE_FIN_JOB_B}
625	else
626	Length=$( create_ts_next_date ${DATE_FIN} )
627	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
628	fi
629
630	(( TotLength = 0 ))
631	COMPTEUR=1
632	DATE_LOOP=${DATE_COURANTE}
633
634	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
635	#
636	Length=$( create_ts_next_date ${DATE_LOOP} )
637	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
638	(( TotLength = TotLength + Length ))
639	(( COMPTEUR = COMPTEUR + 1 ))
640	#
641	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
642	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
643
644	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
645	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
646	DEPOT="FALSE"
647	LEVEL=2
648	else
649	# - LEVEL 3 : WE ARE IN THE LAST LOOP
650	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
651	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
652
653	Length=$( create_ts_next_date ${DATE_FIN} )
654	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
655	DATE_FIN=${PeriodDateEnd}
656	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
657
658	LEVEL=3
659	DEPOT="TRUE"
660	FLAG_B="TRUE"
661	fi
662
663	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
664	for comp in ${config_ListOfComponents[*]} ; do
665	#
666	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
667	#
668	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
669	#
670	eval R_OUT_${comp}=${R_SAVE}/${comp}
671	#
672	i=0
673	#
674	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
675	#
676	# Determine in which output can we find file
677	# IE : Output/MO or Output/DA or ...
678	#
679	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
680	case ${FlagDir} in
681	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
682	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
683	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
684	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
685	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
686	*)
687	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
688	IGCM_sys_Exit "Job create_ts" ;;
689	esac
690
691	# Initialize array
692	#set -A liste_file
693	#set -A liste_file_tmp
694	unset liste_file
695	unset liste_file_tmp
696	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
697
698	COMPTEUR=1
699	DATE_LOOP=${DATE_COURANTE}
700
701	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
702	#
703	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
704	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
705
706	Length2=$( create_ts_next_date ${DATE_LOOP} end)
707	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
708
709	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
710	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
711	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
712	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
713	(( COMPTEUR = COMPTEUR + 1 ))
714	#
715	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
716	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
717
718	# Get selected files
719	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
720	ExitFlag=false
721	IGCM_debug_Verif_Exit_Post
722
723	# Apply IGCM_Patch if needed
724	if [ ! X${RebuildFrequency} = Xtrue ] ; then
725	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
726	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
727	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
728	IGCM_${Patch} ${file_t}
729	done
730	done
731	fi
732	fi
733
734	# WE CAN CONCATENATE FILES AT THIS POINT
735	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
736	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
737	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
738	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
739	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
740
741	liste_coord=" "
742	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
743	liste_coord=${liste_coord}${axis}","
744	done
745
746	# Time axis must be the last dimension
747	time_axis=${axis}
748
749	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
750	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
751	#
752	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
753	#
754	# # Change time axis name to be CF compliant
755	# if [ ${time_axis} != time_counter ]; then
756	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
757	# fi
758	#
759	if [ "${FLAG_B}" = "TRUE" ] ; then
760	# WE CONCATENATE WITH EXISTING FILES
761	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
762	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
763	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
764	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
765	#
766	IGCM_sys_Rm ${file1} ${file2}
767	fi
768	#
769	if [ "${DEPOT}" = "TRUE" ] ; then
770	# WE PUT FINAL FILE ON FILE SYSTEM
771	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
772	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
773	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
774	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
775	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
776	IGCM_sys_Rm ${file_out}
777	#
778	fi
779	#
780	ExitFlag=false
781	IGCM_debug_Verif_Exit_Post
782	done # for var in ${liste_var} ; do
783	IGCM_sys_Rm ${liste_file_tmp[*]}
784	done # for file in ${liste_file_brut} do
785	done # for comp in ${config_ListOfComponents[*]} ; do
786	ExitFlag=false
787	IGCM_debug_Verif_Exit_Post
788	# GO AHEAD IN DEEPEST LOOP
789	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
790	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
791	# FINISH
792
793	# DODS copy
794	for comp in ${config_ListOfComponents[*]} ; do
795	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
796	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
797	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
798	done
799	done
800
801	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
802	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_BIOS
803
804	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_BIOS"
805	export listVarEnv
806
807	#D-
808	# --------------------------------------------------------------------
809	#D- Test if all was right
810	# --------------------------------------------------------------------
811	IGCM_debug_Verif_Exit_Post
812	# --------------------------------------------------------------------
813	#D- Submit MONITORING
814	# --------------------------------------------------------------------
815	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
816
817	#D-
818	# --------------------------------------------------------------------
819	#D- Update the run.card
820	# --------------------------------------------------------------------
821	if [ ${StandAlone} != true ] ; then
822	# Put in run.card end period of time series
823	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
824	# We have finish the job !
825	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
826	fi
827	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: