Context Navigation

source: trunk/libIGCM/AA_create_ts @ 175

Last change on this file since 175 was 175, checked in by mmaipsl, 15 years ago
Add RESOL_BIOS (BIOgeochemistry for Surface) to trunk branch for use with STOMATE component.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 29.1 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ##################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ##################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=2:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #PBS -N TS # Nom du job
57	#-Q- sx9mercure #PBS -j o # regroupement des stdout et stderr
58	#-Q- sx9mercure #PBS -S /usr/bin/ksh # shell de soumission
59	#-Q- sx9mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
60	#-Q- sx9mercure #PBS -l cputim_job=2:00:00 # Limite temps a 2 heures
61	#-Q- sx9mercure #PBS -q scalaire
62	#-Q- default #!/bin/ksh
63	#-Q- default ##################
64	#-Q- default ## DEFAULT HOST ##
65	#-Q- default ##################
66
67	# $Date$
68	# $Author$
69	# $Revision$
70	# IPSL (2006)
71	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
72
73	#set -eu
74	#set -vx
75
76	date
77
78	#-Q- sx8brodie export OMP_NUM_THREADS=1
79	#-Q- aix6 export OMP_NUM_THREADS=1
80
81	########################################################################
82
83	#D- Flag to determine if this job in a standalone mode
84	#D- Default : value from AA_job if any
85	StandAlone=${StandAlone:=true}
86
87	#D- Flag to determine allready produced time series.
88	#D- Default : Commented. Uncomment for StandAlone case if you want to continue a time series
89	#CompletedFlag=
90
91	#D- Flag to determine job's output directory
92	#D- Default : value from libIGCM_post.ksh if any
93	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
94
95	#D- Increased verbosity (1, 2, 3)
96	#D- Default : value from AA_job if any
97	Verbosity=${Verbosity:=3}
98
99	#D- Low level debug : to bypass lib test checks and stack construction
100	#D- Default : value from AA_job if any
101	DEBUG_debug=${DEBUG_debug:=false}
102
103	#D- Low level debug : to bypass lib test checks and stack construction
104	#D- Default : value from AA_job if any
105	libIGCM=${libIGCM:=/path/to/your/libIGCM}
106
107	#D- TEMPORARY Flag to determine atmospheric resolution
108	#D- Default : value from atmospheric driver if any
109	RESOL_ATM=ALL
110
111	#D- TEMPORARY Flag to determine ocean resolution
112	#D- Default : value from ocean driver if any
113	RESOL_OCE=${RESOL_OCE:=ORCA2}
114
115	#D- TEMPORARY Flag to determine ice resolution
116	#D- Default : value from ice driver if any
117	RESOL_ICE=${RESOL_ICE:=ORCA2}
118
119	#D- Flag to determine surface resolution
120	#D- Default : value from surface driver if any
121	RESOL_SRF=ALL
122
123	#D- Flag to determine surface resolution
124	#D- Default : value from surface driver if any
125	RESOL_BIOS=ALL
126
127	########################################################################
128
129	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
130	( ${DEBUG_debug} ) && IGCM_debug_Check
131	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
132	( ${DEBUG_debug} ) && IGCM_card_Check
133	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
134	( ${DEBUG_debug} ) && IGCM_date_Check
135	#-------
136	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
137
138	########################################################################
139
140	#set -vx
141
142	# Useful cleaning function
143	MENAGE ()
144	{
145	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
146	}
147
148	########################################################################
149
150	if [ ${DEBUG_debug} = true ] ; then
151	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
152	echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
153	fi
154	IGCM_sys_Cd ${RUN_DIR_PATH}
155
156	if [ ${StandAlone} = true ] ; then
157	CARD_DIR=${SUBMIT_DIR}
158	else
159	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
160	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
161	fi
162
163	#
164	# First of all
165	#
166	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
167	typeset option
168	for option in ${config_UserChoices[*]} ; do
169	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
170	done
171	#
172	echo
173	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
174	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
175	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
176	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
177	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
178	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
179	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
180	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
181	echo
182
183	#==================================
184	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
185	#
186	# Determine component to take care of depending on kind of task create_ts will perform
187	#
188	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
189	#
190	# Determine period for time series. Default : value from AA_job or from launch_TS if any
191	#
192	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
193	#
194	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
195	#
196	# Determine Dimension of the produced time series : 2D or 3D
197	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
198	Dimension=2D
199	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
200	Dimension=3D
201	else
202	Dimension=""
203	fi
204
205
206	########################################################################
207	#
208	# Depending on requested time series fill following variables :
209	#
210	# - LISTE_FILE_${comp}[*]
211	# - LISTE_VARS_${file}[*]
212	# - LISTE_AXIS_${file}[*]
213	# - LISTE_PATCH_${file}[*]
214	########################################################################
215
216	#
217	# For each selected component determine which files need post-processing
218	#
219	DoJob=false
220	for comp in ${config_ListOfComponents[*]} ; do
221	#
222	ActiveComp=false
223	#
224	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
225	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
226	#
227	card=${CARD_DIR}/COMP/${compname}.card
228
229	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
230	#
231	ListFilesName=${compname}_OutputFiles_List
232	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
233	#
234	if [ X${FileName0} != X${NULL_STR} ] ; then
235	#
236	#IGCM_debug_Print 1 "Component : ${compname}"
237	#
238	# INITIALISATION
239	#
240	typeset i
241	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
242	((NbFiles=${FlagToRead}))
243	((i=${NbFiles}-2))
244	else
245	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
246	i=0
247	fi
248	#
249	until [ $i -ge $NbFiles ]; do
250	#
251	(( i_ = i+2 ))
252	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
253	eval flag_post=${flag_post_}
254	#
255	if [ X${flag_post} != XNONE ] ; then
256	#
257	# For each comp and each file who needs post-processing
258	# determine which variables need time series.
259	#
260	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
261	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
262	#
263	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
264	#
265	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
266	(( i=i+3 ))
267	continue
268	#
269	fi
270	#
271	# If we are not a Chunck type task, skip files/variables handled by chunck task
272	#
273	if [ ! X${TsTask} = XChunck${Dimension} ] ; then
274	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
275	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
276	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
277	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
278	(( i=i+3 ))
279	continue
280	fi
281	fi
282	#
283	# We need coordinates axis to properly fill produced time series
284	#
285	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
286	#
287	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
288	#
289	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
290	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
291	(( i=i+3 ))
292	continue
293	#
294	fi
295	#
296	# We need LIST of variables not allready produced (useful for standalone mode)
297	#
298	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
299	case ${FlagDir} in
300	*Y) TS_Dir=TS_YE ;;
301	*M) TS_Dir=TS_MO ;;
302	*D) TS_Dir=TS_DA ;;
303	HF) TS_Dir=TS_HF ;;
304	INS) TS_Dir=TS_INS ;;
305	esac
306	#
307	count=0
308	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
309	#
310	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
311	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
312	#
313	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
314	if [ ! $? = 0 ] ; then
315	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
316	ActiveComp=true
317	(( count = count + 1 ))
318	fi
319	done
320	#
321	if [ ${count} -gt 0 ] ; then
322	#
323	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
324	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
325	#
326	if [ ! X${RebuildFrequency} = Xtrue ] ; then
327	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
328	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
329	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
330	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
331	if [ Xload${Patch} != X ] ; then
332	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
333	eval load${Patch}=loaded
334	fi
335	done
336	fi
337	fi
338	fi
339	fi
340	(( i=i+3 ))
341	done
342	fi
343	#
344	if [ ${ActiveComp} = true ] ; then
345	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
346	fi
347	done
348
349	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
350	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
351
352	########################################################################
353	#
354	# DEFINE DATE FUNCTIONS
355	#
356	########################################################################
357
358	# Give the end of the next $1 period from $2 date
359	function create_ts_next_date {
360	IGCM_debug_PushStack "create_ts_next_date" $@
361	typeset Length
362	case ${config_UserChoices_PeriodLength} in
363	Y\|y)
364	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
365	IGCM_date_GetYearMonth ${1} year month
366	(( Length=0 ))
367	(( i=0 ))
368	until [ $i -ge $PeriodLengthInYears ] ; do
369	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
370	(( i=i+1 ))
371	done
372	;;
373	M\|m)
374	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
375	IGCM_date_GetYearMonth ${1} year month
376	(( year0 = year ))
377	if [ $(( month + 1 )) -lt 13 ] ; then
378	month0=$(( month + 1 ))
379	else
380	month0=$(( month + 1 - 12 ))
381	(( year = year0 + 1 ))
382	fi
383	(( Length=0 ))
384	(( i=0 ))
385	until [ $i -ge $PeriodLengthInMonths ] ; do
386	if [ $(( month0 + i )) -lt 13 ] ; then
387	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
388	else
389	(( year = year0 + 1 ))
390	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
391	fi
392	(( i=i+1 ))
393	done
394	;;
395	D\|d)
396	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
397	*)
398	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
399	IGCM_debug_Verif_Exit_Post ;;
400	esac
401	echo ${Length}
402
403	IGCM_debug_PopStack "create_ts_next_date"
404	}
405
406	# Give the end of the actual $1 period from $2 date
407	function create_ts_begin_date {
408	IGCM_debug_PushStack "create_ts_begin_date" $@
409	typeset Length
410	case ${config_UserChoices_PeriodLength} in
411	Y\|y)
412	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
413	IGCM_date_GetYearMonth ${1} year month
414	if [ X${2} = Xend ] ; then
415	(( year = year - PeriodLengthInYears + 1))
416	fi
417	(( Length=0 ))
418	(( i=0 ))
419	until [ $i -ge $PeriodLengthInYears ] ; do
420	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
421	(( i=i+1 ))
422	done
423	;;
424	M\|m)
425	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
426	IGCM_date_GetYearMonth ${1} year month
427	if [ X${2} = Xend ] ; then
428	(( month = month - PeriodLengthInMonths + 1 ))
429	fi
430	(( year0 = year ))
431	if [ $month -le 0 ] ; then
432	(( month = month + 12 ))
433	year=$( printf "%04i\n" $(( year - 1 )) )
434	fi
435	month=$( printf "%02i\n" ${month} )
436	(( Length=0 ))
437	(( i=0 ))
438	until [ $i -ge $PeriodLengthInMonths ] ; do
439	if [ $(( month + i )) -lt 13 ] ; then
440	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
441	else
442	(( year = year0 + 1 ))
443	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
444	fi
445	(( i=i+1 ))
446	done
447	;;
448	D\|d)
449	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
450	*)
451	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
452	IGCM_debug_Verif_Exit_Post ;;
453	esac
454	echo ${Length}
455
456	IGCM_debug_PopStack "create_ts_begin_date"
457	}
458
459	########################################################################
460	#
461	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
462	# IF SO BRING THEM IN THE WORKING DIRECTORY
463	#
464	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
465	# - especially for IDRiS
466	# - no special protection against "command too long" !
467	# - libIGCM_sys should handle that or modify following code part
468	#
469	########################################################################
470
471	if [ ${StandAlone} != true ] ; then
472	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
473	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
474	echo \${run_PostProcessing_TimeSeriesRunning}" )
475
476	#if [ X${Running_Flag} = Xy ] ; then
477	# # Time Series Job allready running
478	# IGCM_debug_Print 1 "Time Series Job allready running exit"
479	# exit
480	#fi
481	# Now run.card know we are running
482
483
484
485
486	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
487
488
489
490
491
492	# Get information from last execution
493	#CompletedFlag=$( IGCM_sys_RshMaster \
494	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
495	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
496	fi
497
498	# Is it the first submission or not ?
499
500	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
501	# First Time Series Submission
502	FIRST_PASS=TRUE
503	Length=$( create_ts_begin_date ${DateBegin} begin)
504	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
505	else
506	# Last Completed Time Series
507	FIRST_PASS=FALSE
508	DATE_FIN_JOB_B=${CompletedFlag}
509	date=${DateBegin}_${DATE_FIN_JOB_B}
510	for comp in ${config_ListOfComponents[*]} ; do
511	#
512	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
513	#
514	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
515	#
516	#set -A liste_file
517	unset liste_file
518	#
519	file_path=${R_SAVE}/${comp}/Analyse
520	#
521	i=0
522	file=${NULL_STR}
523	#
524	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
525	#
526	# Determine in which output can we find file
527	# IE : Analyse/TS or Analyse/DA2TS or ...
528	#
529	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
530	case ${FlagDir} in
531	*Y) TS_Dir=TS_YE ;;
532	*M) TS_Dir=TS_MO ;;
533	*D) TS_Dir=TS_DA ;;
534	HF) TS_Dir=TS_HF ;;
535	INS) TS_Dir=TS_INS ;;
536	esac
537	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
538	#
539	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
540	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
541	#
542	done
543	done
544	if [ X${file} != X${NULL_STR} ] ; then
545	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
546	ExitFlag=false
547	IGCM_debug_Verif_Exit_Post
548	fi
549	done
550	#
551	LEVEL=0
552	#
553	fi
554
555	########################################################################
556	#
557	# DEFINE LOOP PARAMETERS
558	#
559	########################################################################
560
561	DATE_COUNT=${DATE_FIN_JOB_B}
562	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
563
564	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
565
566	if [ ${DoJob} = true ] ; then
567	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
568	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
569	Length=$( create_ts_next_date ${DATE_COUNT} )
570	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
571	done
572	else
573	NBRE_FILE_TOT=0
574	fi
575
576	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
577	NBRE_TOUR=1
578	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
579	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
580	else
581	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
582	fi
583
584	CURRENT_LOOP=1
585	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
586	# WE HAVE 3 LEVELS
587	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
588	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
589	# - LEVEL 3 : WE ARE IN THE LAST LOOP
590
591	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
592
593	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
594	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
595	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
596
597	if [ ${FIRST_PASS} = TRUE ] ; then
598	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
599	else
600	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
601	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
602	fi
603
604	DATE_FIN=${PeriodDateEnd}
605	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
606
607	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
608	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
609
610	LEVEL=1
611	DEPOT="TRUE"
612	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
613	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
614	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
615
616	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
617	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
618	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
619	elif [ -z "${DATE_COURANTE}" ] ; then
620	DATE_COURANTE=${DATE_FIN_JOB_B}
621	else
622	Length=$( create_ts_next_date ${DATE_FIN} )
623	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
624	fi
625
626	(( TotLength = 0 ))
627	COMPTEUR=1
628	DATE_LOOP=${DATE_COURANTE}
629
630	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
631	#
632	Length=$( create_ts_next_date ${DATE_LOOP} )
633	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
634	(( TotLength = TotLength + Length ))
635	(( COMPTEUR = COMPTEUR + 1 ))
636	#
637	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
638	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
639
640	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
641	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
642	DEPOT="FALSE"
643	LEVEL=2
644	else
645	# - LEVEL 3 : WE ARE IN THE LAST LOOP
646	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
647	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
648
649	Length=$( create_ts_next_date ${DATE_FIN} )
650	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
651	DATE_FIN=${PeriodDateEnd}
652	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
653
654	LEVEL=3
655	DEPOT="TRUE"
656	FLAG_B="TRUE"
657	fi
658
659	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
660	for comp in ${config_ListOfComponents[*]} ; do
661	#
662	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
663	#
664	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
665	#
666	eval R_OUT_${comp}=${R_SAVE}/${comp}
667	#
668	i=0
669	#
670	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
671	#
672	# Determine in which output can we find file
673	# IE : Output/MO or Output/DA or ...
674	#
675	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
676	case ${FlagDir} in
677	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
678	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
679	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
680	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
681	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
682	*)
683	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
684	IGCM_sys_Exit "Job create_ts" ;;
685	esac
686
687	# Initialize array
688	#set -A liste_file
689	#set -A liste_file_tmp
690	unset liste_file
691	unset liste_file_tmp
692	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
693
694	COMPTEUR=1
695	DATE_LOOP=${DATE_COURANTE}
696
697	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
698	#
699	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
700	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
701
702	Length2=$( create_ts_next_date ${DATE_LOOP} end)
703	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
704
705	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
706	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
707	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
708	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
709	(( COMPTEUR = COMPTEUR + 1 ))
710	#
711	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
712	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
713
714	# Get selected files
715	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
716	ExitFlag=false
717	IGCM_debug_Verif_Exit_Post
718
719	# Apply IGCM_Patch if needed
720	if [ ! X${RebuildFrequency} = Xtrue ] ; then
721	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
722	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
723	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
724	IGCM_${Patch} ${file_t}
725	done
726	done
727	fi
728	fi
729
730	# WE CAN CONCATENATE FILES AT THIS POINT
731	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
732	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
733	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
734	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
735	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
736
737	liste_coord=" "
738	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
739	liste_coord=${liste_coord}${axis}","
740	done
741
742	# Time axis must be the last dimension
743	time_axis=${axis}
744
745	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
746	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
747	#
748	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
749	#
750	# # Change time axis name to be CF compliant
751	# if [ ${time_axis} != time_counter ]; then
752	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
753	# fi
754	#
755	if [ "${FLAG_B}" = "TRUE" ] ; then
756	# WE CONCATENATE WITH EXISTING FILES
757	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
758	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
759	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
760	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
761	#
762	IGCM_sys_Rm ${file1} ${file2}
763	fi
764	#
765	if [ "${DEPOT}" = "TRUE" ] ; then
766	# WE PUT FINAL FILE ON FILE SYSTEM
767	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
768	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
769	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
770	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
771	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
772	IGCM_sys_Rm ${file_out}
773	#
774	fi
775	#
776	ExitFlag=false
777	IGCM_debug_Verif_Exit_Post
778	done # for var in ${liste_var} ; do
779	IGCM_sys_Rm ${liste_file_tmp[*]}
780	done # for file in ${liste_file_brut} do
781	done # for comp in ${config_ListOfComponents[*]} ; do
782	ExitFlag=false
783	IGCM_debug_Verif_Exit_Post
784	# GO AHEAD IN DEEPEST LOOP
785	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
786	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
787	# FINISH
788
789	# DODS copy
790	for comp in ${config_ListOfComponents[*]} ; do
791	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
792	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
793	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
794	done
795	done
796
797	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
798	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF ; export RESOL_BIOS
799
800	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF,RESOL_BIOS"
801	export listVarEnv
802
803	#D-
804	# --------------------------------------------------------------------
805	#D- Test if all was right
806	# --------------------------------------------------------------------
807	IGCM_debug_Verif_Exit_Post
808	# --------------------------------------------------------------------
809	#D- Submit MONITORING
810	# --------------------------------------------------------------------
811	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
812
813	#D-
814	# --------------------------------------------------------------------
815	#D- Update the run.card
816	# --------------------------------------------------------------------
817	if [ ${StandAlone} != true ] ; then
818	# Put in run.card end period of time series
819	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
820	# We have finish the job !
821	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
822	fi
823	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: