Context Navigation

source: trunk/libIGCM/AA_create_ts @ 152

Last change on this file since 152 was 149, checked in by sdipsl, 15 years ago
New post-processing functionnality for cmip5 like post-processing You can ask for fixed lenght time series (10 years chunck or 50 years chunck or ...) and specify it for each files. Separation of concern between 2D and 3D fields. Backward compatible with previous style comp.card example : TimeSeriesVars2D = (cldt, psol, q2m, slp, SWnetOR, SWdownOR, LWdownOR, precip) ChunckJob2D = NONE TimeSeriesVars3D = (temp, theta, ovap, ovapinit, geop, vitu, vitv, vitw, pres) ChunckJob3D = 10Y In case of asynchronous rebuild save patched files on archive filesystem. (so that they have good time axis)
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 28.4 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ##################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ##################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_SRF ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_SRF ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=2:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- default #!/bin/ksh
53	#-Q- default ##################
54	#-Q- default ## DEFAULT HOST ##
55	#-Q- default ##################
56
57	# $Date$
58	# $Author$
59	# $Revision$
60	# IPSL (2006)
61	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
62
63	#set -eu
64	#set -vx
65
66	date
67
68	#-Q- sx8brodie export OMP_NUM_THREADS=1
69	#-Q- aix6 export OMP_NUM_THREADS=1
70
71	########################################################################
72
73	#D- Flag to determine if this job in a standalone mode
74	#D- Default : value from AA_job if any
75	StandAlone=${StandAlone:=true}
76
77	#D- Flag to determine allready produced time series.
78	#D- Default : Commented. Uncomment for StandAlone case if you want to continue a time series
79	#CompletedFlag=
80
81	#D- Flag to determine job's output directory
82	#D- Default : value from libIGCM_post.ksh if any
83	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
84
85	#D- Increased verbosity (1, 2, 3)
86	#D- Default : value from AA_job if any
87	Verbosity=${Verbosity:=3}
88
89	#D- Low level debug : to bypass lib test checks and stack construction
90	#D- Default : value from AA_job if any
91	DEBUG_debug=${DEBUG_debug:=false}
92
93	#D- Low level debug : to bypass lib test checks and stack construction
94	#D- Default : value from AA_job if any
95	libIGCM=${libIGCM:=/path/to/your/libIGCM}
96
97	#D- TEMPORARY Flag to determine atmospheric resolution
98	#D- Default : value from atmospheric driver if any
99	RESOL_ATM=ALL
100
101	#D- TEMPORARY Flag to determine ocean resolution
102	#D- Default : value from ocean driver if any
103	RESOL_OCE=${RESOL_OCE:=ORCA2}
104
105	#D- TEMPORARY Flag to determine ice resolution
106	#D- Default : value from ice driver if any
107	RESOL_ICE=${RESOL_ICE:=ORCA2}
108
109	#D- Flag to determine surface resolution
110	#D- Default : value from surface driver if any
111	RESOL_SRF=ALL
112
113	########################################################################
114
115	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
116	( ${DEBUG_debug} ) && IGCM_debug_Check
117	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
118	( ${DEBUG_debug} ) && IGCM_card_Check
119	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
120	( ${DEBUG_debug} ) && IGCM_date_Check
121	#-------
122	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
123
124	########################################################################
125
126	#set -vx
127
128	# Useful cleaning function
129	MENAGE ()
130	{
131	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
132	}
133
134	########################################################################
135
136	if [ ${DEBUG_debug} = true ] ; then
137	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
138	echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
139	fi
140	IGCM_sys_Cd ${RUN_DIR_PATH}
141
142	if [ ${StandAlone} = true ] ; then
143	CARD_DIR=${SUBMIT_DIR}
144	else
145	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
146	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
147	fi
148
149	#
150	# First of all
151	#
152	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
153	typeset option
154	for option in ${config_UserChoices[*]} ; do
155	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
156	done
157	#
158	echo
159	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
160	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
161	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
162	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
163	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
164	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
165	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
166	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
167	echo
168
169	#==================================
170	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
171	#
172	# Determine component to take care of depending on kind of task create_ts will perform
173	#
174	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
175	#
176	# Determine period for time series. Default : value from AA_job or from launch_TS if any
177	#
178	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
179	#
180	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
181	#
182	# Determine Dimension of the produced time series : 2D or 3D
183	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
184	Dimension=2D
185	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
186	Dimension=3D
187	else
188	Dimension=""
189	fi
190
191
192	########################################################################
193	#
194	# Depending on requested time series fill following variables :
195	#
196	# - LISTE_FILE_${comp}[*]
197	# - LISTE_VARS_${file}[*]
198	# - LISTE_AXIS_${file}[*]
199	# - LISTE_PATCH_${file}[*]
200	########################################################################
201
202	#
203	# For each selected component determine which files need post-processing
204	#
205	DoJob=false
206	for comp in ${config_ListOfComponents[*]} ; do
207	#
208	ActiveComp=false
209	#
210	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
211	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
212	#
213	card=${CARD_DIR}/COMP/${compname}.card
214
215	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
216	#
217	ListFilesName=${compname}_OutputFiles_List
218	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
219	#
220	if [ X${FileName0} != X${NULL_STR} ] ; then
221	#
222	#IGCM_debug_Print 1 "Component : ${compname}"
223	#
224	# INITIALISATION
225	#
226	typeset i
227	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
228	((NbFiles=${FlagToRead}))
229	((i=${NbFiles}-2))
230	else
231	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
232	i=0
233	fi
234	#
235	until [ $i -ge $NbFiles ]; do
236	#
237	(( i_ = i+2 ))
238	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
239	eval flag_post=${flag_post_}
240	#
241	if [ X${flag_post} != XNONE ] ; then
242	#
243	# For each comp and each file who needs post-processing
244	# determine which variables need time series.
245	#
246	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
247	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
248	#
249	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
250	#
251	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
252	(( i=i+3 ))
253	continue
254	#
255	fi
256	#
257	# If we are not a Chunck type task, skip files/variables handled by chunck task
258	#
259	if [ ! X${TsTask} = XChunck${Dimension} ] ; then
260	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
261	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
262	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
263	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
264	(( i=i+3 ))
265	continue
266	fi
267	fi
268	#
269	# We need coordinates axis to properly fill produced time series
270	#
271	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
272	#
273	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
274	#
275	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
276	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
277	(( i=i+3 ))
278	continue
279	#
280	fi
281	#
282	# We need LIST of variables not allready produced (useful for standalone mode)
283	#
284	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
285	case ${FlagDir} in
286	*Y) TS_Dir=TS_YE ;;
287	*M) TS_Dir=TS_MO ;;
288	*D) TS_Dir=TS_DA ;;
289	HF) TS_Dir=TS_HF ;;
290	INS) TS_Dir=TS_INS ;;
291	esac
292	#
293	count=0
294	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
295	#
296	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
297	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
298	#
299	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
300	if [ ! $? = 0 ] ; then
301	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
302	ActiveComp=true
303	(( count = count + 1 ))
304	fi
305	done
306	#
307	if [ ${count} -gt 0 ] ; then
308	#
309	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
310	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
311	#
312	if [ ! ${RebuildFrequency} = true ] ; then
313	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
314	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
315	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
316	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
317	if [ Xload${Patch} != X ] ; then
318	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
319	eval load${Patch}=loaded
320	fi
321	done
322	fi
323	fi
324	fi
325	fi
326	(( i=i+3 ))
327	done
328	fi
329	#
330	if [ ${ActiveComp} = true ] ; then
331	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
332	fi
333	done
334
335	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
336	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
337
338	########################################################################
339	#
340	# DEFINE DATE FUNCTIONS
341	#
342	########################################################################
343
344	# Give the end of the next $1 period from $2 date
345	function create_ts_next_date {
346	IGCM_debug_PushStack "create_ts_next_date" $@
347	typeset Length
348	case ${config_UserChoices_PeriodLength} in
349	Y\|y)
350	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
351	IGCM_date_GetYearMonth ${1} year month
352	(( Length=0 ))
353	(( i=0 ))
354	until [ $i -ge $PeriodLengthInYears ] ; do
355	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
356	(( i=i+1 ))
357	done
358	;;
359	M\|m)
360	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
361	IGCM_date_GetYearMonth ${1} year month
362	(( year0 = year ))
363	if [ $(( month + 1 )) -lt 13 ] ; then
364	month0=$(( month + 1 ))
365	else
366	month0=$(( month + 1 - 12 ))
367	(( year = year0 + 1 ))
368	fi
369	(( Length=0 ))
370	(( i=0 ))
371	until [ $i -ge $PeriodLengthInMonths ] ; do
372	if [ $(( month0 + i )) -lt 13 ] ; then
373	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
374	else
375	(( year = year0 + 1 ))
376	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
377	fi
378	(( i=i+1 ))
379	done
380	;;
381	D\|d)
382	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
383	*)
384	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
385	IGCM_debug_Verif_Exit_Post ;;
386	esac
387	echo ${Length}
388
389	IGCM_debug_PopStack "create_ts_next_date"
390	}
391
392	# Give the end of the actual $1 period from $2 date
393	function create_ts_begin_date {
394	IGCM_debug_PushStack "create_ts_begin_date" $@
395	typeset Length
396	case ${config_UserChoices_PeriodLength} in
397	Y\|y)
398	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
399	IGCM_date_GetYearMonth ${1} year month
400	if [ X${2} = Xend ] ; then
401	(( year = year - PeriodLengthInYears + 1))
402	fi
403	(( Length=0 ))
404	(( i=0 ))
405	until [ $i -ge $PeriodLengthInYears ] ; do
406	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
407	(( i=i+1 ))
408	done
409	;;
410	M\|m)
411	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
412	IGCM_date_GetYearMonth ${1} year month
413	if [ X${2} = Xend ] ; then
414	(( month = month - PeriodLengthInMonths + 1 ))
415	fi
416	(( year0 = year ))
417	if [ $month -le 0 ] ; then
418	(( month = month + 12 ))
419	year=$( printf "%04i\n" $(( year - 1 )) )
420	fi
421	month=$( printf "%02i\n" ${month} )
422	(( Length=0 ))
423	(( i=0 ))
424	until [ $i -ge $PeriodLengthInMonths ] ; do
425	if [ $(( month + i )) -lt 13 ] ; then
426	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
427	else
428	(( year = year0 + 1 ))
429	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
430	fi
431	(( i=i+1 ))
432	done
433	;;
434	D\|d)
435	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
436	*)
437	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
438	IGCM_debug_Verif_Exit_Post ;;
439	esac
440	echo ${Length}
441
442	IGCM_debug_PopStack "create_ts_begin_date"
443	}
444
445	########################################################################
446	#
447	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
448	# IF SO BRING THEM IN THE WORKING DIRECTORY
449	#
450	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
451	# - especially for IDRiS
452	# - no special protection against "command too long" !
453	# - libIGCM_sys should handle that or modify following code part
454	#
455	########################################################################
456
457	if [ ${StandAlone} != true ] ; then
458	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
459	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
460	echo \${run_PostProcessing_TimeSeriesRunning}" )
461
462	#if [ X${Running_Flag} = Xy ] ; then
463	# # Time Series Job allready running
464	# IGCM_debug_Print 1 "Time Series Job allready running exit"
465	# exit
466	#fi
467	# Now run.card know we are running
468
469
470
471
472	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
473
474
475
476
477
478	# Get information from last execution
479	#CompletedFlag=$( IGCM_sys_RshMaster \
480	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
481	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
482	fi
483
484	# Is it the first submission or not ?
485
486	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
487	# First Time Series Submission
488	FIRST_PASS=TRUE
489	Length=$( create_ts_begin_date ${DateBegin} begin)
490	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
491	else
492	# Last Completed Time Series
493	FIRST_PASS=FALSE
494	DATE_FIN_JOB_B=${CompletedFlag}
495	date=${DateBegin}_${DATE_FIN_JOB_B}
496	for comp in ${config_ListOfComponents[*]} ; do
497	#
498	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
499	#
500	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
501	#
502	#set -A liste_file
503	unset liste_file
504	#
505	file_path=${R_SAVE}/${comp}/Analyse
506	#
507	i=0
508	file=${NULL_STR}
509	#
510	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
511	#
512	# Determine in which output can we find file
513	# IE : Analyse/TS or Analyse/DA2TS or ...
514	#
515	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
516	case ${FlagDir} in
517	*Y) TS_Dir=TS_YE ;;
518	*M) TS_Dir=TS_MO ;;
519	*D) TS_Dir=TS_DA ;;
520	HF) TS_Dir=TS_HF ;;
521	INS) TS_Dir=TS_INS ;;
522	esac
523	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
524	#
525	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
526	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
527	#
528	done
529	done
530	if [ X${file} != X${NULL_STR} ] ; then
531	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
532	ExitFlag=false
533	IGCM_debug_Verif_Exit_Post
534	fi
535	done
536	#
537	LEVEL=0
538	#
539	fi
540
541	########################################################################
542	#
543	# DEFINE LOOP PARAMETERS
544	#
545	########################################################################
546
547	DATE_COUNT=${DATE_FIN_JOB_B}
548	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
549
550	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
551
552	if [ ${DoJob} = true ] ; then
553	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
554	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
555	Length=$( create_ts_next_date ${DATE_COUNT} )
556	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
557	done
558	else
559	NBRE_FILE_TOT=0
560	fi
561
562	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
563	NBRE_TOUR=1
564	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
565	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
566	else
567	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
568	fi
569
570	CURRENT_LOOP=1
571	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
572	# WE HAVE 3 LEVELS
573	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
574	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
575	# - LEVEL 3 : WE ARE IN THE LAST LOOP
576
577	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
578
579	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
580	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
581	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
582
583	if [ ${FIRST_PASS} = TRUE ] ; then
584	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
585	else
586	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
587	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
588	fi
589
590	DATE_FIN=${PeriodDateEnd}
591	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
592
593	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
594	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
595
596	LEVEL=1
597	DEPOT="TRUE"
598	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
599	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
600	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
601
602	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
603	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
604	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
605	elif [ -z "${DATE_COURANTE}" ] ; then
606	DATE_COURANTE=${DATE_FIN_JOB_B}
607	else
608	Length=$( create_ts_next_date ${DATE_FIN} )
609	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
610	fi
611
612	(( TotLength = 0 ))
613	COMPTEUR=1
614	DATE_LOOP=${DATE_COURANTE}
615
616	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
617	#
618	Length=$( create_ts_next_date ${DATE_LOOP} )
619	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
620	(( TotLength = TotLength + Length ))
621	(( COMPTEUR = COMPTEUR + 1 ))
622	#
623	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
624	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
625
626	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
627	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
628	DEPOT="FALSE"
629	LEVEL=2
630	else
631	# - LEVEL 3 : WE ARE IN THE LAST LOOP
632	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
633	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
634
635	Length=$( create_ts_next_date ${DATE_FIN} )
636	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
637	DATE_FIN=${PeriodDateEnd}
638	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
639
640	LEVEL=3
641	DEPOT="TRUE"
642	FLAG_B="TRUE"
643	fi
644
645	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
646	for comp in ${config_ListOfComponents[*]} ; do
647	#
648	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
649	#
650	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
651	#
652	eval R_OUT_${comp}=${R_SAVE}/${comp}
653	#
654	i=0
655	#
656	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
657	#
658	# Determine in which output can we find file
659	# IE : Output/MO or Output/DA or ...
660	#
661	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
662	case ${FlagDir} in
663	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
664	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
665	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
666	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
667	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
668	*)
669	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
670	IGCM_sys_Exit "Job create_ts" ;;
671	esac
672
673	# Initialize array
674	#set -A liste_file
675	#set -A liste_file_tmp
676	unset liste_file
677	unset liste_file_tmp
678	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
679
680	COMPTEUR=1
681	DATE_LOOP=${DATE_COURANTE}
682
683	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
684	#
685	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
686	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
687
688	Length2=$( create_ts_next_date ${DATE_LOOP} end)
689	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
690
691	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
692	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
693	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
694	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
695	(( COMPTEUR = COMPTEUR + 1 ))
696	#
697	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
698	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
699
700	# Get selected files
701	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
702	ExitFlag=false
703	IGCM_debug_Verif_Exit_Post
704
705	# Apply IGCM_Patch if needed
706	if [ ! ${RebuildFrequency} = true ] ; then
707	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
708	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
709	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
710	IGCM_${Patch} ${file_t}
711	done
712	done
713	fi
714	fi
715
716	# WE CAN CONCATENATE FILES AT THIS POINT
717	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
718	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
719	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
720	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
721	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
722
723	liste_coord=" "
724	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
725	liste_coord=${liste_coord}${axis}","
726	done
727
728	# Time axis must be the last dimension
729	time_axis=${axis}
730
731	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
732	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
733	#
734	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
735	#
736	# # Change time axis name to be CF compliant
737	# if [ ${time_axis} != time_counter ]; then
738	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
739	# fi
740	#
741	if [ "${FLAG_B}" = "TRUE" ] ; then
742	# WE CONCATENATE WITH EXISTING FILES
743	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
744	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
745	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
746	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
747	#
748	IGCM_sys_Rm ${file1} ${file2}
749	fi
750	#
751	if [ "${DEPOT}" = "TRUE" ] ; then
752	# WE PUT FINAL FILE ON FILE SYSTEM
753	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
754	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
755	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
756	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
757	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
758	IGCM_sys_Rm ${file_out}
759	#
760	fi
761	#
762	ExitFlag=false
763	IGCM_debug_Verif_Exit_Post
764	done # for var in ${liste_var} ; do
765	IGCM_sys_Rm ${liste_file_tmp[*]}
766	done # for file in ${liste_file_brut} do
767	done # for comp in ${config_ListOfComponents[*]} ; do
768	ExitFlag=false
769	IGCM_debug_Verif_Exit_Post
770	# GO AHEAD IN DEEPEST LOOP
771	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
772	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
773	# FINISH
774
775	# DODS copy
776	for comp in ${config_ListOfComponents[*]} ; do
777	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
778	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
779	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
780	done
781	done
782
783	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
784	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_SRF
785
786	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_SRF"
787	export listVarEnv
788
789	#D-
790	# --------------------------------------------------------------------
791	#D- Test if all was right
792	# --------------------------------------------------------------------
793	IGCM_debug_Verif_Exit_Post
794	# --------------------------------------------------------------------
795	#D- Submit MONITORING
796	# --------------------------------------------------------------------
797	[ X${TsTask} = X2D ] && IGCM_sys_QsubPost monitoring
798
799	#D-
800	# --------------------------------------------------------------------
801	#D- Update the run.card
802	# --------------------------------------------------------------------
803	if [ ${StandAlone} != true ] ; then
804	# Put in run.card end period of time series
805	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
806	# We have finish the job !
807	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
808	fi
809	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: