Context Navigation

source: trunk/libIGCM/AA_create_ts @ 206

Last change on this file since 206 was 206, checked in by sdipsl, 15 years ago
Add LMDZ4 39 levels detection, launch atlas accordingly. Try to clarify which variables need to be change in case one use StandAlone? mode Clean up
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 29.5 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- default #!/bin/ksh
65	#-Q- default ##################
66	#-Q- default ## DEFAULT HOST ##
67	#-Q- default ##################
68
69	# $Date$
70	# $Author$
71	# $Revision$
72	# IPSL (2006)
73	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
74
75	#set -eu
76	#set -vx
77
78	date
79
80	#-Q- sx8brodie export OMP_NUM_THREADS=1
81	#-Q- aix6 export OMP_NUM_THREADS=1
82
83	########################################################################
84
85	#D- Flag to determine if this job in a standalone mode
86	#D- Default : value from AA_job if any
87	StandAlone=${StandAlone:=true}
88
89	#D- Low level debug : to bypass lib test checks and stack construction
90	#D- Default : value from AA_job if any
91	libIGCM=${libIGCM:=/path/to/your/libIGCM}
92
93	#D- Flag to determine allready produced time series. Empty if you start from the beginning
94	#D- Default : Commented. Uncomment for StandAlone case if you want to continue a time series
95	CompletedFlag=${CompletedFlag:=20001230}
96
97	#D- Flag to determine job's output directory
98	#D- Default : value from libIGCM_post.ksh if any
99	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
100
101	#D- Increased verbosity (1, 2, 3)
102	#D- Default : value from AA_job if any
103	Verbosity=${Verbosity:=3}
104
105	#D- Low level debug : to bypass lib test checks and stack construction
106	#D- Default : value from AA_job if any
107	DEBUG_debug=${DEBUG_debug:=false}
108
109	#D- TEMPORARY Flag to determine atmospheric resolution
110	#D- Default : value from atmospheric driver if any
111	RESOL_ATM=ALL
112
113	#D- TEMPORARY Flag to determine ocean resolution
114	#D- Default : value from ocean driver if any
115	RESOL_OCE=${RESOL_OCE:=ORCA2}
116
117	#D- TEMPORARY Flag to determine ice resolution
118	#D- Default : value from ice driver if any
119	RESOL_ICE=${RESOL_ICE:=ORCA2}
120
121	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
122	#D- Default : value from ice driver if any
123	RESOL_MBG=${RESOL_MBG:=ORCA2}
124
125	#D- Flag to determine surface resolution
126	#D- Default : value from surface driver if any
127	RESOL_SRF=ALL
128
129	#D- Flag to determine surface resolution
130	#D- Default : value from surface driver if any
131	RESOL_BIOS=ALL
132
133	########################################################################
134
135	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
136	( ${DEBUG_debug} ) && IGCM_debug_Check
137	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
138	( ${DEBUG_debug} ) && IGCM_card_Check
139	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
140	( ${DEBUG_debug} ) && IGCM_date_Check
141	#-------
142	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
143
144	########################################################################
145
146	#set -vx
147
148	# Useful cleaning function
149	MENAGE ()
150	{
151	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
152	}
153
154	########################################################################
155
156	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
157	IGCM_sys_Cd ${RUN_DIR_PATH}
158
159	if [ ${StandAlone} = true ] ; then
160	CARD_DIR=${SUBMIT_DIR}
161	else
162	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
163	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
164	fi
165
166	#
167	# First of all
168	#
169	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
170	typeset option
171	for option in ${config_UserChoices[*]} ; do
172	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
173	done
174	#
175	echo
176	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
177	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
178	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
179	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
180	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
181	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
182	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
183	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
184	echo
185
186	#==================================
187	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
188	#
189	# Determine component to take care of depending on kind of task create_ts will perform
190	#
191	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
192	#
193	# Determine period for time series. Default : value from AA_job or from launch_TS if any
194	#
195	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
196	#
197	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
198	#
199	# Determine Dimension of the produced time series : 2D or 3D
200	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
201	Dimension=2D
202	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
203	Dimension=3D
204	else
205	Dimension=""
206	fi
207
208
209	########################################################################
210	#
211	# Depending on requested time series fill following variables :
212	#
213	# - LISTE_FILE_${comp}[*]
214	# - LISTE_VARS_${file}[*]
215	# - LISTE_AXIS_${file}[*]
216	# - LISTE_PATCH_${file}[*]
217	########################################################################
218
219	#
220	# For each selected component determine which files need post-processing
221	#
222	DoJob=false
223	for comp in ${config_ListOfComponents[*]} ; do
224	#
225	ActiveComp=false
226	#
227	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
228	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
229	#
230	card=${CARD_DIR}/COMP/${compname}.card
231
232	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
233	#
234	ListFilesName=${compname}_OutputFiles_List
235	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
236	#
237	if [ X${FileName0} != X${NULL_STR} ] ; then
238	#
239	#IGCM_debug_Print 1 "Component : ${compname}"
240	#
241	# INITIALISATION
242	#
243	typeset i
244	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
245	((NbFiles=${FlagToRead}))
246	((i=${NbFiles}-2))
247	else
248	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
249	i=0
250	fi
251	#
252	until [ $i -ge $NbFiles ]; do
253	#
254	(( i_ = i+2 ))
255	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
256	eval flag_post=${flag_post_}
257	#
258	if [ X${flag_post} != XNONE ] ; then
259	#
260	# For each comp and each file who needs post-processing
261	# determine which variables need time series.
262	#
263	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
264	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
265	#
266	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
267	#
268	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
269	(( i=i+3 ))
270	continue
271	#
272	fi
273	#
274	# If we are not a Chunck type task, skip files/variables handled by chunck task
275	#
276	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
277	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
278	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
279	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
280	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
281	(( i=i+3 ))
282	continue
283	fi
284	fi
285	#
286	# We need coordinates axis to properly fill produced time series
287	#
288	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
289	#
290	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
291	#
292	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
293	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
294	(( i=i+3 ))
295	continue
296	#
297	fi
298	#
299	# We need LIST of variables not allready produced (useful for standalone mode)
300	#
301	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
302	case ${FlagDir} in
303	*Y) TS_Dir=TS_YE ;;
304	*M) TS_Dir=TS_MO ;;
305	*D) TS_Dir=TS_DA ;;
306	HF) TS_Dir=TS_HF ;;
307	INS) TS_Dir=TS_INS ;;
308	esac
309	#
310	count=0
311	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
312	#
313	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
314	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
315	#
316	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
317	if [ ! $? = 0 ] ; then
318	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
319	ActiveComp=true
320	(( count = count + 1 ))
321	fi
322	done
323	#
324	if [ ${count} -gt 0 ] ; then
325	#
326	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
327	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
328	#
329	if [ ! X${RebuildFrequency} = Xtrue ] ; then
330	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
331	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
332	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
333	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
334	if [ Xload${Patch} != X ] ; then
335	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
336	eval load${Patch}=loaded
337	fi
338	done
339	fi
340	fi
341	fi
342	fi
343	(( i=i+3 ))
344	done
345	fi
346	#
347	if [ ${ActiveComp} = true ] ; then
348	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
349	fi
350	done
351
352	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
353	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
354
355	########################################################################
356	#
357	# DEFINE DATE FUNCTIONS
358	#
359	########################################################################
360
361	# Give the end of the next $1 period from $2 date
362	function create_ts_next_date {
363	IGCM_debug_PushStack "create_ts_next_date" $@
364	typeset Length
365	case ${config_UserChoices_PeriodLength} in
366	Y\|y)
367	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
368	IGCM_date_GetYearMonth ${1} year month
369	(( Length=0 ))
370	(( i=0 ))
371	until [ $i -ge $PeriodLengthInYears ] ; do
372	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
373	(( i=i+1 ))
374	done
375	;;
376	M\|m)
377	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
378	IGCM_date_GetYearMonth ${1} year month
379	(( year0 = year ))
380	if [ $(( month + 1 )) -lt 13 ] ; then
381	month0=$(( month + 1 ))
382	else
383	month0=$(( month + 1 - 12 ))
384	(( year = year0 + 1 ))
385	fi
386	(( Length=0 ))
387	(( i=0 ))
388	until [ $i -ge $PeriodLengthInMonths ] ; do
389	if [ $(( month0 + i )) -lt 13 ] ; then
390	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
391	else
392	(( year = year0 + 1 ))
393	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
394	fi
395	(( i=i+1 ))
396	done
397	;;
398	D\|d)
399	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
400	*)
401	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
402	IGCM_debug_Verif_Exit_Post ;;
403	esac
404	echo ${Length}
405
406	IGCM_debug_PopStack "create_ts_next_date"
407	}
408
409	# Give the end of the actual $1 period from $2 date
410	function create_ts_begin_date {
411	IGCM_debug_PushStack "create_ts_begin_date" $@
412	typeset Length
413	case ${config_UserChoices_PeriodLength} in
414	Y\|y)
415	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
416	IGCM_date_GetYearMonth ${1} year month
417	if [ X${2} = Xend ] ; then
418	(( year = year - PeriodLengthInYears + 1))
419	fi
420	(( Length=0 ))
421	(( i=0 ))
422	until [ $i -ge $PeriodLengthInYears ] ; do
423	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
424	(( i=i+1 ))
425	done
426	;;
427	M\|m)
428	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
429	IGCM_date_GetYearMonth ${1} year month
430	if [ X${2} = Xend ] ; then
431	(( month = month - PeriodLengthInMonths + 1 ))
432	fi
433	(( year0 = year ))
434	if [ $month -le 0 ] ; then
435	(( month = month + 12 ))
436	year=$( printf "%04i\n" $(( year - 1 )) )
437	fi
438	month=$( printf "%02i\n" ${month} )
439	(( Length=0 ))
440	(( i=0 ))
441	until [ $i -ge $PeriodLengthInMonths ] ; do
442	if [ $(( month + i )) -lt 13 ] ; then
443	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
444	else
445	(( year = year0 + 1 ))
446	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
447	fi
448	(( i=i+1 ))
449	done
450	;;
451	D\|d)
452	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
453	*)
454	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
455	IGCM_debug_Verif_Exit_Post ;;
456	esac
457	echo ${Length}
458
459	IGCM_debug_PopStack "create_ts_begin_date"
460	}
461
462	########################################################################
463	#
464	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
465	# IF SO BRING THEM IN THE WORKING DIRECTORY
466	#
467	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
468	# - especially for IDRiS
469	# - no special protection against "command too long" !
470	# - libIGCM_sys should handle that or modify following code part
471	#
472	########################################################################
473
474	if [ ${StandAlone} != true ] ; then
475	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
476	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
477	echo \${run_PostProcessing_TimeSeriesRunning}" )
478
479	#if [ X${Running_Flag} = Xy ] ; then
480	# # Time Series Job allready running
481	# IGCM_debug_Print 1 "Time Series Job allready running exit"
482	# exit
483	#fi
484	# Now run.card know we are running
485
486
487
488
489	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
490
491
492
493
494
495	# Get information from last execution
496	#CompletedFlag=$( IGCM_sys_RshMaster \
497	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
498	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
499	fi
500
501	# Is it the first submission or not ?
502
503	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
504	# First Time Series Submission
505	FIRST_PASS=TRUE
506	Length=$( create_ts_begin_date ${DateBegin} begin)
507	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
508	else
509	# Last Completed Time Series
510	FIRST_PASS=FALSE
511	DATE_FIN_JOB_B=${CompletedFlag}
512	date=${DateBegin}_${DATE_FIN_JOB_B}
513	for comp in ${config_ListOfComponents[*]} ; do
514	#
515	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
516	#
517	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
518	#
519	#set -A liste_file
520	unset liste_file
521	#
522	file_path=${R_SAVE}/${comp}/Analyse
523	#
524	i=0
525	file=${NULL_STR}
526	#
527	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
528	#
529	# Determine in which output can we find file
530	# IE : Analyse/TS or Analyse/DA2TS or ...
531	#
532	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
533	case ${FlagDir} in
534	*Y) TS_Dir=TS_YE ;;
535	*M) TS_Dir=TS_MO ;;
536	*D) TS_Dir=TS_DA ;;
537	HF) TS_Dir=TS_HF ;;
538	INS) TS_Dir=TS_INS ;;
539	esac
540	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
541	#
542	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
543	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
544	#
545	done
546	done
547	if [ X${file} != X${NULL_STR} ] ; then
548	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
549	ExitFlag=false
550	IGCM_debug_Verif_Exit_Post
551	fi
552	done
553	#
554	LEVEL=0
555	#
556	fi
557
558	########################################################################
559	#
560	# DEFINE LOOP PARAMETERS
561	#
562	########################################################################
563
564	DATE_COUNT=${DATE_FIN_JOB_B}
565	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
566
567	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
568
569	if [ ${DoJob} = true ] ; then
570	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
571	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
572	Length=$( create_ts_next_date ${DATE_COUNT} )
573	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
574	done
575	else
576	NBRE_FILE_TOT=0
577	fi
578
579	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
580	NBRE_TOUR=1
581	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
582	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
583	else
584	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
585	fi
586
587	CURRENT_LOOP=1
588	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
589	# WE HAVE 3 LEVELS
590	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
591	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
592	# - LEVEL 3 : WE ARE IN THE LAST LOOP
593
594	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
595
596	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
597	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
598	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
599
600	if [ ${FIRST_PASS} = TRUE ] ; then
601	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
602	else
603	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
604	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
605	fi
606
607	DATE_FIN=${PeriodDateEnd}
608	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
609
610	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
611	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
612
613	LEVEL=1
614	DEPOT="TRUE"
615	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
616	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
617	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
618
619	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
620	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
621	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
622	elif [ -z "${DATE_COURANTE}" ] ; then
623	DATE_COURANTE=${DATE_FIN_JOB_B}
624	else
625	Length=$( create_ts_next_date ${DATE_FIN} )
626	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
627	fi
628
629	(( TotLength = 0 ))
630	COMPTEUR=1
631	DATE_LOOP=${DATE_COURANTE}
632
633	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
634	#
635	Length=$( create_ts_next_date ${DATE_LOOP} )
636	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
637	(( TotLength = TotLength + Length ))
638	(( COMPTEUR = COMPTEUR + 1 ))
639	#
640	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
641	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
642
643	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
644	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
645	DEPOT="FALSE"
646	LEVEL=2
647	else
648	# - LEVEL 3 : WE ARE IN THE LAST LOOP
649	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
650	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
651
652	Length=$( create_ts_next_date ${DATE_FIN} )
653	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
654	DATE_FIN=${PeriodDateEnd}
655	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
656
657	LEVEL=3
658	DEPOT="TRUE"
659	FLAG_B="TRUE"
660	fi
661
662	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
663	for comp in ${config_ListOfComponents[*]} ; do
664	#
665	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
666	#
667	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
668	#
669	eval R_OUT_${comp}=${R_SAVE}/${comp}
670	#
671	i=0
672	#
673	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
674	#
675	# Determine in which output can we find file
676	# IE : Output/MO or Output/DA or ...
677	#
678	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
679	case ${FlagDir} in
680	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
681	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
682	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
683	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
684	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
685	*)
686	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
687	IGCM_sys_Exit "Job create_ts" ;;
688	esac
689
690	# Initialize array
691	#set -A liste_file
692	#set -A liste_file_tmp
693	unset liste_file
694	unset liste_file_tmp
695	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
696
697	COMPTEUR=1
698	DATE_LOOP=${DATE_COURANTE}
699
700	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
701	#
702	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
703	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
704
705	Length2=$( create_ts_next_date ${DATE_LOOP} end)
706	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
707
708	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
709	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
710	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
711	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
712	(( COMPTEUR = COMPTEUR + 1 ))
713	#
714	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
715	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
716
717	# Get selected files
718	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
719	ExitFlag=false
720	IGCM_debug_Verif_Exit_Post
721
722	# Apply IGCM_Patch if needed
723	if [ ! X${RebuildFrequency} = Xtrue ] ; then
724	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
725	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
726	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
727	IGCM_${Patch} ${file_t}
728	done
729	done
730	fi
731	fi
732
733	# WE CAN CONCATENATE FILES AT THIS POINT
734	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
735	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
736	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
737	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
738	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
739
740	liste_coord=" "
741	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
742	liste_coord=${liste_coord}${axis}","
743	done
744
745	# Time axis must be the last dimension
746	time_axis=${axis}
747
748	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
749	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
750	#
751	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
752	#
753	# # Change time axis name to be CF compliant
754	# if [ ${time_axis} != time_counter ]; then
755	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
756	# fi
757	#
758	if [ "${FLAG_B}" = "TRUE" ] ; then
759	# WE CONCATENATE WITH EXISTING FILES
760	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
761	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
762	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
763	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
764	#
765	IGCM_sys_Rm ${file1} ${file2}
766	fi
767	#
768	if [ "${DEPOT}" = "TRUE" ] ; then
769	# WE PUT FINAL FILE ON FILE SYSTEM
770	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
771	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
772	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
773	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
774	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
775	IGCM_sys_Rm ${file_out}
776	#
777	fi
778	#
779	ExitFlag=false
780	IGCM_debug_Verif_Exit_Post
781	done # for var in ${liste_var} ; do
782	IGCM_sys_Rm ${liste_file_tmp[*]}
783	done # for file in ${liste_file_brut} do
784	done # for comp in ${config_ListOfComponents[*]} ; do
785	ExitFlag=false
786	IGCM_debug_Verif_Exit_Post
787	# GO AHEAD IN DEEPEST LOOP
788	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
789	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
790	# FINISH
791
792	# DODS copy
793	for comp in ${config_ListOfComponents[*]} ; do
794	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
795	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
796	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
797	done
798	done
799
800	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
801	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_BIOS
802
803	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_BIOS"
804	export listVarEnv
805
806	#D-
807	# --------------------------------------------------------------------
808	#D- Test if all was right
809	# --------------------------------------------------------------------
810	IGCM_debug_Verif_Exit_Post
811	# --------------------------------------------------------------------
812	#D- Submit MONITORING
813	# --------------------------------------------------------------------
814	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
815
816	#D-
817	# --------------------------------------------------------------------
818	#D- Update the run.card
819	# --------------------------------------------------------------------
820	if [ ${StandAlone} != true ] ; then
821	# Put in run.card end period of time series
822	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
823	# We have finish the job !
824	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
825	fi
826
827	# Clean RUN_DIR_PATH (necessary for cesium)
828	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
829
830	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: