Context Navigation

AA_create_ts @ 222

Last change on this file since 222 was 222, checked in by aclsce, 14 years ago

Added titane (CCRT Xeon machine) as target machine :

for the moment, post-treatment will be done on cesium.

Removed the use of dmget command on platine (CCRT Itanium machine)

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Date Author Revision

File size: 30.5 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 20:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- titane #!/bin/ksh
65	#-Q- titane ######################
66	#-Q- titane ## TITANE CEA ##
67	#-Q- titane ######################
68	#-Q- titane #MSUB -r TS # Nom du job
69	#-Q- titane #MSUB -N 1 # Reservation du noeud
70	#-Q- titane #MSUB -n 1 # Reservation du processus
71	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
72	#-Q- titane #MSUB -E "-j o"
73	#-Q- titane #MSUB -E "-S /bin/ksh"
74	#-Q- titane ##MSUB -e nco.out # Sortie standard
75	#-Q- titane ##MSUB -o nco.out # Sortie standard
76	#-Q- default #!/bin/ksh
77	#-Q- default ##################
78	#-Q- default ## DEFAULT HOST ##
79	#-Q- default ##################
80
81	# $Date$
82	# $Author$
83	# $Revision$
84	# IPSL (2006)
85	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
86
87	#set -eu
88	#set -vx
89
90	date
91
92	#-Q- sx8brodie export OMP_NUM_THREADS=1
93	#-Q- aix6 export OMP_NUM_THREADS=1
94
95	########################################################################
96
97	#D- Flag to determine if this job in a standalone mode
98	#D- Default : value from AA_job if any
99	StandAlone=${StandAlone:=true}
100
101	#D- Low level debug : to bypass lib test checks and stack construction
102	#D- Default : value from AA_job if any
103	libIGCM=${libIGCM:=/path/to/your/libIGCM}
104
105	#D- Flag to determine allready produced time series. Empty if you start from the beginning
106	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
107	CompletedFlag=${CompletedFlag:=}
108
109	#D- Flag to determine task type this script will perform.
110	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
111	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
112	TsTask=${TsTask:=2D}
113
114	#D- Flag to determine if rebuild process has been performed asynchronously
115	#D- Possible value true or false.
116	#D- If true raw files has allready been patched by rebuild job
117	#D- If false create_ts will patch the raw files
118	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
119	RebuildFrequency=${RebuildFrequency:=true}
120
121	#D- Flag to determine job's output directory
122	#D- Default : value from libIGCM_post.ksh if any
123	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
124
125	#D- Increased verbosity (1, 2, 3)
126	#D- Default : value from AA_job if any
127	Verbosity=${Verbosity:=3}
128
129	#D- Low level debug : to bypass lib test checks and stack construction
130	#D- Default : value from AA_job if any
131	DEBUG_debug=${DEBUG_debug:=false}
132
133	#D- TEMPORARY Flag to determine atmospheric resolution
134	#D- Default : value from atmospheric driver if any
135	RESOL_ATM=${RESOL_ATM:=ALL}
136
137	#D- TEMPORARY Flag to determine ocean resolution
138	#D- Default : value from ocean driver if any
139	RESOL_OCE=${RESOL_OCE:=ORCA2}
140
141	#D- TEMPORARY Flag to determine ice resolution
142	#D- Default : value from ice driver if any
143	RESOL_ICE=${RESOL_ICE:=ORCA2}
144
145	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
146	#D- Default : value from ice driver if any
147	RESOL_MBG=${RESOL_MBG:=ORCA2}
148
149	#D- Flag to determine surface resolution
150	#D- Default : value from surface driver if any
151	RESOL_SRF=ALL
152
153	#D- Flag to determine surface resolution
154	#D- Default : value from surface driver if any
155	RESOL_SBG=ALL
156
157	########################################################################
158
159	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
160	( ${DEBUG_debug} ) && IGCM_debug_Check
161	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
162	( ${DEBUG_debug} ) && IGCM_card_Check
163	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
164	( ${DEBUG_debug} ) && IGCM_date_Check
165	#-------
166	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
167
168	########################################################################
169
170	#set -vx
171
172	# Useful cleaning function
173	MENAGE ()
174	{
175	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
176	}
177
178	########################################################################
179
180	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
181	IGCM_sys_Cd ${RUN_DIR_PATH}
182
183	if [ ${StandAlone} = true ] ; then
184	CARD_DIR=${SUBMIT_DIR}
185	else
186	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
187	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
188	fi
189
190	#
191	# First of all
192	#
193	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
194	typeset option
195	for option in ${config_UserChoices[*]} ; do
196	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
197	done
198	#
199	echo
200	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
201	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
202	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
203	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
204	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
205	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
206	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
207	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
208	echo
209
210	#==================================
211	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
212	#
213	# Determine component to take care of depending on kind of task create_ts will perform
214	#
215	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
216	#
217	# Determine period for time series. Default : value from AA_job or from launch_TS if any
218	#
219	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
220	#
221	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
222	#
223	# Determine Dimension of the produced time series : 2D or 3D
224	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
225	Dimension=2D
226	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
227	Dimension=3D
228	else
229	Dimension=""
230	fi
231
232
233	########################################################################
234	#
235	# Depending on requested time series fill following variables :
236	#
237	# - LISTE_FILE_${comp}[*]
238	# - LISTE_VARS_${file}[*]
239	# - LISTE_AXIS_${file}[*]
240	# - LISTE_PATCH_${file}[*]
241	########################################################################
242
243	#
244	# For each selected component determine which files need post-processing
245	#
246	DoJob=false
247	for comp in ${config_ListOfComponents[*]} ; do
248	#
249	ActiveComp=false
250	#
251	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
252	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
253	#
254	card=${CARD_DIR}/COMP/${compname}.card
255
256	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
257	#
258	ListFilesName=${compname}_OutputFiles_List
259	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
260	#
261	if [ X${FileName0} != X${NULL_STR} ] ; then
262	#
263	#IGCM_debug_Print 1 "Component : ${compname}"
264	#
265	# INITIALISATION
266	#
267	typeset i
268	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
269	((NbFiles=${FlagToRead}))
270	((i=${NbFiles}-2))
271	else
272	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
273	i=0
274	fi
275	#
276	until [ $i -ge $NbFiles ]; do
277	#
278	(( i_ = i+2 ))
279	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
280	eval flag_post=${flag_post_}
281	#
282	if [ X${flag_post} != XNONE ] ; then
283	#
284	# For each comp and each file who needs post-processing
285	# determine which variables need time series.
286	#
287	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
288	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
289	#
290	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
291	#
292	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
293	(( i=i+3 ))
294	continue
295	#
296	fi
297	#
298	# If we are not a Chunck type task, skip files/variables handled by chunck task
299	#
300	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
301	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
302	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
303	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
304	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
305	(( i=i+3 ))
306	continue
307	fi
308	fi
309	#
310	# We need coordinates axis to properly fill produced time series
311	#
312	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
313	#
314	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
315	#
316	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
317	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
318	(( i=i+3 ))
319	continue
320	#
321	fi
322	#
323	# We need LIST of variables not allready produced (useful for standalone mode)
324	#
325	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
326	case ${FlagDir} in
327	*Y) TS_Dir=TS_YE ;;
328	*M) TS_Dir=TS_MO ;;
329	*D) TS_Dir=TS_DA ;;
330	HF) TS_Dir=TS_HF ;;
331	INS) TS_Dir=TS_INS ;;
332	esac
333	#
334	count=0
335	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
336	#
337	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
338	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
339	#
340	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
341	if [ ! $? = 0 ] ; then
342	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
343	ActiveComp=true
344	(( count = count + 1 ))
345	fi
346	done
347	#
348	if [ ${count} -gt 0 ] ; then
349	#
350	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
351	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
352	#
353	if [ ! X${RebuildFrequency} = Xtrue ] ; then
354	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
355	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
356	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
357	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
358	if [ Xload${Patch} != X ] ; then
359	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
360	eval load${Patch}=loaded
361	fi
362	done
363	fi
364	fi
365	fi
366	fi
367	(( i=i+3 ))
368	done
369	fi
370	#
371	if [ ${ActiveComp} = true ] ; then
372	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
373	fi
374	done
375
376	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
377	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
378
379	########################################################################
380	#
381	# DEFINE DATE FUNCTIONS
382	#
383	########################################################################
384
385	# Give the end of the next $1 period from $2 date
386	function create_ts_next_date {
387	IGCM_debug_PushStack "create_ts_next_date" $@
388	typeset Length
389	case ${config_UserChoices_PeriodLength} in
390	Y\|y)
391	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
392	IGCM_date_GetYearMonth ${1} year month
393	(( Length=0 ))
394	(( i=0 ))
395	until [ $i -ge $PeriodLengthInYears ] ; do
396	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
397	(( i=i+1 ))
398	done
399	;;
400	M\|m)
401	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
402	IGCM_date_GetYearMonth ${1} year month
403	(( year0 = year ))
404	if [ $(( month + 1 )) -lt 13 ] ; then
405	month0=$(( month + 1 ))
406	else
407	month0=$(( month + 1 - 12 ))
408	(( year = year0 + 1 ))
409	fi
410	(( Length=0 ))
411	(( i=0 ))
412	until [ $i -ge $PeriodLengthInMonths ] ; do
413	if [ $(( month0 + i )) -lt 13 ] ; then
414	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
415	else
416	(( year = year0 + 1 ))
417	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
418	fi
419	(( i=i+1 ))
420	done
421	;;
422	D\|d)
423	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
424	*)
425	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
426	IGCM_debug_Verif_Exit_Post ;;
427	esac
428	echo ${Length}
429
430	IGCM_debug_PopStack "create_ts_next_date"
431	}
432
433	# Give the end of the actual $1 period from $2 date
434	function create_ts_begin_date {
435	IGCM_debug_PushStack "create_ts_begin_date" $@
436	typeset Length
437	case ${config_UserChoices_PeriodLength} in
438	Y\|y)
439	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
440	IGCM_date_GetYearMonth ${1} year month
441	if [ X${2} = Xend ] ; then
442	(( year = year - PeriodLengthInYears + 1))
443	fi
444	(( Length=0 ))
445	(( i=0 ))
446	until [ $i -ge $PeriodLengthInYears ] ; do
447	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
448	(( i=i+1 ))
449	done
450	;;
451	M\|m)
452	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
453	IGCM_date_GetYearMonth ${1} year month
454	if [ X${2} = Xend ] ; then
455	(( month = month - PeriodLengthInMonths + 1 ))
456	fi
457	(( year0 = year ))
458	if [ $month -le 0 ] ; then
459	(( month = month + 12 ))
460	year=$( printf "%04i\n" $(( year - 1 )) )
461	fi
462	month=$( printf "%02i\n" ${month} )
463	(( Length=0 ))
464	(( i=0 ))
465	until [ $i -ge $PeriodLengthInMonths ] ; do
466	if [ $(( month + i )) -lt 13 ] ; then
467	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
468	else
469	(( year = year0 + 1 ))
470	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
471	fi
472	(( i=i+1 ))
473	done
474	;;
475	D\|d)
476	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
477	*)
478	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
479	IGCM_debug_Verif_Exit_Post ;;
480	esac
481	echo ${Length}
482
483	IGCM_debug_PopStack "create_ts_begin_date"
484	}
485
486	########################################################################
487	#
488	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
489	# IF SO BRING THEM IN THE WORKING DIRECTORY
490	#
491	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
492	# - especially for IDRiS
493	# - no special protection against "command too long" !
494	# - libIGCM_sys should handle that or modify following code part
495	#
496	########################################################################
497
498	if [ ${StandAlone} != true ] ; then
499	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
500	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
501	echo \${run_PostProcessing_TimeSeriesRunning}" )
502
503	#if [ X${Running_Flag} = Xy ] ; then
504	# # Time Series Job allready running
505	# IGCM_debug_Print 1 "Time Series Job allready running exit"
506	# exit
507	#fi
508	# Now run.card know we are running
509
510
511
512
513	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
514
515
516
517
518
519	# Get information from last execution
520	#CompletedFlag=$( IGCM_sys_RshMaster \
521	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
522	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
523	fi
524
525	# Is it the first submission or not ?
526
527	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
528	# First Time Series Submission
529	FIRST_PASS=TRUE
530	Length=$( create_ts_begin_date ${DateBegin} begin)
531	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
532	else
533	# Last Completed Time Series
534	FIRST_PASS=FALSE
535	DATE_FIN_JOB_B=${CompletedFlag}
536	date=${DateBegin}_${DATE_FIN_JOB_B}
537	for comp in ${config_ListOfComponents[*]} ; do
538	#
539	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
540	#
541	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
542	#
543	#set -A liste_file
544	unset liste_file
545	#
546	file_path=${R_SAVE}/${comp}/Analyse
547	#
548	i=0
549	file=${NULL_STR}
550	#
551	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
552	#
553	# Determine in which output can we find file
554	# IE : Analyse/TS or Analyse/DA2TS or ...
555	#
556	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
557	case ${FlagDir} in
558	*Y) TS_Dir=TS_YE ;;
559	*M) TS_Dir=TS_MO ;;
560	*D) TS_Dir=TS_DA ;;
561	HF) TS_Dir=TS_HF ;;
562	INS) TS_Dir=TS_INS ;;
563	esac
564	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
565	#
566	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
567	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
568	#
569	done
570	done
571	if [ X${file} != X${NULL_STR} ] ; then
572	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
573	ExitFlag=false
574	IGCM_debug_Verif_Exit_Post
575	fi
576	done
577	#
578	LEVEL=0
579	#
580	fi
581
582	########################################################################
583	#
584	# DEFINE LOOP PARAMETERS
585	#
586	########################################################################
587
588	DATE_COUNT=${DATE_FIN_JOB_B}
589	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
590
591	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
592
593	if [ ${DoJob} = true ] ; then
594	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
595	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
596	Length=$( create_ts_next_date ${DATE_COUNT} )
597	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
598	done
599	else
600	NBRE_FILE_TOT=0
601	fi
602
603	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
604	NBRE_TOUR=1
605	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
606	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
607	else
608	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
609	fi
610
611	CURRENT_LOOP=1
612	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
613	# WE HAVE 3 LEVELS
614	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
615	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
616	# - LEVEL 3 : WE ARE IN THE LAST LOOP
617
618	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
619
620	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
621	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
622	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
623
624	if [ ${FIRST_PASS} = TRUE ] ; then
625	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
626	else
627	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
628	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
629	fi
630
631	DATE_FIN=${PeriodDateEnd}
632	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
633
634	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
635	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
636
637	LEVEL=1
638	DEPOT="TRUE"
639	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
640	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
641	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
642
643	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
644	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
645	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
646	elif [ -z "${DATE_COURANTE}" ] ; then
647	DATE_COURANTE=${DATE_FIN_JOB_B}
648	else
649	Length=$( create_ts_next_date ${DATE_FIN} )
650	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
651	fi
652
653	(( TotLength = 0 ))
654	COMPTEUR=1
655	DATE_LOOP=${DATE_COURANTE}
656
657	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
658	#
659	Length=$( create_ts_next_date ${DATE_LOOP} )
660	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
661	(( TotLength = TotLength + Length ))
662	(( COMPTEUR = COMPTEUR + 1 ))
663	#
664	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
665	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
666
667	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
668	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
669	DEPOT="FALSE"
670	LEVEL=2
671	else
672	# - LEVEL 3 : WE ARE IN THE LAST LOOP
673	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
674	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
675
676	Length=$( create_ts_next_date ${DATE_FIN} )
677	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
678	DATE_FIN=${PeriodDateEnd}
679	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
680
681	LEVEL=3
682	DEPOT="TRUE"
683	FLAG_B="TRUE"
684	fi
685
686	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
687	for comp in ${config_ListOfComponents[*]} ; do
688	#
689	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
690	#
691	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
692	#
693	eval R_OUT_${comp}=${R_SAVE}/${comp}
694	#
695	i=0
696	#
697	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
698	#
699	# Determine in which output can we find file
700	# IE : Output/MO or Output/DA or ...
701	#
702	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
703	case ${FlagDir} in
704	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
705	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
706	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
707	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
708	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
709	*)
710	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
711	IGCM_sys_Exit "Job create_ts" ;;
712	esac
713
714	# Initialize array
715	#set -A liste_file
716	#set -A liste_file_tmp
717	unset liste_file
718	unset liste_file_tmp
719	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
720
721	COMPTEUR=1
722	DATE_LOOP=${DATE_COURANTE}
723
724	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
725	#
726	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
727	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
728
729	Length2=$( create_ts_next_date ${DATE_LOOP} end)
730	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
731
732	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
733	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
734	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
735	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
736	(( COMPTEUR = COMPTEUR + 1 ))
737	#
738	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
739	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
740
741	# Get selected files
742	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
743	ExitFlag=false
744	IGCM_debug_Verif_Exit_Post
745
746	# Apply IGCM_Patch if needed
747	if [ ! X${RebuildFrequency} = Xtrue ] ; then
748	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
749	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
750	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
751	IGCM_${Patch} ${file_t}
752	done
753	done
754	fi
755	fi
756
757	# WE CAN CONCATENATE FILES AT THIS POINT
758	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
759	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
760	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
761	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
762	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
763
764	liste_coord=" "
765	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
766	liste_coord=${liste_coord}${axis}","
767	done
768
769	# Time axis must be the last dimension
770	time_axis=${axis}
771
772	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
773	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
774	#
775	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
776	#
777	# # Change time axis name to be CF compliant
778	# if [ ${time_axis} != time_counter ]; then
779	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
780	# fi
781	#
782	if [ "${FLAG_B}" = "TRUE" ] ; then
783	# WE CONCATENATE WITH EXISTING FILES
784	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
785	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
786	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
787	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
788	#
789	IGCM_sys_Rm ${file1} ${file2}
790	fi
791	#
792	if [ "${DEPOT}" = "TRUE" ] ; then
793	# WE PUT FINAL FILE ON FILE SYSTEM
794	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
795	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
796	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
797	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
798	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
799	IGCM_sys_Rm ${file_out}
800	#
801	fi
802	#
803	ExitFlag=false
804	IGCM_debug_Verif_Exit_Post
805	done # for var in ${liste_var} ; do
806	IGCM_sys_Rm ${liste_file_tmp[*]}
807	done # for file in ${liste_file_brut} do
808	done # for comp in ${config_ListOfComponents[*]} ; do
809	ExitFlag=false
810	IGCM_debug_Verif_Exit_Post
811	# GO AHEAD IN DEEPEST LOOP
812	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
813	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
814	# FINISH
815
816	# DODS copy
817	for comp in ${config_ListOfComponents[*]} ; do
818	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
819	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
820	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
821	done
822	done
823
824	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
825	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
826
827	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
828	export listVarEnv
829
830	#D-
831	# --------------------------------------------------------------------
832	#D- Test if all was right
833	# --------------------------------------------------------------------
834	IGCM_debug_Verif_Exit_Post
835	# --------------------------------------------------------------------
836	#D- Submit MONITORING
837	# --------------------------------------------------------------------
838	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
839
840	#D-
841	# --------------------------------------------------------------------
842	#D- Update the run.card
843	# --------------------------------------------------------------------
844	if [ ${StandAlone} != true ] ; then
845	# Put in run.card end period of time series
846	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
847	# We have finish the job !
848	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
849	fi
850
851	# Clean RUN_DIR_PATH (necessary for cesium)
852	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
853
854	date

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/AA_create_ts @ 222

Download in other formats: