Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh @ 1186

Last change on this file since 1186 was 1186, checked in by sdipsl, 9 years ago
Ada bugfix. House keeping side effect when calling scipt interactively (clean_month, TimeSeriesChecker?, ...)
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 35.3 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for obelix
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=${MASTER:=$( hostname )}
71
72	#D-
73	#D-#==================================================
74	#D-Program used in libIGCM
75	#D-#==================================================
76
77	# Submit command
78	typeset SUBMIT=${SUBMIT:=qsub}
79	# rsync with path
80	typeset -r RSYNC=/usr/bin/rsync
81	# RSYNC_opt args to rsync
82	typeset -r RSYNC_opt="-va"
83	# ie storage filesystem
84	typeset -r STOREHOST=${MASTER}
85
86	#====================================================
87	# Set environment tools (ferret, nco, cdo, rebuild, ...)
88	#====================================================
89	. /home/users/igcmg/.atlas_env_asterix_ksh
90	export PATH=${PATH}:/home/users/igcmg/rebuild/bin/
91
92	#====================================================
93	# Set lf95 environment
94	#====================================================
95	. /usr/local/install/lf6481/bash_laheyfort_setup
96
97	#====================================================
98	# Host specific DIRECTORIES
99	#====================================================
100
101	#====================================================
102	#- MirrorlibIGCM for frontend
103	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
104
105	#====================================================
106	#- libIGCM_POST for frontend
107	typeset -r libIGCM_POST=${libIGCM}
108
109	#====================================================
110	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
111	typeset -r R_EXE="${MODIPSL}/bin"
112
113	#====================================================
114	#- SUBMIT_DIR : submission dir
115	if [ X${PBS_O_WORKDIR} != X ] ; then
116	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
117	else
118	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
119	fi
120
121	#====================================================
122	#- IN
123	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
124	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/home/orchideeshare/igcmg/IGCM}
125
126	#====================================================
127	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
128	if [ X${PBS_JOBID} != X ] ; then
129	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/scratch/$PBS_O_LOGNAME.$PBS_JOBID}
130	else
131	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/tmp/tmp$$}
132	fi
133
134	#====================================================
135	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
136	typeset -r OUTCOMMAND_PATH=/tmp
137
138	#====================================================
139	#- HOST_MPIRUN_COMMAND
140	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
141
142	#====================================================
143	#- Max number of arguments passed to nco operator or demigration command
144	UNIX_MAX_LIMIT=360
145
146	#====================================================
147	#- set PackDefault to false on obelix
148	PackDefault=false
149
150	#====================================================
151	#- Number of core per node (max number of OpenMP task)
152	NUM_COREPERNODE=1
153
154	#====================================================
155	#- Default number of MPI task for IPSL coupled model
156	#- required for backward compatibility
157	#-
158	DEFAULT_NUM_PROC_OCE=1
159	DEFAULT_NUM_PROC_CPL=1
160	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
161	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
162
163	#D-#==================================================
164	#D-function IGCM_sys_defineArchives
165	#D-* Purpose:
166	#D-* Define ARCHIVE : Dedicated to large files
167	#D-* Define STORAGE : Dedicated to small/medium files
168	#D-* Define R_OUT : Output tree located on ARCHIVE
169	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
170	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
171	#D-* if SpaceName=TEST nothing special will hapen
172	#D-* Examples:
173	#D-
174	function IGCM_sys_defineArchives {
175	IGCM_debug_PushStack "IGCM_sys_defineArchives"
176
177	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
178	#====================================================
179	#- ARCHIVE (dedicated to large files)
180	ARCHIVE=${config_UserChoices_ARCHIVE}
181	else
182	#====================================================
183	#- ARCHIVE (dedicated to large files)
184	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
185	fi
186
187	if [ ! X${config_UserChoices_STORAGE} = X ]; then
188	#====================================================
189	#- STORAGE (dedicated to small/medium files)
190	STORAGE=${config_UserChoices_STORAGE}
191	else
192	#====================================================
193	#- STORAGE (dedicated to small/medium files)
194	STORAGE=${ARCHIVE}
195	fi
196
197	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
198
199	#====================================================
200	#- R_OUT
201	R_OUT=${ARCHIVE}/IGCM_OUT
202
203	#====================================================
204	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
205	R_FIG=${STORAGE}/IGCM_OUT
206
207	#====================================================
208	#- R_BUF (ONLY FOR double copy an scratch)
209	R_BUF=${STORAGE}/IGCM_OUT
210
211	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
212	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
213	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
214
215	IGCM_debug_PopStack "IGCM_sys_defineArchives"
216	}
217
218	#D-#==================================================
219	#D-function IGCM_sys_RshArchive
220	#D-* Purpose: Archive rsh command
221	#D-* Examples:
222	#D-
223	function IGCM_sys_RshArchive {
224	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
225	/bin/ksh <<-EOF
226	${@}
227	EOF
228	status=$?
229	if [ ${status} -gt 0 ] ; then
230	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
231	IGCM_debug_Exit "IGCM_sys_RshArchive"
232	fi
233	IGCM_debug_PopStack "IGCM_sys_RshArchive"
234	}
235
236	#D-#==================================================
237	#D-function IGCM_sys_RshArchive_NoError
238	#D-* Purpose: Archive rsh command, without error
239	#D-* used only in monitoring.job
240	#D-* Examples:
241	#D-
242	function IGCM_sys_RshArchive_NoError {
243	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
244	/bin/ksh <<-EOF
245	${@} 2> /dev/null
246	EOF
247	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
248	}
249
250	#D-#==================================================
251	#D-function IGCM_sys_MkdirArchive
252	#D-* Purpose: Mkdir on Archive
253	#D-* Examples:
254	#D-
255	function IGCM_sys_MkdirArchive {
256	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
257	if ( $DEBUG_sys ) ; then
258	echo "IGCM_sys_MkdirArchive :" $@
259	fi
260	#- creation de repertoire sur le serveur fichier
261	if [ ! -d ${1} ]; then
262	\mkdir -p $1
263	status=$?
264
265	if [ ${status} -gt 0 ] ; then
266	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
267	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
268	fi
269	fi
270	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
271	}
272
273	#D-#==================================================
274	#D-function IGCM_sys_TestDirArchive
275	#D-* Purpose: Test Directory that must exists on Archive
276	#D-* Examples:
277	#D-
278	function IGCM_sys_TestDirArchive {
279	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
280	if ( $DEBUG_sys ) ; then
281	echo "IGCM_sys_TestDirArchive :" $@
282	fi
283	typeset ExistFlag
284	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
285	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
286	return ${ExistFlag}
287	}
288
289	#D-#==================================================
290	#D-function IGCM_sys_IsFileArchived
291	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
292	#D-* Examples:
293	#D-
294	function IGCM_sys_IsFileArchived {
295	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
296	if ( $DEBUG_sys ) ; then
297	echo "IGCM_sys_IsFileArchived :" $@
298	fi
299	typeset IsArchivedFlag
300	# Never archived for this system
301	IsArchivedFlag=1
302	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
303
304	return ${IsArchivedFlag}
305	}
306
307	#D-#==================================================
308	#D-function IGCM_sys_TestFileArchive
309	#D-* Purpose: Test file that must NOT EXISTS on Archive
310	#D-* Examples:
311	#D-
312	function IGCM_sys_TestFileArchive {
313	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
314	typeset ExistFlag
315	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
316	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
317
318	return ${ExistFlag}
319	}
320
321	#D-#==================================================
322	#D-function IGCM_sys_CountFileArchive
323	#D-* Purpose: Count files on Archive filesystem
324	#D-* Examples:
325	#D-
326	function IGCM_sys_CountFileArchive {
327	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
328	ls ${@} 2>/dev/null \| wc -l
329	if [ $? -gt 0 ] ; then
330	echo "IGCM_sys_CountFileArchive : erreur."
331	fi
332	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
333	}
334
335	#D-#==================================================
336	#D-function IGCM_sys_Tree
337	#D-* Purpose: Tree directories with files on ${ARCHIVE}
338	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
339	#D-
340	function IGCM_sys_Tree {
341	IGCM_debug_PushStack "IGCM_sys_Tree" $@
342	if ( $DEBUG_sys ) ; then
343	echo "IGCM_sys_Tree :" $@
344	fi
345
346	\tree -f $@
347
348	IGCM_debug_PopStack "IGCM_sys_Tree"
349	}
350
351	#D-#==================================================
352	#D-function IGCM_sys_Qsub
353	#D-* Purpose: Qsub new job
354	#D-* Examples:
355	#D-
356	function IGCM_sys_Qsub {
357	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
358	if ( $DEBUG_sys ) ; then
359	echo "IGCM_sys_Qsub :" $@
360	fi
361	typeset options status
362	options="-o ${SUBMIT_DIR}/${Script_Output}"
363	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
364	status=$?
365
366	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
367	if [ ${status} -gt 0 ] ; then
368	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
369	IGCM_debug_Exit "IGCM_sys_Qsub"
370	else
371	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
372	fi
373	IGCM_debug_PopStack "IGCM_sys_Qsub"
374	}
375
376	#D-#==================================================
377	#D-function IGCM_sys_QsubPost
378	#D-* Purpose: Qsub new job on scalaire
379	#D-* Examples:
380	#D-
381	function IGCM_sys_QsubPost {
382	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
383	if ( $DEBUG_sys ) ; then
384	echo "IGCM_sys_QsubPost :" $@
385	fi
386	typeset options status
387	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
388	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
389	status=$?
390
391	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
392	if [ ${status} -gt 0 ] ; then
393	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
394	IGCM_debug_Exit "IGCM_sys_QsubPost"
395	else
396	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
397	fi
398	IGCM_debug_PopStack "IGCM_sys_QsubPost"
399	}
400
401	#D-*************************
402	#D- File transfer functions
403	#D-*************************
404	#D-
405
406	#D-#==================================================
407	#D-function IGCM_sys_RmRunDir
408	#D-* Purpose: rm tmpdir (dummy function most of the time batch
409	#D- scheduler will do the job)
410	#D-* Examples:
411	#D-
412	function IGCM_sys_RmRunDir {
413	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
414	if ( $DEBUG_sys ) ; then
415	echo "IGCM_sys_RmRunDir :" $@
416	echo "Dummy call, let the scheduler do that."
417	fi
418	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
419	}
420
421	#D-#==================================================
422	#D-function IGCM_sys_Put_Dir
423	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
424	#D-* Examples:
425	#D-
426	function IGCM_sys_Put_Dir {
427	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
428	if ( $DEBUG_sys ) ; then
429	echo "IGCM_sys_Put_Dir :" $@
430	fi
431	if [ $DRYRUN = 0 ]; then
432	if [ ! -d ${1} ] ; then
433	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
434	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
435	return
436	fi
437
438	typeset status
439
440	# Only if we use rsync
441	#IGCM_sys_TestDirArchive $( dirname $2 )
442	#
443	#USUAL WAY
444	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
445	status=$?
446
447	if [ ${status} -gt 0 ] ; then
448	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
449	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
450	IGCM_debug_Exit "IGCM_sys_Put_Dir"
451	else
452	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
453	fi
454	fi
455	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
456	}
457
458	#D-#==================================================
459	#D-function IGCM_sys_Get_Dir
460	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
461	#D-* Examples:
462	#D-
463	function IGCM_sys_Get_Dir {
464	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
465	if ( $DEBUG_sys ) ; then
466	echo "IGCM_sys_Get_Dir :" $@
467	fi
468	if [ $DRYRUN = 0 ]; then
469	typeset NB_ESSAI DELAI status i
470	# number of tentative
471	NB_ESSAI=3
472	# time delay between tentative
473	DELAI=2
474
475	#
476	# USUAL WAY
477	i=0
478	while [ $i -lt $NB_ESSAI ] ; do
479	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
480	status=$?
481	if [ ${status} -gt 0 ] ; then
482	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
483	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
484	sleep $DELAI
485	else
486	break
487	fi
488	(( i = i + 1 ))
489	done
490
491	if [ ${status} -gt 0 ] ; then
492	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
493	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
494	IGCM_debug_Exit "IGCM_sys_Get_Dir"
495	else
496	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
497	fi
498	fi
499	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
500	}
501
502	#D-#==================================================
503	#D-function IGCM_sys_Put_Rest
504	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
505	#D- File and target directory must exist.
506	#D-* Examples:
507	#D-
508	function IGCM_sys_Put_Rest {
509	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
510	if ( $DEBUG_sys ) ; then
511	echo "IGCM_sys_Put_Rest :" $@
512	fi
513	if [ $DRYRUN = 0 ]; then
514	if [ ! -f ${1} ] ; then
515	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
516	IGCM_debug_Exit "IGCM_sys_Put_Rest"
517	fi
518
519	typeset status
520	#
521	# USUAL WAY
522	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
523	status=$?
524
525	# #RSYNC WITH NETWORK SSH CALL
526	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
527	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
528
529	# #RSYNC WITH NFS USE
530	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
531	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
532
533	# status=$?
534	# IGCM_sys_Rsync_out $status
535
536	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
537	# (( status=status+$? ))
538
539	if [ ${status} -gt 0 ] ; then
540	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
541	[ -f ${1} ] && ls -l ${1}
542	[ -f ${2} ] && ls -l ${2}
543	[ -f ${2}/${1} ] && ls -l ${2}/${1}
544	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
545	IGCM_debug_Exit "IGCM_sys_Put_Rest"
546	else
547
548	if [ X${JobType} = XRUN ] ; then
549	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
550	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
551	fi
552
553	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
554	fi
555	fi
556	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
557	}
558
559	#D-#==================================================
560	#D-function IGCM_sys_Put_Out
561	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
562	#D-* Examples:
563	#D-
564	function IGCM_sys_Put_Out {
565	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
566	if ( $DEBUG_sys ) ; then
567	echo "IGCM_sys_Put_Out :" $@
568	fi
569
570	typeset status
571
572	if [ $DRYRUN = 0 ]; then
573	if [ ! -f ${1} ] ; then
574	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
575	IGCM_debug_PopStack "IGCM_sys_Put_Out"
576	return 1
577	fi
578	#
579	IGCM_sys_MkdirArchive $( dirname $2 )
580	#
581	if [ X${JobType} = XRUN ] ; then
582	if [ X${3} = X ] ; then
583	IGCM_sys_Chmod 444 ${1}
584	fi
585	fi
586	#
587
588	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
589	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
590	status=$?
591	IGCM_sys_Rsync_out $status
592
593	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
594	(( status=status+$? ))
595
596	if [ ${status} -gt 0 ] ; then
597	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
598	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
599	IGCM_debug_Exit "IGCM_sys_Put_Out"
600	else
601	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
602	fi
603	fi
604	IGCM_debug_PopStack "IGCM_sys_Put_Out"
605	return 0
606	}
607
608	#D-#==================================================
609	#D-function IGCM_sys_Get
610	#D-* Purpose: Get a file from ${ARCHIVE}
611	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
612	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
613	function IGCM_sys_Get {
614	IGCM_debug_PushStack "IGCM_sys_Get" $@
615
616	typeset DEST status dm_liste
617
618	if ( $DEBUG_sys ) ; then
619	echo "IGCM_sys_Get :" $@
620	fi
621	if [ $DRYRUN -le 2 ]; then
622	if [ X${1} = X'/l' ] ; then
623	eval set +A dm_liste \${${2}}
624	else
625	dm_liste=${1}
626	fi
627	eval DEST=\${${#}}
628
629	# test if the (first) file is present in the old computation :
630	IGCM_sys_TestFileArchive ${dm_liste[0]}
631	status=$?
632	if [ ${status} -gt 0 ] ; then
633	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
634	IGCM_debug_Exit "IGCM_sys_Get"
635	return
636	fi
637
638	#USUAL WAY
639	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
640	status=$?
641
642	if [ ${status} -gt 0 ] ; then
643	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
644	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
645	IGCM_debug_Exit "IGCM_sys_Get"
646	else
647	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
648	fi
649	fi
650	IGCM_debug_PopStack "IGCM_sys_Get"
651	}
652
653	#D-#==================================================
654	#D-function IGCM_sys_GetDate_Monitoring
655	#D-* Purpose: get the last year for which the monitoring has been computed
656	#D-* Examples:
657	#D-
658	function IGCM_sys_GetDate_Monitoring {
659	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
660	if ( $DEBUG_sys ) ; then
661	echo "IGCM_sys_GetDate_Monitoring :" $@
662	fi
663
664	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
665
666	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
667	}
668
669	#D-#==================================================
670	#D-function IGCM_sys_Dods_Rm
671	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
672	#D-* Examples:
673	#D-
674	function IGCM_sys_Dods_Rm {
675	if ( $DEBUG_sys ) ; then
676	echo "IGCM_sys_Dods_Rm :" $@
677	fi
678	return 0
679	}
680
681	#D-#==================================================
682	#D-function IGCM_sys_Dods_Cp
683	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
684	#D-* Examples:
685	#D-
686	function IGCM_sys_Dods_Cp {
687	if ( $DEBUG_sys ) ; then
688	echo "IGCM_sys_Dods_Cp :" $@
689	fi
690	return 0
691	}
692
693	#D-#==================================================
694	#D-function IGCM_sys_Put_Dods
695	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
696	#D-* Examples:
697	#D-
698	function IGCM_sys_Put_Dods {
699	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
700	if ( $DEBUG_sys ) ; then
701	echo "IGCM_sys_Put_Dods :" $@
702	fi
703	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
704	}
705
706	##############################################################
707	# REBUILD OPERATOR
708
709	#D-#==================================================
710	#D-function IGCM_sys_sync
711	#D-* Purpose: flush buffer on disk (dummy function on Ada)
712	#D-* Examples:
713	#D-
714	function IGCM_sys_sync {
715	IGCM_debug_PushStack "IGCM_sys_sync" $@
716	if ( $DEBUG_sys ) ; then
717	echo "IGCM_sys_sync :" $@
718	echo "Dummy call, let the system do that."
719	fi
720	IGCM_debug_PopStack "IGCM_sys_sync"
721	}
722
723	############################################################
724	# Activate Running Environnment Variables
725
726	#D-#==================================================
727	#D-function IGCM_sys_desactiv_variables
728	#D-* Purpose: set environement variables prior to execution
729	#D-* Examples:
730	#D-
731	function IGCM_sys_activ_variables {
732	IGCM_debug_PushStack "IGCM_sys_activ_variables"
733	if ( $DEBUG_sys ) ; then
734	echo "IGCM_sys_activ_variables"
735	fi
736
737	# --------------------------------------------------------------------
738	#D- MPI specifications
739	# --------------------------------------------------------------------
740
741	# --------------------------------------------------------------------
742	#D- Other specifications
743	# --------------------------------------------------------------------
744
745	IGCM_debug_PopStack "IGCM_sys_activ_variables"
746	}
747
748	############################################################
749	# Desactivate Running Environnment Variables
750
751	#D-#==================================================
752	#D-function IGCM_sys_desactiv_variables
753	#D-* Purpose: unset environement variables after execution
754	#D-* Examples:
755	#D-
756	function IGCM_sys_desactiv_variables {
757	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
758	if ( $DEBUG_sys ) ; then
759	echo "IGCM_sys_desactiv_variables"
760	fi
761	# --------------------------------------------------------------------
762	#D- MPI specifications
763	# --------------------------------------------------------------------
764
765	# --------------------------------------------------------------------
766	#D- Other specifications
767	# --------------------------------------------------------------------
768
769	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
770	}
771
772	############################################################
773	# Build MPI/OMP scripts run file (dummy function)
774
775	#D-#==================================================
776	#D-function IGCM_sys_build_run_file
777	#D-* Purpose: build run file (deprecated)
778	#D-* Examples:
779	#D-
780	function IGCM_sys_build_run_file {
781
782	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
783
784	}
785
786	############################################################
787	# Build MPI/OMP scripts
788
789	#D-#==================================================
790	#D-function IGCM_sys_build_execution_scripts
791	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
792	#D-* Examples:
793	#D-
794	function IGCM_sys_build_execution_scripts
795	{
796	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
797	if ( $DEBUG_sys ) ; then
798	echo "IGCM_sys_build_execution_scripts " $@
799	fi
800
801	typeset nodes listnodes init_node start_num init_exec comp ExeNameIn ExeNameOut
802	typeset node_num_current node_current comp_proc_mpi_loc comp_proc_omp_loc
803	typeset num_corempi nombre_restant_node nombre_restant_comp
804
805	if ( ${OK_PARA_MPMD} ) ; then
806
807	if [ -f run_file ] ; then
808	IGCM_sys_Rm -f run_file
809	fi
810	touch run_file
811
812	if ( ${OK_PARA_OMP} ) ; then
813
814	# Hosts treatment
815
816	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
817
818	i=0
819	rm -f hosts
820	IGCM_debug_Print 1 "sys Obelix, Hosts avaible :"
821	for nodes in `cat hosts.tmp` ; do
822	host[$i]=$nodes
823	echo "${host[$i]} slots=1 max_slots=1" >> hosts
824	IGCM_debug_Print 1 ${host[$i]}
825	i=$((i+1))
826	done
827	rm -f hosts.tmp
828
829	listnodes=${host[*]}
830
831	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
832
833	# Initialisation
834
835	init_node=y
836	node_num_current=0
837	start_num=0
838	init_exec=n
839
840	# Build run_file
841
842	# First loop on the components for the coupler ie oasis (only if oasis3)
843	# the coupler ie oasis3 must be the first one
844	for comp in ${config_ListOfComponents[*]} ; do
845
846	if [ "X${comp}" = "XCPL" ] ; then
847
848	eval ExeNameIn=\${config_Executable_${comp}[0]}
849	eval ExeNameOut=\${config_Executable_${comp}[1]}
850
851	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
852	echo "" >> script_${ExeNameOut}.ksh
853	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
854	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
855	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
856	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
857	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
858
859	init_node=n
860
861	(( nombre_restant_node = NUM_COREPERNODE - 1 ))
862	node_num_current=0
863	node_current=${host[${node_num_current}]}
864
865	EXECUTION="${EXECUTION} -H ${node_current} -np 1 ./script_${ExeNameOut}.ksh"
866
867	init_exec=y
868	start_num=1
869
870	fi
871
872	done
873
874	# Then loop on the components (except for oasis)
875
876	for comp in ${config_ListOfComponents[*]} ; do
877
878	eval ExeNameIn=\${config_Executable_${comp}[0]}
879	eval ExeNameOut=\${config_Executable_${comp}[1]}
880
881	# Only if we really have an executable for the component :
882	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
883
884	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
885	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
886
887	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
888	# echo "set -vx" >> script_${ExeNameOut}.ksh
889	echo "" >> script_${ExeNameOut}.ksh
890	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
891	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
892	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
893	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
894	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
895	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
896	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
897
898	node_num=0
899
900	# We define the number of MPI process to be assigned for the component
901
902	nombre_restant_comp=${comp_proc_mpi_loc}
903
904	# Loop on the allocated nodes
905
906	for node in ${listnodes} ; do
907
908	# We go to the current node
909	if [ ${node_num} = ${node_num_current} ] ; then
910
911	node_current=${host[${node_num_current}]}
912
913	# If first time on the node : initialisation
914
915	if [ ${init_node} = y ] ; then
916	nombre_restant_node=${NUM_COREPERNODE}
917	fi
918
919	# Test on the number of OMP threads
920
921	if [ ${comp_proc_omp_loc} -gt ${nombre_restant_node} ] ; then
922	(( node_num = node_num + 1 ))
923	node_num_current=${node_num}
924	init_node=y
925	continue
926	fi
927
928	# Number of MPI process to assign
929
930	(( num_corempi = nombre_restant_node / comp_proc_omp_loc ))
931
932	if [ ${num_corempi} -gt ${nombre_restant_comp} ] ; then
933	num_corempi=${nombre_restant_comp}
934	fi
935
936	(( nombre_restant_node = nombre_restant_node - num_corempi * comp_proc_omp_loc ))
937	(( nombre_restant_comp = nombre_restant_comp - num_corempi ))
938
939	if [ ${init_exec} = y ] ; then
940	EXECUTION="${EXECUTION} : -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
941	else
942	EXECUTION="${EXECUTION} -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
943	init_exec=y
944	fi
945
946	(( start_num = num_corempi + start_num ))
947
948	else
949
950	(( node_num = node_num + 1 ))
951	continue
952	fi
953
954	# Test on the number of core/process remaining on the node/component
955
956	if [ ${nombre_restant_node} = 0 ] ; then
957	(( node_num = node_num + 1 ))
958	node_num_current=${node_num}
959	init_node=y
960
961	if [ ${nombre_restant_comp} = 0 ] ; then
962	break 1
963	fi
964	else
965
966	node_num_current=${node_num}
967	init_node=n
968
969	if [ ${nombre_restant_comp} = 0 ] ; then
970	break 1
971	fi
972	fi
973	done
974	fi
975	done
976
977	else
978
979	# Then first loop on the components for the coupler ie oasis
980
981	## the coupler ie oasis must be the first one
982	for comp in ${config_ListOfComponents[*]} ; do
983
984	eval ExeNameOut=\${config_Executable_${comp}[1]}
985
986	# for CPL component only
987	if [ "X${comp}" = "XCPL" ] ; then
988	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
989	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
990	fi
991	done
992
993	# Then second loop on the components
994
995	for comp in ${config_ListOfComponents[*]} ; do
996
997	eval ExeNameOut=\${config_Executable_${comp}[1]}
998
999	# Only if we really have an executable for the component and not the coupler ie oasis:
1000	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1001	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1002	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1003	fi
1004	done
1005	IGCM_sys_Chmod u+x run_file
1006
1007	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
1008
1009	fi
1010
1011	else # Only one executable. launch it.
1012
1013	for comp in ${config_ListOfComponents[*]} ; do
1014
1015	# Only if we really have an executable for the component :
1016	eval ExeNameOut=\${config_Executable_${comp}[1]}
1017	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1018
1019	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1020	echo "" >> script_${ExeNameOut}.ksh
1021	if ( ${OK_PARA_OMP} ) ; then
1022	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1023	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1024	fi
1025	if ( ${OK_PARA_MPI} ) ; then
1026	# Default : mpirun used if nb_proc gt 1
1027	# pour sortie out/err par process
1028	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
1029	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1030	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1031	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1032	else
1033	# Default : mpirun is NOT used if nb_proc eq 1
1034	# pour sortie out/err par process
1035	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1036	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1037	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1038	EXECUTION="time ./script_${ExeNameOut}.ksh"
1039	fi
1040
1041	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
1042	cat script_${ExeNameOut}.ksh
1043
1044	fi
1045
1046	done
1047
1048	fi
1049
1050	IGCM_debug_Print 1 "sys Obelix : execution command is"
1051	IGCM_debug_Print 1 "$EXECUTION"
1052
1053	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1054	}
1055
1056	#D-#==================================================
1057	#D-function IGCM_sys_check_path
1058	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1059	#D-* do not point to an important use directory. Stop immediately in that case.
1060	#D-* Examples:
1061	#D-
1062	function IGCM_sys_check_path {
1063	IGCM_debug_PushStack "IGCM_sys_check_path"
1064	if ( $DEBUG_sys ) ; then
1065	echo "IGCM_sys_check_path"
1066	fi
1067
1068	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1069	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1070	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1071	IGCM_debug_Exit "This will stop the job"
1072	fi
1073	IGCM_debug_PopStack "IGCM_sys_check_path"
1074	}
1075
1076	#D-#==================================================
1077	#D-function IGCM_sys_check_quota. Dummy call here
1078	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1079	#D-* Examples:
1080	#D-
1081	function IGCM_sys_check_quota {
1082	IGCM_debug_PushStack "IGCM_sys_check_quota"
1083	if ( $DEBUG_sys ) ; then
1084	echo "IGCM_sys_check_quota"
1085	fi
1086	IGCM_debug_PopStack "IGCM_sys_check_quota"
1087	}
1088
1089	#D-#==================================================
1090	#D-function IGCM_sys_GetJobID
1091	#D-* Purpose: Check if job_name is currently
1092	#D- running or in queue
1093	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1094	#D-
1095	function IGCM_sys_GetJobID {
1096	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1097	if ( $DEBUG_sys ) ; then
1098	echo "IGCM_sys_GetJobID"
1099	fi
1100
1101	# With -f option, the full job name is given in the last column
1102	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
1103
1104	eval ${3}=${ID}
1105	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1106	}
1107
1108	#D-#==================================================
1109	#D-function IGCM_sys_CountJobInQueue
1110	#D-* Purpose: Check if job_name is currently
1111	#D- running or in queue
1112	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1113	#D-
1114	function IGCM_sys_CountJobInQueue {
1115	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1116	if ( $DEBUG_sys ) ; then
1117	echo "IGCM_sys_CountJobInQueue"
1118	fi
1119	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1120	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: