Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh @ 1204

Last change on this file since 1204 was 1201, checked in by sdipsl, 9 years ago
add jobWarningDelay in seconds to supervision messages see #253 (threshold for heartbeat) add accountingProject to supervision messages see #255 (may ease gencmip6 data production tracking)
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 35.4 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for obelix
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=${MASTER:=$( hostname )}
71	# project name
72	typeset PROJECT=NONE
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=${PBS_WALLTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=qsub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89
90	#====================================================
91	# Set environment tools (ferret, nco, cdo, rebuild, ...)
92	#====================================================
93	. /home/users/igcmg/.atlas_env_asterix_ksh
94	export PATH=${PATH}:/home/users/igcmg/rebuild/bin/
95
96	#====================================================
97	# Set lf95 environment
98	#====================================================
99	. /usr/local/install/lf6481/bash_laheyfort_setup
100
101	#====================================================
102	# Host specific DIRECTORIES
103	#====================================================
104
105	#====================================================
106	#- MirrorlibIGCM for frontend
107	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
108
109	#====================================================
110	#- libIGCM_POST for frontend
111	typeset -r libIGCM_POST=${libIGCM}
112
113	#====================================================
114	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
115	typeset -r R_EXE="${MODIPSL}/bin"
116
117	#====================================================
118	#- SUBMIT_DIR : submission dir
119	if [ X${PBS_O_WORKDIR} != X ] ; then
120	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
121	else
122	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
123	fi
124
125	#====================================================
126	#- IN
127	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
128	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/home/orchideeshare/igcmg/IGCM}
129
130	#====================================================
131	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
132	if [ X${PBS_JOBID} != X ] ; then
133	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/scratch/$PBS_O_LOGNAME.$PBS_JOBID}
134	else
135	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/tmp/tmp$$}
136	fi
137
138	#====================================================
139	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
140	typeset -r OUTCOMMAND_PATH=/tmp
141
142	#====================================================
143	#- HOST_MPIRUN_COMMAND
144	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
145
146	#====================================================
147	#- Max number of arguments passed to nco operator or demigration command
148	UNIX_MAX_LIMIT=360
149
150	#====================================================
151	#- set PackDefault to false on obelix
152	PackDefault=false
153
154	#====================================================
155	#- Number of core per node (max number of OpenMP task)
156	NUM_COREPERNODE=1
157
158	#====================================================
159	#- Default number of MPI task for IPSL coupled model
160	#- required for backward compatibility
161	#-
162	DEFAULT_NUM_PROC_OCE=1
163	DEFAULT_NUM_PROC_CPL=1
164	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
165	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
166
167	#D-#==================================================
168	#D-function IGCM_sys_defineArchives
169	#D-* Purpose:
170	#D-* Define ARCHIVE : Dedicated to large files
171	#D-* Define STORAGE : Dedicated to small/medium files
172	#D-* Define R_OUT : Output tree located on ARCHIVE
173	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
174	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
175	#D-* if SpaceName=TEST nothing special will hapen
176	#D-* Examples:
177	#D-
178	function IGCM_sys_defineArchives {
179	IGCM_debug_PushStack "IGCM_sys_defineArchives"
180
181	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
182	#====================================================
183	#- ARCHIVE (dedicated to large files)
184	ARCHIVE=${config_UserChoices_ARCHIVE}
185	else
186	#====================================================
187	#- ARCHIVE (dedicated to large files)
188	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
189	fi
190
191	if [ ! X${config_UserChoices_STORAGE} = X ]; then
192	#====================================================
193	#- STORAGE (dedicated to small/medium files)
194	STORAGE=${config_UserChoices_STORAGE}
195	else
196	#====================================================
197	#- STORAGE (dedicated to small/medium files)
198	STORAGE=${ARCHIVE}
199	fi
200
201	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
202
203	#====================================================
204	#- R_OUT
205	R_OUT=${ARCHIVE}/IGCM_OUT
206
207	#====================================================
208	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
209	R_FIG=${STORAGE}/IGCM_OUT
210
211	#====================================================
212	#- R_BUF (ONLY FOR double copy an scratch)
213	R_BUF=${STORAGE}/IGCM_OUT
214
215	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
216	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
217	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
218
219	IGCM_debug_PopStack "IGCM_sys_defineArchives"
220	}
221
222	#D-#==================================================
223	#D-function IGCM_sys_RshArchive
224	#D-* Purpose: Archive rsh command
225	#D-* Examples:
226	#D-
227	function IGCM_sys_RshArchive {
228	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
229	/bin/ksh <<-EOF
230	${@}
231	EOF
232	status=$?
233	if [ ${status} -gt 0 ] ; then
234	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
235	IGCM_debug_Exit "IGCM_sys_RshArchive"
236	fi
237	IGCM_debug_PopStack "IGCM_sys_RshArchive"
238	}
239
240	#D-#==================================================
241	#D-function IGCM_sys_RshArchive_NoError
242	#D-* Purpose: Archive rsh command, without error
243	#D-* used only in monitoring.job
244	#D-* Examples:
245	#D-
246	function IGCM_sys_RshArchive_NoError {
247	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
248	/bin/ksh <<-EOF
249	${@} 2> /dev/null
250	EOF
251	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
252	}
253
254	#D-#==================================================
255	#D-function IGCM_sys_MkdirArchive
256	#D-* Purpose: Mkdir on Archive
257	#D-* Examples:
258	#D-
259	function IGCM_sys_MkdirArchive {
260	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
261	if ( $DEBUG_sys ) ; then
262	echo "IGCM_sys_MkdirArchive :" $@
263	fi
264	#- creation de repertoire sur le serveur fichier
265	if [ ! -d ${1} ]; then
266	\mkdir -p $1
267	status=$?
268
269	if [ ${status} -gt 0 ] ; then
270	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
271	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
272	fi
273	fi
274	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
275	}
276
277	#D-#==================================================
278	#D-function IGCM_sys_TestDirArchive
279	#D-* Purpose: Test Directory that must exists on Archive
280	#D-* Examples:
281	#D-
282	function IGCM_sys_TestDirArchive {
283	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
284	if ( $DEBUG_sys ) ; then
285	echo "IGCM_sys_TestDirArchive :" $@
286	fi
287	typeset ExistFlag
288	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
289	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
290	return ${ExistFlag}
291	}
292
293	#D-#==================================================
294	#D-function IGCM_sys_IsFileArchived
295	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
296	#D-* Examples:
297	#D-
298	function IGCM_sys_IsFileArchived {
299	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
300	if ( $DEBUG_sys ) ; then
301	echo "IGCM_sys_IsFileArchived :" $@
302	fi
303	typeset IsArchivedFlag
304	# Never archived for this system
305	IsArchivedFlag=1
306	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
307
308	return ${IsArchivedFlag}
309	}
310
311	#D-#==================================================
312	#D-function IGCM_sys_TestFileArchive
313	#D-* Purpose: Test file that must NOT EXISTS on Archive
314	#D-* Examples:
315	#D-
316	function IGCM_sys_TestFileArchive {
317	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
318	typeset ExistFlag
319	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
320	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
321
322	return ${ExistFlag}
323	}
324
325	#D-#==================================================
326	#D-function IGCM_sys_CountFileArchive
327	#D-* Purpose: Count files on Archive filesystem
328	#D-* Examples:
329	#D-
330	function IGCM_sys_CountFileArchive {
331	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
332	ls ${@} 2>/dev/null \| wc -l
333	if [ $? -gt 0 ] ; then
334	echo "IGCM_sys_CountFileArchive : erreur."
335	fi
336	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
337	}
338
339	#D-#==================================================
340	#D-function IGCM_sys_Tree
341	#D-* Purpose: Tree directories with files on ${ARCHIVE}
342	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
343	#D-
344	function IGCM_sys_Tree {
345	IGCM_debug_PushStack "IGCM_sys_Tree" $@
346	if ( $DEBUG_sys ) ; then
347	echo "IGCM_sys_Tree :" $@
348	fi
349
350	\tree -f $@
351
352	IGCM_debug_PopStack "IGCM_sys_Tree"
353	}
354
355	#D-#==================================================
356	#D-function IGCM_sys_Qsub
357	#D-* Purpose: Qsub new job
358	#D-* Examples:
359	#D-
360	function IGCM_sys_Qsub {
361	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
362	if ( $DEBUG_sys ) ; then
363	echo "IGCM_sys_Qsub :" $@
364	fi
365	typeset options status
366	options="-o ${SUBMIT_DIR}/${Script_Output}"
367	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
368	status=$?
369
370	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
371	if [ ${status} -gt 0 ] ; then
372	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
373	IGCM_debug_Exit "IGCM_sys_Qsub"
374	else
375	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
376	fi
377	IGCM_debug_PopStack "IGCM_sys_Qsub"
378	}
379
380	#D-#==================================================
381	#D-function IGCM_sys_QsubPost
382	#D-* Purpose: Qsub new job on scalaire
383	#D-* Examples:
384	#D-
385	function IGCM_sys_QsubPost {
386	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
387	if ( $DEBUG_sys ) ; then
388	echo "IGCM_sys_QsubPost :" $@
389	fi
390	typeset options status
391	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
392	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
393	status=$?
394
395	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
396	if [ ${status} -gt 0 ] ; then
397	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
398	IGCM_debug_Exit "IGCM_sys_QsubPost"
399	else
400	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
401	fi
402	IGCM_debug_PopStack "IGCM_sys_QsubPost"
403	}
404
405	#D-*************************
406	#D- File transfer functions
407	#D-*************************
408	#D-
409
410	#D-#==================================================
411	#D-function IGCM_sys_RmRunDir
412	#D-* Purpose: rm tmpdir (dummy function most of the time batch
413	#D- scheduler will do the job)
414	#D-* Examples:
415	#D-
416	function IGCM_sys_RmRunDir {
417	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
418	if ( $DEBUG_sys ) ; then
419	echo "IGCM_sys_RmRunDir :" $@
420	echo "Dummy call, let the scheduler do that."
421	fi
422	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
423	}
424
425	#D-#==================================================
426	#D-function IGCM_sys_Put_Dir
427	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
428	#D-* Examples:
429	#D-
430	function IGCM_sys_Put_Dir {
431	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
432	if ( $DEBUG_sys ) ; then
433	echo "IGCM_sys_Put_Dir :" $@
434	fi
435	if [ $DRYRUN = 0 ]; then
436	if [ ! -d ${1} ] ; then
437	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
438	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
439	return
440	fi
441
442	typeset status
443
444	# Only if we use rsync
445	#IGCM_sys_TestDirArchive $( dirname $2 )
446	#
447	#USUAL WAY
448	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
449	status=$?
450
451	if [ ${status} -gt 0 ] ; then
452	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
453	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
454	IGCM_debug_Exit "IGCM_sys_Put_Dir"
455	else
456	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
457	fi
458	fi
459	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
460	}
461
462	#D-#==================================================
463	#D-function IGCM_sys_Get_Dir
464	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
465	#D-* Examples:
466	#D-
467	function IGCM_sys_Get_Dir {
468	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
469	if ( $DEBUG_sys ) ; then
470	echo "IGCM_sys_Get_Dir :" $@
471	fi
472	if [ $DRYRUN = 0 ]; then
473	typeset NB_ESSAI DELAI status i
474	# number of tentative
475	NB_ESSAI=3
476	# time delay between tentative
477	DELAI=2
478
479	#
480	# USUAL WAY
481	i=0
482	while [ $i -lt $NB_ESSAI ] ; do
483	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
484	status=$?
485	if [ ${status} -gt 0 ] ; then
486	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
487	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
488	sleep $DELAI
489	else
490	break
491	fi
492	(( i = i + 1 ))
493	done
494
495	if [ ${status} -gt 0 ] ; then
496	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
497	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
498	IGCM_debug_Exit "IGCM_sys_Get_Dir"
499	else
500	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
501	fi
502	fi
503	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
504	}
505
506	#D-#==================================================
507	#D-function IGCM_sys_Put_Rest
508	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
509	#D- File and target directory must exist.
510	#D-* Examples:
511	#D-
512	function IGCM_sys_Put_Rest {
513	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
514	if ( $DEBUG_sys ) ; then
515	echo "IGCM_sys_Put_Rest :" $@
516	fi
517	if [ $DRYRUN = 0 ]; then
518	if [ ! -f ${1} ] ; then
519	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
520	IGCM_debug_Exit "IGCM_sys_Put_Rest"
521	fi
522
523	typeset status
524	#
525	# USUAL WAY
526	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
527	status=$?
528
529	# #RSYNC WITH NETWORK SSH CALL
530	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
531	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
532
533	# #RSYNC WITH NFS USE
534	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
535	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
536
537	# status=$?
538	# IGCM_sys_Rsync_out $status
539
540	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
541	# (( status=status+$? ))
542
543	if [ ${status} -gt 0 ] ; then
544	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
545	[ -f ${1} ] && ls -l ${1}
546	[ -f ${2} ] && ls -l ${2}
547	[ -f ${2}/${1} ] && ls -l ${2}/${1}
548	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
549	IGCM_debug_Exit "IGCM_sys_Put_Rest"
550	else
551
552	if [ X${JobType} = XRUN ] ; then
553	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
554	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
555	fi
556
557	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
558	fi
559	fi
560	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
561	}
562
563	#D-#==================================================
564	#D-function IGCM_sys_Put_Out
565	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
566	#D-* Examples:
567	#D-
568	function IGCM_sys_Put_Out {
569	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
570	if ( $DEBUG_sys ) ; then
571	echo "IGCM_sys_Put_Out :" $@
572	fi
573
574	typeset status
575
576	if [ $DRYRUN = 0 ]; then
577	if [ ! -f ${1} ] ; then
578	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
579	IGCM_debug_PopStack "IGCM_sys_Put_Out"
580	return 1
581	fi
582	#
583	IGCM_sys_MkdirArchive $( dirname $2 )
584	#
585	if [ X${JobType} = XRUN ] ; then
586	if [ X${3} = X ] ; then
587	IGCM_sys_Chmod 444 ${1}
588	fi
589	fi
590	#
591
592	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
594	status=$?
595	IGCM_sys_Rsync_out $status
596
597	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
598	(( status=status+$? ))
599
600	if [ ${status} -gt 0 ] ; then
601	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
602	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
603	IGCM_debug_Exit "IGCM_sys_Put_Out"
604	else
605	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
606	fi
607	fi
608	IGCM_debug_PopStack "IGCM_sys_Put_Out"
609	return 0
610	}
611
612	#D-#==================================================
613	#D-function IGCM_sys_Get
614	#D-* Purpose: Get a file from ${ARCHIVE}
615	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
616	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
617	function IGCM_sys_Get {
618	IGCM_debug_PushStack "IGCM_sys_Get" $@
619
620	typeset DEST status dm_liste
621
622	if ( $DEBUG_sys ) ; then
623	echo "IGCM_sys_Get :" $@
624	fi
625	if [ $DRYRUN -le 2 ]; then
626	if [ X${1} = X'/l' ] ; then
627	eval set +A dm_liste \${${2}}
628	else
629	dm_liste=${1}
630	fi
631	eval DEST=\${${#}}
632
633	# test if the (first) file is present in the old computation :
634	IGCM_sys_TestFileArchive ${dm_liste[0]}
635	status=$?
636	if [ ${status} -gt 0 ] ; then
637	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
638	IGCM_debug_Exit "IGCM_sys_Get"
639	return
640	fi
641
642	#USUAL WAY
643	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
644	status=$?
645
646	if [ ${status} -gt 0 ] ; then
647	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
648	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
649	IGCM_debug_Exit "IGCM_sys_Get"
650	else
651	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
652	fi
653	fi
654	IGCM_debug_PopStack "IGCM_sys_Get"
655	}
656
657	#D-#==================================================
658	#D-function IGCM_sys_GetDate_Monitoring
659	#D-* Purpose: get the last year for which the monitoring has been computed
660	#D-* Examples:
661	#D-
662	function IGCM_sys_GetDate_Monitoring {
663	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
664	if ( $DEBUG_sys ) ; then
665	echo "IGCM_sys_GetDate_Monitoring :" $@
666	fi
667
668	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
669
670	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
671	}
672
673	#D-#==================================================
674	#D-function IGCM_sys_Dods_Rm
675	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
676	#D-* Examples:
677	#D-
678	function IGCM_sys_Dods_Rm {
679	if ( $DEBUG_sys ) ; then
680	echo "IGCM_sys_Dods_Rm :" $@
681	fi
682	return 0
683	}
684
685	#D-#==================================================
686	#D-function IGCM_sys_Dods_Cp
687	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
688	#D-* Examples:
689	#D-
690	function IGCM_sys_Dods_Cp {
691	if ( $DEBUG_sys ) ; then
692	echo "IGCM_sys_Dods_Cp :" $@
693	fi
694	return 0
695	}
696
697	#D-#==================================================
698	#D-function IGCM_sys_Put_Dods
699	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
700	#D-* Examples:
701	#D-
702	function IGCM_sys_Put_Dods {
703	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
704	if ( $DEBUG_sys ) ; then
705	echo "IGCM_sys_Put_Dods :" $@
706	fi
707	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
708	}
709
710	##############################################################
711	# REBUILD OPERATOR
712
713	#D-#==================================================
714	#D-function IGCM_sys_sync
715	#D-* Purpose: flush buffer on disk (dummy function on Ada)
716	#D-* Examples:
717	#D-
718	function IGCM_sys_sync {
719	IGCM_debug_PushStack "IGCM_sys_sync" $@
720	if ( $DEBUG_sys ) ; then
721	echo "IGCM_sys_sync :" $@
722	echo "Dummy call, let the system do that."
723	fi
724	IGCM_debug_PopStack "IGCM_sys_sync"
725	}
726
727	############################################################
728	# Activate Running Environnment Variables
729
730	#D-#==================================================
731	#D-function IGCM_sys_desactiv_variables
732	#D-* Purpose: set environement variables prior to execution
733	#D-* Examples:
734	#D-
735	function IGCM_sys_activ_variables {
736	IGCM_debug_PushStack "IGCM_sys_activ_variables"
737	if ( $DEBUG_sys ) ; then
738	echo "IGCM_sys_activ_variables"
739	fi
740
741	# --------------------------------------------------------------------
742	#D- MPI specifications
743	# --------------------------------------------------------------------
744
745	# --------------------------------------------------------------------
746	#D- Other specifications
747	# --------------------------------------------------------------------
748
749	IGCM_debug_PopStack "IGCM_sys_activ_variables"
750	}
751
752	############################################################
753	# Desactivate Running Environnment Variables
754
755	#D-#==================================================
756	#D-function IGCM_sys_desactiv_variables
757	#D-* Purpose: unset environement variables after execution
758	#D-* Examples:
759	#D-
760	function IGCM_sys_desactiv_variables {
761	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
762	if ( $DEBUG_sys ) ; then
763	echo "IGCM_sys_desactiv_variables"
764	fi
765	# --------------------------------------------------------------------
766	#D- MPI specifications
767	# --------------------------------------------------------------------
768
769	# --------------------------------------------------------------------
770	#D- Other specifications
771	# --------------------------------------------------------------------
772
773	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
774	}
775
776	############################################################
777	# Build MPI/OMP scripts run file (dummy function)
778
779	#D-#==================================================
780	#D-function IGCM_sys_build_run_file
781	#D-* Purpose: build run file (deprecated)
782	#D-* Examples:
783	#D-
784	function IGCM_sys_build_run_file {
785
786	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
787
788	}
789
790	############################################################
791	# Build MPI/OMP scripts
792
793	#D-#==================================================
794	#D-function IGCM_sys_build_execution_scripts
795	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
796	#D-* Examples:
797	#D-
798	function IGCM_sys_build_execution_scripts
799	{
800	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
801	if ( $DEBUG_sys ) ; then
802	echo "IGCM_sys_build_execution_scripts " $@
803	fi
804
805	typeset nodes listnodes init_node start_num init_exec comp ExeNameIn ExeNameOut
806	typeset node_num_current node_current comp_proc_mpi_loc comp_proc_omp_loc
807	typeset num_corempi nombre_restant_node nombre_restant_comp
808
809	if ( ${OK_PARA_MPMD} ) ; then
810
811	if [ -f run_file ] ; then
812	IGCM_sys_Rm -f run_file
813	fi
814	touch run_file
815
816	if ( ${OK_PARA_OMP} ) ; then
817
818	# Hosts treatment
819
820	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
821
822	i=0
823	rm -f hosts
824	IGCM_debug_Print 1 "sys Obelix, Hosts avaible :"
825	for nodes in `cat hosts.tmp` ; do
826	host[$i]=$nodes
827	echo "${host[$i]} slots=1 max_slots=1" >> hosts
828	IGCM_debug_Print 1 ${host[$i]}
829	i=$((i+1))
830	done
831	rm -f hosts.tmp
832
833	listnodes=${host[*]}
834
835	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
836
837	# Initialisation
838
839	init_node=y
840	node_num_current=0
841	start_num=0
842	init_exec=n
843
844	# Build run_file
845
846	# First loop on the components for the coupler ie oasis (only if oasis3)
847	# the coupler ie oasis3 must be the first one
848	for comp in ${config_ListOfComponents[*]} ; do
849
850	if [ "X${comp}" = "XCPL" ] ; then
851
852	eval ExeNameIn=\${config_Executable_${comp}[0]}
853	eval ExeNameOut=\${config_Executable_${comp}[1]}
854
855	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
856	echo "" >> script_${ExeNameOut}.ksh
857	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
858	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
859	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
860	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
861	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
862
863	init_node=n
864
865	(( nombre_restant_node = NUM_COREPERNODE - 1 ))
866	node_num_current=0
867	node_current=${host[${node_num_current}]}
868
869	EXECUTION="${EXECUTION} -H ${node_current} -np 1 ./script_${ExeNameOut}.ksh"
870
871	init_exec=y
872	start_num=1
873
874	fi
875
876	done
877
878	# Then loop on the components (except for oasis)
879
880	for comp in ${config_ListOfComponents[*]} ; do
881
882	eval ExeNameIn=\${config_Executable_${comp}[0]}
883	eval ExeNameOut=\${config_Executable_${comp}[1]}
884
885	# Only if we really have an executable for the component :
886	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
887
888	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
889	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
890
891	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
892	# echo "set -vx" >> script_${ExeNameOut}.ksh
893	echo "" >> script_${ExeNameOut}.ksh
894	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
895	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
896	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
897	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
898	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
899	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
900	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
901
902	node_num=0
903
904	# We define the number of MPI process to be assigned for the component
905
906	nombre_restant_comp=${comp_proc_mpi_loc}
907
908	# Loop on the allocated nodes
909
910	for node in ${listnodes} ; do
911
912	# We go to the current node
913	if [ ${node_num} = ${node_num_current} ] ; then
914
915	node_current=${host[${node_num_current}]}
916
917	# If first time on the node : initialisation
918
919	if [ ${init_node} = y ] ; then
920	nombre_restant_node=${NUM_COREPERNODE}
921	fi
922
923	# Test on the number of OMP threads
924
925	if [ ${comp_proc_omp_loc} -gt ${nombre_restant_node} ] ; then
926	(( node_num = node_num + 1 ))
927	node_num_current=${node_num}
928	init_node=y
929	continue
930	fi
931
932	# Number of MPI process to assign
933
934	(( num_corempi = nombre_restant_node / comp_proc_omp_loc ))
935
936	if [ ${num_corempi} -gt ${nombre_restant_comp} ] ; then
937	num_corempi=${nombre_restant_comp}
938	fi
939
940	(( nombre_restant_node = nombre_restant_node - num_corempi * comp_proc_omp_loc ))
941	(( nombre_restant_comp = nombre_restant_comp - num_corempi ))
942
943	if [ ${init_exec} = y ] ; then
944	EXECUTION="${EXECUTION} : -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
945	else
946	EXECUTION="${EXECUTION} -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
947	init_exec=y
948	fi
949
950	(( start_num = num_corempi + start_num ))
951
952	else
953
954	(( node_num = node_num + 1 ))
955	continue
956	fi
957
958	# Test on the number of core/process remaining on the node/component
959
960	if [ ${nombre_restant_node} = 0 ] ; then
961	(( node_num = node_num + 1 ))
962	node_num_current=${node_num}
963	init_node=y
964
965	if [ ${nombre_restant_comp} = 0 ] ; then
966	break 1
967	fi
968	else
969
970	node_num_current=${node_num}
971	init_node=n
972
973	if [ ${nombre_restant_comp} = 0 ] ; then
974	break 1
975	fi
976	fi
977	done
978	fi
979	done
980
981	else
982
983	# Then first loop on the components for the coupler ie oasis
984
985	## the coupler ie oasis must be the first one
986	for comp in ${config_ListOfComponents[*]} ; do
987
988	eval ExeNameOut=\${config_Executable_${comp}[1]}
989
990	# for CPL component only
991	if [ "X${comp}" = "XCPL" ] ; then
992	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
993	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
994	fi
995	done
996
997	# Then second loop on the components
998
999	for comp in ${config_ListOfComponents[*]} ; do
1000
1001	eval ExeNameOut=\${config_Executable_${comp}[1]}
1002
1003	# Only if we really have an executable for the component and not the coupler ie oasis:
1004	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1005	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1006	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1007	fi
1008	done
1009	IGCM_sys_Chmod u+x run_file
1010
1011	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
1012
1013	fi
1014
1015	else # Only one executable. launch it.
1016
1017	for comp in ${config_ListOfComponents[*]} ; do
1018
1019	# Only if we really have an executable for the component :
1020	eval ExeNameOut=\${config_Executable_${comp}[1]}
1021	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1022
1023	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1024	echo "" >> script_${ExeNameOut}.ksh
1025	if ( ${OK_PARA_OMP} ) ; then
1026	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1027	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1028	fi
1029	if ( ${OK_PARA_MPI} ) ; then
1030	# Default : mpirun used if nb_proc gt 1
1031	# pour sortie out/err par process
1032	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
1033	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1034	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1035	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1036	else
1037	# Default : mpirun is NOT used if nb_proc eq 1
1038	# pour sortie out/err par process
1039	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1040	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1041	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1042	EXECUTION="time ./script_${ExeNameOut}.ksh"
1043	fi
1044
1045	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
1046	cat script_${ExeNameOut}.ksh
1047
1048	fi
1049
1050	done
1051
1052	fi
1053
1054	IGCM_debug_Print 1 "sys Obelix : execution command is"
1055	IGCM_debug_Print 1 "$EXECUTION"
1056
1057	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1058	}
1059
1060	#D-#==================================================
1061	#D-function IGCM_sys_check_path
1062	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1063	#D-* do not point to an important use directory. Stop immediately in that case.
1064	#D-* Examples:
1065	#D-
1066	function IGCM_sys_check_path {
1067	IGCM_debug_PushStack "IGCM_sys_check_path"
1068	if ( $DEBUG_sys ) ; then
1069	echo "IGCM_sys_check_path"
1070	fi
1071
1072	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1073	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1074	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1075	IGCM_debug_Exit "This will stop the job"
1076	fi
1077	IGCM_debug_PopStack "IGCM_sys_check_path"
1078	}
1079
1080	#D-#==================================================
1081	#D-function IGCM_sys_check_quota. Dummy call here
1082	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1083	#D-* Examples:
1084	#D-
1085	function IGCM_sys_check_quota {
1086	IGCM_debug_PushStack "IGCM_sys_check_quota"
1087	if ( $DEBUG_sys ) ; then
1088	echo "IGCM_sys_check_quota"
1089	fi
1090	IGCM_debug_PopStack "IGCM_sys_check_quota"
1091	}
1092
1093	#D-#==================================================
1094	#D-function IGCM_sys_GetJobID
1095	#D-* Purpose: Check if job_name is currently
1096	#D- running or in queue
1097	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1098	#D-
1099	function IGCM_sys_GetJobID {
1100	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1101	if ( $DEBUG_sys ) ; then
1102	echo "IGCM_sys_GetJobID"
1103	fi
1104
1105	# With -f option, the full job name is given in the last column
1106	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
1107
1108	eval ${3}=${ID}
1109	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1110	}
1111
1112	#D-#==================================================
1113	#D-function IGCM_sys_CountJobInQueue
1114	#D-* Purpose: Check if job_name is currently
1115	#D- running or in queue
1116	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1117	#D-
1118	function IGCM_sys_CountJobInQueue {
1119	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1120	if ( $DEBUG_sys ) ; then
1121	echo "IGCM_sys_CountJobInQueue"
1122	fi
1123	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1124	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: