Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh @ 1232

Last change on this file since 1232 was 1230, checked in by sdipsl, 9 years ago
MPI/OMP handling refactoring IGCM_config_ConfigureExexution will define the MPMD/SPMDO/MPI/OMP context 6 execution types are introduced and will be documented later prerequisite to have ins_job editing the headers IGCM_sys_build_execution_scripts will be adapted later
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 35.3 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for obelix
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=${MASTER:=$( hostname )}
71	# project name
72	typeset PROJECT=NONE
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=${PBS_WALLTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=qsub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89
90	#====================================================
91	# Set environment tools (ferret, nco, cdo, rebuild, ...)
92	#====================================================
93	. /home/users/igcmg/.atlas_env_asterix_ksh
94	export PATH=${PATH}:/home/users/igcmg/rebuild/bin/
95
96	#====================================================
97	# Set lf95 environment
98	#====================================================
99	. /usr/local/install/lf6481/bash_laheyfort_setup
100
101	#====================================================
102	# Host specific DIRECTORIES
103	#====================================================
104
105	#====================================================
106	#- MirrorlibIGCM for frontend
107	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
108
109	#====================================================
110	#- libIGCM_POST for frontend
111	typeset -r libIGCM_POST=${libIGCM}
112
113	#====================================================
114	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
115	typeset -r R_EXE="${MODIPSL}/bin"
116
117	#====================================================
118	#- SUBMIT_DIR : submission dir
119	if [ X${PBS_O_WORKDIR} != X ] ; then
120	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
121	else
122	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
123	fi
124
125	#====================================================
126	#- IN
127	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
128	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/home/orchideeshare/igcmg/IGCM}
129
130	#====================================================
131	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
132	if [ X${PBS_JOBID} != X ] ; then
133	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/scratch/$PBS_O_LOGNAME.$PBS_JOBID}
134	else
135	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/tmp/tmp$$}
136	fi
137
138	#====================================================
139	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
140	typeset -r OUTCOMMAND_PATH=/tmp
141
142	#====================================================
143	#- HOST_MPIRUN_COMMAND
144	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
145
146	#====================================================
147	#- Max number of arguments passed to nco operator or demigration command
148	UNIX_MAX_LIMIT=360
149
150	#====================================================
151	#- set PackDefault to false on obelix
152	PackDefault=false
153
154	#====================================================
155	#- Number of core per node (max number of OpenMP task)
156	NUM_COREPERNODE=1
157
158	#====================================================
159	#- Default number of MPI task for IPSL coupled model
160	#- required for backward compatibility
161	#-
162	DEFAULT_NUM_PROC_OCE=1
163	DEFAULT_NUM_PROC_CPL=1
164	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
165
166	#D-#==================================================
167	#D-function IGCM_sys_defineArchives
168	#D-* Purpose:
169	#D-* Define ARCHIVE : Dedicated to large files
170	#D-* Define STORAGE : Dedicated to small/medium files
171	#D-* Define R_OUT : Output tree located on ARCHIVE
172	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
173	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
174	#D-* if SpaceName=TEST nothing special will hapen
175	#D-* Examples:
176	#D-
177	function IGCM_sys_defineArchives {
178	IGCM_debug_PushStack "IGCM_sys_defineArchives"
179
180	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
181	#====================================================
182	#- ARCHIVE (dedicated to large files)
183	ARCHIVE=${config_UserChoices_ARCHIVE}
184	else
185	#====================================================
186	#- ARCHIVE (dedicated to large files)
187	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
188	fi
189
190	if [ ! X${config_UserChoices_STORAGE} = X ]; then
191	#====================================================
192	#- STORAGE (dedicated to small/medium files)
193	STORAGE=${config_UserChoices_STORAGE}
194	else
195	#====================================================
196	#- STORAGE (dedicated to small/medium files)
197	STORAGE=${ARCHIVE}
198	fi
199
200	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
201
202	#====================================================
203	#- R_OUT
204	R_OUT=${ARCHIVE}/IGCM_OUT
205
206	#====================================================
207	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
208	R_FIG=${STORAGE}/IGCM_OUT
209
210	#====================================================
211	#- R_BUF (ONLY FOR double copy an scratch)
212	R_BUF=${STORAGE}/IGCM_OUT
213
214	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
215	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
216	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
217
218	IGCM_debug_PopStack "IGCM_sys_defineArchives"
219	}
220
221	#D-#==================================================
222	#D-function IGCM_sys_RshArchive
223	#D-* Purpose: Archive rsh command
224	#D-* Examples:
225	#D-
226	function IGCM_sys_RshArchive {
227	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
228	/bin/ksh <<-EOF
229	${@}
230	EOF
231	status=$?
232	if [ ${status} -gt 0 ] ; then
233	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
234	IGCM_debug_Exit "IGCM_sys_RshArchive"
235	fi
236	IGCM_debug_PopStack "IGCM_sys_RshArchive"
237	}
238
239	#D-#==================================================
240	#D-function IGCM_sys_RshArchive_NoError
241	#D-* Purpose: Archive rsh command, without error
242	#D-* used only in monitoring.job
243	#D-* Examples:
244	#D-
245	function IGCM_sys_RshArchive_NoError {
246	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
247	/bin/ksh <<-EOF
248	${@} 2> /dev/null
249	EOF
250	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
251	}
252
253	#D-#==================================================
254	#D-function IGCM_sys_MkdirArchive
255	#D-* Purpose: Mkdir on Archive
256	#D-* Examples:
257	#D-
258	function IGCM_sys_MkdirArchive {
259	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
260	if ( $DEBUG_sys ) ; then
261	echo "IGCM_sys_MkdirArchive :" $@
262	fi
263	#- creation de repertoire sur le serveur fichier
264	if [ ! -d ${1} ]; then
265	\mkdir -p $1
266	status=$?
267
268	if [ ${status} -gt 0 ] ; then
269	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
270	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
271	fi
272	fi
273	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
274	}
275
276	#D-#==================================================
277	#D-function IGCM_sys_TestDirArchive
278	#D-* Purpose: Test Directory that must exists on Archive
279	#D-* Examples:
280	#D-
281	function IGCM_sys_TestDirArchive {
282	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
283	if ( $DEBUG_sys ) ; then
284	echo "IGCM_sys_TestDirArchive :" $@
285	fi
286	typeset ExistFlag
287	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
288	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
289	return ${ExistFlag}
290	}
291
292	#D-#==================================================
293	#D-function IGCM_sys_IsFileArchived
294	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
295	#D-* Examples:
296	#D-
297	function IGCM_sys_IsFileArchived {
298	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
299	if ( $DEBUG_sys ) ; then
300	echo "IGCM_sys_IsFileArchived :" $@
301	fi
302	typeset IsArchivedFlag
303	# Never archived for this system
304	IsArchivedFlag=1
305	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
306
307	return ${IsArchivedFlag}
308	}
309
310	#D-#==================================================
311	#D-function IGCM_sys_TestFileArchive
312	#D-* Purpose: Test file that must NOT EXISTS on Archive
313	#D-* Examples:
314	#D-
315	function IGCM_sys_TestFileArchive {
316	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
317	typeset ExistFlag
318	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
319	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
320
321	return ${ExistFlag}
322	}
323
324	#D-#==================================================
325	#D-function IGCM_sys_CountFileArchive
326	#D-* Purpose: Count files on Archive filesystem
327	#D-* Examples:
328	#D-
329	function IGCM_sys_CountFileArchive {
330	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
331	ls ${@} 2>/dev/null \| wc -l
332	if [ $? -gt 0 ] ; then
333	echo "IGCM_sys_CountFileArchive : erreur."
334	fi
335	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
336	}
337
338	#D-#==================================================
339	#D-function IGCM_sys_Tree
340	#D-* Purpose: Tree directories with files on ${ARCHIVE}
341	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
342	#D-
343	function IGCM_sys_Tree {
344	IGCM_debug_PushStack "IGCM_sys_Tree" $@
345	if ( $DEBUG_sys ) ; then
346	echo "IGCM_sys_Tree :" $@
347	fi
348
349	\tree -f $@
350
351	IGCM_debug_PopStack "IGCM_sys_Tree"
352	}
353
354	#D-#==================================================
355	#D-function IGCM_sys_Qsub
356	#D-* Purpose: Qsub new job
357	#D-* Examples:
358	#D-
359	function IGCM_sys_Qsub {
360	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
361	if ( $DEBUG_sys ) ; then
362	echo "IGCM_sys_Qsub :" $@
363	fi
364	typeset options status
365	options="-o ${SUBMIT_DIR}/${Script_Output}"
366	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
367	status=$?
368
369	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
370	if [ ${status} -gt 0 ] ; then
371	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
372	IGCM_debug_Exit "IGCM_sys_Qsub"
373	else
374	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
375	fi
376	IGCM_debug_PopStack "IGCM_sys_Qsub"
377	}
378
379	#D-#==================================================
380	#D-function IGCM_sys_QsubPost
381	#D-* Purpose: Qsub new job on scalaire
382	#D-* Examples:
383	#D-
384	function IGCM_sys_QsubPost {
385	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
386	if ( $DEBUG_sys ) ; then
387	echo "IGCM_sys_QsubPost :" $@
388	fi
389	typeset options status
390	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
391	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
392	status=$?
393
394	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
395	if [ ${status} -gt 0 ] ; then
396	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
397	IGCM_debug_Exit "IGCM_sys_QsubPost"
398	else
399	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
400	fi
401	IGCM_debug_PopStack "IGCM_sys_QsubPost"
402	}
403
404	#D-*************************
405	#D- File transfer functions
406	#D-*************************
407	#D-
408
409	#D-#==================================================
410	#D-function IGCM_sys_RmRunDir
411	#D-* Purpose: rm tmpdir (dummy function most of the time batch
412	#D- scheduler will do the job)
413	#D-* Examples:
414	#D-
415	function IGCM_sys_RmRunDir {
416	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
417	if ( $DEBUG_sys ) ; then
418	echo "IGCM_sys_RmRunDir :" $@
419	echo "Dummy call, let the scheduler do that."
420	fi
421	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
422	}
423
424	#D-#==================================================
425	#D-function IGCM_sys_Put_Dir
426	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
427	#D-* Examples:
428	#D-
429	function IGCM_sys_Put_Dir {
430	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
431	if ( $DEBUG_sys ) ; then
432	echo "IGCM_sys_Put_Dir :" $@
433	fi
434	if [ $DRYRUN = 0 ]; then
435	if [ ! -d ${1} ] ; then
436	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
437	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
438	return
439	fi
440
441	typeset status
442
443	# Only if we use rsync
444	#IGCM_sys_TestDirArchive $( dirname $2 )
445	#
446	#USUAL WAY
447	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
448	status=$?
449
450	if [ ${status} -gt 0 ] ; then
451	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
452	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
453	IGCM_debug_Exit "IGCM_sys_Put_Dir"
454	else
455	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
456	fi
457	fi
458	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
459	}
460
461	#D-#==================================================
462	#D-function IGCM_sys_Get_Dir
463	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
464	#D-* Examples:
465	#D-
466	function IGCM_sys_Get_Dir {
467	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
468	if ( $DEBUG_sys ) ; then
469	echo "IGCM_sys_Get_Dir :" $@
470	fi
471	if [ $DRYRUN = 0 ]; then
472	typeset NB_ESSAI DELAI status i
473	# number of tentative
474	NB_ESSAI=3
475	# time delay between tentative
476	DELAI=2
477
478	#
479	# USUAL WAY
480	i=0
481	while [ $i -lt $NB_ESSAI ] ; do
482	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
483	status=$?
484	if [ ${status} -gt 0 ] ; then
485	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
486	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
487	sleep $DELAI
488	else
489	break
490	fi
491	(( i = i + 1 ))
492	done
493
494	if [ ${status} -gt 0 ] ; then
495	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
496	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
497	IGCM_debug_Exit "IGCM_sys_Get_Dir"
498	else
499	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
500	fi
501	fi
502	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
503	}
504
505	#D-#==================================================
506	#D-function IGCM_sys_Put_Rest
507	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
508	#D- File and target directory must exist.
509	#D-* Examples:
510	#D-
511	function IGCM_sys_Put_Rest {
512	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
513	if ( $DEBUG_sys ) ; then
514	echo "IGCM_sys_Put_Rest :" $@
515	fi
516	if [ $DRYRUN = 0 ]; then
517	if [ ! -f ${1} ] ; then
518	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
519	IGCM_debug_Exit "IGCM_sys_Put_Rest"
520	fi
521
522	typeset status
523	#
524	# USUAL WAY
525	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
526	status=$?
527
528	# #RSYNC WITH NETWORK SSH CALL
529	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
530	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
531
532	# #RSYNC WITH NFS USE
533	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
534	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
535
536	# status=$?
537	# IGCM_sys_Rsync_out $status
538
539	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
540	# (( status=status+$? ))
541
542	if [ ${status} -gt 0 ] ; then
543	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
544	[ -f ${1} ] && ls -l ${1}
545	[ -f ${2} ] && ls -l ${2}
546	[ -f ${2}/${1} ] && ls -l ${2}/${1}
547	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
548	IGCM_debug_Exit "IGCM_sys_Put_Rest"
549	else
550
551	if [ X${JobType} = XRUN ] ; then
552	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
553	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
554	fi
555
556	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
557	fi
558	fi
559	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
560	}
561
562	#D-#==================================================
563	#D-function IGCM_sys_Put_Out
564	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
565	#D-* Examples:
566	#D-
567	function IGCM_sys_Put_Out {
568	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
569	if ( $DEBUG_sys ) ; then
570	echo "IGCM_sys_Put_Out :" $@
571	fi
572
573	typeset status
574
575	if [ $DRYRUN = 0 ]; then
576	if [ ! -f ${1} ] ; then
577	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
578	IGCM_debug_PopStack "IGCM_sys_Put_Out"
579	return 1
580	fi
581	#
582	IGCM_sys_MkdirArchive $( dirname $2 )
583	#
584	if [ X${JobType} = XRUN ] ; then
585	if [ X${3} = X ] ; then
586	IGCM_sys_Chmod 444 ${1}
587	fi
588	fi
589	#
590
591	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
592	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593	status=$?
594	IGCM_sys_Rsync_out $status
595
596	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
597	(( status=status+$? ))
598
599	if [ ${status} -gt 0 ] ; then
600	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
601	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
602	IGCM_debug_Exit "IGCM_sys_Put_Out"
603	else
604	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
605	fi
606	fi
607	IGCM_debug_PopStack "IGCM_sys_Put_Out"
608	return 0
609	}
610
611	#D-#==================================================
612	#D-function IGCM_sys_Get
613	#D-* Purpose: Get a file from ${ARCHIVE}
614	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
615	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
616	function IGCM_sys_Get {
617	IGCM_debug_PushStack "IGCM_sys_Get" $@
618
619	typeset DEST status dm_liste
620
621	if ( $DEBUG_sys ) ; then
622	echo "IGCM_sys_Get :" $@
623	fi
624	if [ $DRYRUN -le 2 ]; then
625	if [ X${1} = X'/l' ] ; then
626	eval set +A dm_liste \${${2}}
627	else
628	dm_liste=${1}
629	fi
630	eval DEST=\${${#}}
631
632	# test if the (first) file is present in the old computation :
633	IGCM_sys_TestFileArchive ${dm_liste[0]}
634	status=$?
635	if [ ${status} -gt 0 ] ; then
636	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
637	IGCM_debug_Exit "IGCM_sys_Get"
638	return
639	fi
640
641	#USUAL WAY
642	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
643	status=$?
644
645	if [ ${status} -gt 0 ] ; then
646	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
647	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
648	IGCM_debug_Exit "IGCM_sys_Get"
649	else
650	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
651	fi
652	fi
653	IGCM_debug_PopStack "IGCM_sys_Get"
654	}
655
656	#D-#==================================================
657	#D-function IGCM_sys_GetDate_Monitoring
658	#D-* Purpose: get the last year for which the monitoring has been computed
659	#D-* Examples:
660	#D-
661	function IGCM_sys_GetDate_Monitoring {
662	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
663	if ( $DEBUG_sys ) ; then
664	echo "IGCM_sys_GetDate_Monitoring :" $@
665	fi
666
667	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
668
669	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
670	}
671
672	#D-#==================================================
673	#D-function IGCM_sys_Dods_Rm
674	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
675	#D-* Examples:
676	#D-
677	function IGCM_sys_Dods_Rm {
678	if ( $DEBUG_sys ) ; then
679	echo "IGCM_sys_Dods_Rm :" $@
680	fi
681	return 0
682	}
683
684	#D-#==================================================
685	#D-function IGCM_sys_Dods_Cp
686	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
687	#D-* Examples:
688	#D-
689	function IGCM_sys_Dods_Cp {
690	if ( $DEBUG_sys ) ; then
691	echo "IGCM_sys_Dods_Cp :" $@
692	fi
693	return 0
694	}
695
696	#D-#==================================================
697	#D-function IGCM_sys_Put_Dods
698	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
699	#D-* Examples:
700	#D-
701	function IGCM_sys_Put_Dods {
702	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
703	if ( $DEBUG_sys ) ; then
704	echo "IGCM_sys_Put_Dods :" $@
705	fi
706	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
707	}
708
709	##############################################################
710	# REBUILD OPERATOR
711
712	#D-#==================================================
713	#D-function IGCM_sys_sync
714	#D-* Purpose: flush buffer on disk (dummy function on Ada)
715	#D-* Examples:
716	#D-
717	function IGCM_sys_sync {
718	IGCM_debug_PushStack "IGCM_sys_sync" $@
719	if ( $DEBUG_sys ) ; then
720	echo "IGCM_sys_sync :" $@
721	echo "Dummy call, let the system do that."
722	fi
723	IGCM_debug_PopStack "IGCM_sys_sync"
724	}
725
726	############################################################
727	# Activate Running Environnment Variables
728
729	#D-#==================================================
730	#D-function IGCM_sys_desactiv_variables
731	#D-* Purpose: set environement variables prior to execution
732	#D-* Examples:
733	#D-
734	function IGCM_sys_activ_variables {
735	IGCM_debug_PushStack "IGCM_sys_activ_variables"
736	if ( $DEBUG_sys ) ; then
737	echo "IGCM_sys_activ_variables"
738	fi
739
740	# --------------------------------------------------------------------
741	#D- MPI specifications
742	# --------------------------------------------------------------------
743
744	# --------------------------------------------------------------------
745	#D- Other specifications
746	# --------------------------------------------------------------------
747
748	IGCM_debug_PopStack "IGCM_sys_activ_variables"
749	}
750
751	############################################################
752	# Desactivate Running Environnment Variables
753
754	#D-#==================================================
755	#D-function IGCM_sys_desactiv_variables
756	#D-* Purpose: unset environement variables after execution
757	#D-* Examples:
758	#D-
759	function IGCM_sys_desactiv_variables {
760	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
761	if ( $DEBUG_sys ) ; then
762	echo "IGCM_sys_desactiv_variables"
763	fi
764	# --------------------------------------------------------------------
765	#D- MPI specifications
766	# --------------------------------------------------------------------
767
768	# --------------------------------------------------------------------
769	#D- Other specifications
770	# --------------------------------------------------------------------
771
772	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
773	}
774
775	############################################################
776	# Build MPI/OMP scripts run file (dummy function)
777
778	#D-#==================================================
779	#D-function IGCM_sys_build_run_file
780	#D-* Purpose: build run file (deprecated)
781	#D-* Examples:
782	#D-
783	function IGCM_sys_build_run_file {
784
785	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
786
787	}
788
789	############################################################
790	# Build MPI/OMP scripts
791
792	#D-#==================================================
793	#D-function IGCM_sys_build_execution_scripts
794	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
795	#D-* Examples:
796	#D-
797	function IGCM_sys_build_execution_scripts
798	{
799	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
800	if ( $DEBUG_sys ) ; then
801	echo "IGCM_sys_build_execution_scripts " $@
802	fi
803
804	typeset nodes listnodes init_node start_num init_exec comp ExeNameIn ExeNameOut
805	typeset node_num_current node_current comp_proc_mpi_loc comp_proc_omp_loc
806	typeset num_corempi nombre_restant_node nombre_restant_comp
807
808	if ( ${OK_PARA_MPMD} ) ; then
809
810	if [ -f run_file ] ; then
811	IGCM_sys_Rm -f run_file
812	fi
813	touch run_file
814
815	if ( ${OK_PARA_OMP} ) ; then
816
817	# Hosts treatment
818
819	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
820
821	i=0
822	rm -f hosts
823	IGCM_debug_Print 1 "sys Obelix, Hosts avaible :"
824	for nodes in `cat hosts.tmp` ; do
825	host[$i]=$nodes
826	echo "${host[$i]} slots=1 max_slots=1" >> hosts
827	IGCM_debug_Print 1 ${host[$i]}
828	i=$((i+1))
829	done
830	rm -f hosts.tmp
831
832	listnodes=${host[*]}
833
834	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
835
836	# Initialisation
837
838	init_node=y
839	node_num_current=0
840	start_num=0
841	init_exec=n
842
843	# Build run_file
844
845	# First loop on the components for the coupler ie oasis (only if oasis3)
846	# the coupler ie oasis3 must be the first one
847	for comp in ${config_ListOfComponents[*]} ; do
848
849	if [ "X${comp}" = "XCPL" ] ; then
850
851	eval ExeNameIn=\${config_Executable_${comp}[0]}
852	eval ExeNameOut=\${config_Executable_${comp}[1]}
853
854	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
855	echo "" >> script_${ExeNameOut}.ksh
856	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
857	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
858	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
859	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
860	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
861
862	init_node=n
863
864	(( nombre_restant_node = NUM_COREPERNODE - 1 ))
865	node_num_current=0
866	node_current=${host[${node_num_current}]}
867
868	EXECUTION="${EXECUTION} -H ${node_current} -np 1 ./script_${ExeNameOut}.ksh"
869
870	init_exec=y
871	start_num=1
872
873	fi
874
875	done
876
877	# Then loop on the components (except for oasis)
878
879	for comp in ${config_ListOfComponents[*]} ; do
880
881	eval ExeNameIn=\${config_Executable_${comp}[0]}
882	eval ExeNameOut=\${config_Executable_${comp}[1]}
883
884	# Only if we really have an executable for the component :
885	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
886
887	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
888	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
889
890	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
891	# echo "set -vx" >> script_${ExeNameOut}.ksh
892	echo "" >> script_${ExeNameOut}.ksh
893	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
894	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
895	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
896	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
897	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
898	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
899	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
900
901	node_num=0
902
903	# We define the number of MPI process to be assigned for the component
904
905	nombre_restant_comp=${comp_proc_mpi_loc}
906
907	# Loop on the allocated nodes
908
909	for node in ${listnodes} ; do
910
911	# We go to the current node
912	if [ ${node_num} = ${node_num_current} ] ; then
913
914	node_current=${host[${node_num_current}]}
915
916	# If first time on the node : initialisation
917
918	if [ ${init_node} = y ] ; then
919	nombre_restant_node=${NUM_COREPERNODE}
920	fi
921
922	# Test on the number of OMP threads
923
924	if [ ${comp_proc_omp_loc} -gt ${nombre_restant_node} ] ; then
925	(( node_num = node_num + 1 ))
926	node_num_current=${node_num}
927	init_node=y
928	continue
929	fi
930
931	# Number of MPI process to assign
932
933	(( num_corempi = nombre_restant_node / comp_proc_omp_loc ))
934
935	if [ ${num_corempi} -gt ${nombre_restant_comp} ] ; then
936	num_corempi=${nombre_restant_comp}
937	fi
938
939	(( nombre_restant_node = nombre_restant_node - num_corempi * comp_proc_omp_loc ))
940	(( nombre_restant_comp = nombre_restant_comp - num_corempi ))
941
942	if [ ${init_exec} = y ] ; then
943	EXECUTION="${EXECUTION} : -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
944	else
945	EXECUTION="${EXECUTION} -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
946	init_exec=y
947	fi
948
949	(( start_num = num_corempi + start_num ))
950
951	else
952
953	(( node_num = node_num + 1 ))
954	continue
955	fi
956
957	# Test on the number of core/process remaining on the node/component
958
959	if [ ${nombre_restant_node} = 0 ] ; then
960	(( node_num = node_num + 1 ))
961	node_num_current=${node_num}
962	init_node=y
963
964	if [ ${nombre_restant_comp} = 0 ] ; then
965	break 1
966	fi
967	else
968
969	node_num_current=${node_num}
970	init_node=n
971
972	if [ ${nombre_restant_comp} = 0 ] ; then
973	break 1
974	fi
975	fi
976	done
977	fi
978	done
979
980	else
981
982	# Then first loop on the components for the coupler ie oasis
983
984	## the coupler ie oasis must be the first one
985	for comp in ${config_ListOfComponents[*]} ; do
986
987	eval ExeNameOut=\${config_Executable_${comp}[1]}
988
989	# for CPL component only
990	if [ "X${comp}" = "XCPL" ] ; then
991	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
992	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
993	fi
994	done
995
996	# Then second loop on the components
997
998	for comp in ${config_ListOfComponents[*]} ; do
999
1000	eval ExeNameOut=\${config_Executable_${comp}[1]}
1001
1002	# Only if we really have an executable for the component and not the coupler ie oasis:
1003	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1004	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1005	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1006	fi
1007	done
1008	IGCM_sys_Chmod u+x run_file
1009
1010	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
1011
1012	fi
1013
1014	else # Only one executable. launch it.
1015
1016	for comp in ${config_ListOfComponents[*]} ; do
1017
1018	# Only if we really have an executable for the component :
1019	eval ExeNameOut=\${config_Executable_${comp}[1]}
1020	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1021
1022	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1023	echo "" >> script_${ExeNameOut}.ksh
1024	if ( ${OK_PARA_OMP} ) ; then
1025	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1026	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1027	fi
1028	if ( ${OK_PARA_MPI} ) ; then
1029	# Default : mpirun used if nb_proc gt 1
1030	# pour sortie out/err par process
1031	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
1032	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1033	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1034	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1035	else
1036	# Default : mpirun is NOT used if nb_proc eq 1
1037	# pour sortie out/err par process
1038	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1039	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1040	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1041	EXECUTION="time ./script_${ExeNameOut}.ksh"
1042	fi
1043
1044	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
1045	cat script_${ExeNameOut}.ksh
1046
1047	fi
1048
1049	done
1050
1051	fi
1052
1053	IGCM_debug_Print 1 "sys Obelix : execution command is"
1054	IGCM_debug_Print 1 "$EXECUTION"
1055
1056	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1057	}
1058
1059	#D-#==================================================
1060	#D-function IGCM_sys_check_path
1061	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1062	#D-* do not point to an important use directory. Stop immediately in that case.
1063	#D-* Examples:
1064	#D-
1065	function IGCM_sys_check_path {
1066	IGCM_debug_PushStack "IGCM_sys_check_path"
1067	if ( $DEBUG_sys ) ; then
1068	echo "IGCM_sys_check_path"
1069	fi
1070
1071	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1072	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1073	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1074	IGCM_debug_Exit "This will stop the job"
1075	fi
1076	IGCM_debug_PopStack "IGCM_sys_check_path"
1077	}
1078
1079	#D-#==================================================
1080	#D-function IGCM_sys_check_quota. Dummy call here
1081	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1082	#D-* Examples:
1083	#D-
1084	function IGCM_sys_check_quota {
1085	IGCM_debug_PushStack "IGCM_sys_check_quota"
1086	if ( $DEBUG_sys ) ; then
1087	echo "IGCM_sys_check_quota"
1088	fi
1089	IGCM_debug_PopStack "IGCM_sys_check_quota"
1090	}
1091
1092	#D-#==================================================
1093	#D-function IGCM_sys_GetJobID
1094	#D-* Purpose: Check if job_name is currently
1095	#D- running or in queue
1096	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1097	#D-
1098	function IGCM_sys_GetJobID {
1099	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1100	if ( $DEBUG_sys ) ; then
1101	echo "IGCM_sys_GetJobID"
1102	fi
1103
1104	# With -f option, the full job name is given in the last column
1105	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
1106
1107	eval ${3}=${ID}
1108	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1109	}
1110
1111	#D-#==================================================
1112	#D-function IGCM_sys_CountJobInQueue
1113	#D-* Purpose: Check if job_name is currently
1114	#D- running or in queue
1115	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1116	#D-
1117	function IGCM_sys_CountJobInQueue {
1118	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1119	if ( $DEBUG_sys ) ; then
1120	echo "IGCM_sys_CountJobInQueue"
1121	fi
1122	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1123	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: