Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_default.ksh @ 1232

Last change on this file since 1232 was 1230, checked in by sdipsl, 9 years ago
MPI/OMP handling refactoring IGCM_config_ConfigureExexution will define the MPMD/SPMDO/MPI/OMP context 6 execution types are introduced and will be documented later prerequisite to have ins_job editing the headers IGCM_sys_build_execution_scripts will be adapted later
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 33.9 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip
5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys Default host
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=${MASTER:=$( hostname )}
71	# project name
72	typeset PROJECT=NONE
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=NONE
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=qsub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89
90	#====================================================
91	# Set environment tools (ferret, nco, cdo, rebuild, ...)
92	#====================================================
93	#. /home/${LOGIN}/.atlas_env_${HOST}_ksh
94	#export PATH=${PATH}:/home/${LOGIN}/rebuild/bin/
95
96	#====================================================
97	# Host specific DIRECTORIES
98	#====================================================
99
100	#====================================================
101	#- MirrorlibIGCM for frontend
102	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
103
104	#====================================================
105	#- libIGCM_POST for frontend
106	typeset -r libIGCM_POST=${libIGCM}
107
108	#====================================================
109	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
110	typeset -r R_EXE="${MODIPSL}/bin"
111
112	#====================================================
113	#- SUBMIT_DIR : submission dir
114	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
115
116	#====================================================
117	#- IN
118	typeset -r R_IN=${R_IN:=/home/${LOGIN}/IGCM}
119	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/home/${LOGIN}/IGCM}
120
121	#====================================================
122	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
123	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/${LOGIN}/RUN_DIR/tmp$$}
124
125	#====================================================
126	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
127	typeset -r OUTCOMMAND_PATH=/tmp
128
129	#====================================================
130	#- HOST_MPIRUN_COMMAND
131	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
132
133	#====================================================
134	#- Max number of arguments passed to nco operator or demigration command
135	UNIX_MAX_LIMIT=360
136
137	#====================================================
138	#- set PackDefault to false on obelix
139	PackDefault=false
140
141	#====================================================
142	#- Number of core per node (max number of OpenMP task)
143	NUM_COREPERNODE=1
144
145	#====================================================
146	#- Default number of MPI task for IPSL coupled model
147	#- required for backward compatibility
148	#-
149	DEFAULT_NUM_PROC_OCE=1
150	DEFAULT_NUM_PROC_CPL=1
151	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
152
153	#D-#==================================================
154	#D-function IGCM_sys_defineArchives
155	#D-* Purpose:
156	#D-* Define ARCHIVE : Dedicated to large files
157	#D-* Define STORAGE : Dedicated to small/medium files
158	#D-* Define R_OUT : Output tree located on ARCHIVE
159	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
160	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
161	#D-* if SpaceName=TEST nothing special will hapen
162	#D-* Examples:
163	#D-
164	function IGCM_sys_defineArchives {
165	IGCM_debug_PushStack "IGCM_sys_defineArchives"
166
167	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
168	#====================================================
169	#- ARCHIVE (dedicated to large files)
170	ARCHIVE=${config_UserChoices_ARCHIVE}
171	else
172	#====================================================
173	#- ARCHIVE (dedicated to large files)
174	ARCHIVE=${ARCHIVE:=/home/${LOGIN}}
175	fi
176
177	if [ ! X${config_UserChoices_STORAGE} = X ]; then
178	#====================================================
179	#- STORAGE (dedicated to small/medium files)
180	STORAGE=${config_UserChoices_STORAGE}
181	else
182	#====================================================
183	#- STORAGE (dedicated to small/medium files)
184	STORAGE=${ARCHIVE}
185	fi
186
187	# ON DEFAULT MACHINE NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
188
189	#====================================================
190	#- R_OUT
191	R_OUT=${ARCHIVE}/IGCM_OUT
192
193	#====================================================
194	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
195	R_FIG=${STORAGE}/IGCM_OUT
196
197	#====================================================
198	#- R_BUF (ONLY FOR double copy an scratch)
199	R_BUF=${STORAGE}/IGCM_OUT
200
201	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
202	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
203	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
204
205	IGCM_debug_PopStack "IGCM_sys_defineArchives"
206	}
207
208	#D-#==================================================
209	#D-function IGCM_sys_RshArchive
210	#D-* Purpose: Archive rsh command
211	#D-* Examples:
212	#D-
213	function IGCM_sys_RshArchive {
214	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
215	/bin/ksh <<-EOF
216	${@}
217	EOF
218	status=$?
219	if [ ${status} -gt 0 ] ; then
220	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
221	IGCM_debug_Exit "IGCM_sys_RshArchive"
222	fi
223	IGCM_debug_PopStack "IGCM_sys_RshArchive"
224	}
225
226	#D-#==================================================
227	#D-function IGCM_sys_RshArchive_NoError
228	#D-* Purpose: Archive rsh command, without error
229	#D-* used only in monitoring.job
230	#D-* Examples:
231	#D-
232	function IGCM_sys_RshArchive_NoError {
233	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
234	/bin/ksh <<-EOF
235	${@} 2> /dev/null
236	EOF
237	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
238	}
239
240	#D-#==================================================
241	#D-function IGCM_sys_MkdirArchive
242	#D-* Purpose: Mkdir on Archive
243	#D-* Examples:
244	#D-
245	function IGCM_sys_MkdirArchive {
246	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
247	if ( $DEBUG_sys ) ; then
248	echo "IGCM_sys_MkdirArchive :" $@
249	fi
250	#- creation de repertoire sur le serveur fichier
251	if [ ! -d ${1} ]; then
252	\mkdir -p $1
253	status=$?
254
255	if [ ${status} -gt 0 ] ; then
256	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
257	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
258	fi
259	fi
260	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
261	}
262
263	#D-#==================================================
264	#D-function IGCM_sys_TestDirArchive
265	#D-* Purpose: Test Directory that must exists on Archive
266	#D-* Examples:
267	#D-
268	function IGCM_sys_TestDirArchive {
269	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
270	if ( $DEBUG_sys ) ; then
271	echo "IGCM_sys_TestDirArchive :" $@
272	fi
273	typeset ExistFlag
274	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
275	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
276	return ${ExistFlag}
277	}
278
279	#D-#==================================================
280	#D-function IGCM_sys_IsFileArchived
281	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
282	#D-* Examples:
283	#D-
284	function IGCM_sys_IsFileArchived {
285	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
286	if ( $DEBUG_sys ) ; then
287	echo "IGCM_sys_IsFileArchived :" $@
288	fi
289	typeset IsArchivedFlag
290	# Never archived for this system
291	IsArchivedFlag=1
292	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
293
294	return ${IsArchivedFlag}
295	}
296
297	#D-#==================================================
298	#D-function IGCM_sys_TestFileArchive
299	#D-* Purpose: Test file that must NOT EXISTS on Archive
300	#D-* Examples:
301	#D-
302	function IGCM_sys_TestFileArchive {
303	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
304	typeset ExistFlag
305	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
306	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
307
308	return ${ExistFlag}
309	}
310
311	#D-#==================================================
312	#D-function IGCM_sys_CountFileArchive
313	#D-* Purpose: Count files on Archive filesystem
314	#D-* Examples:
315	#D-
316	function IGCM_sys_CountFileArchive {
317	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
318	ls ${@} 2>/dev/null \| wc -l
319	if [ $? -gt 0 ] ; then
320	echo "IGCM_sys_CountFileArchive : erreur."
321	fi
322	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
323	}
324
325	#D-#==================================================
326	#D-function IGCM_sys_Tree
327	#D-* Purpose: Tree directories with files on ${ARCHIVE}
328	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
329	#D-
330	function IGCM_sys_Tree {
331	IGCM_debug_PushStack "IGCM_sys_Tree" $@
332	if ( $DEBUG_sys ) ; then
333	echo "IGCM_sys_Tree :" $@
334	fi
335
336	\tree -f $@
337
338	IGCM_debug_PopStack "IGCM_sys_Tree"
339	}
340
341	#D-#==================================================
342	#D-function IGCM_sys_Qsub
343	#D-* Purpose: Qsub new job
344	#D-* Examples:
345	#D-
346	function IGCM_sys_Qsub {
347	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
348	if ( $DEBUG_sys ) ; then
349	echo "IGCM_sys_Qsub :" $@
350	fi
351	echo "${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &"
352	${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &
353	if [ $? -gt 0 ] ; then
354	echo "IGCM_sys_Qsub : erreur ${options} $1"
355	IGCM_debug_Exit "IGCM_sys_Qsub"
356	fi
357	IGCM_debug_PopStack "IGCM_sys_Qsub"
358	}
359
360	#D-#==================================================
361	#D-function IGCM_sys_QsubPost
362	#D-* Purpose: Qsub new job on scalaire
363	#D-* Examples:
364	#D-
365	function IGCM_sys_QsubPost {
366	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
367	if ( $DEBUG_sys ) ; then
368	echo "IGCM_sys_QsubPost :" $@
369	fi
370	echo "${libIGCM_POST}/$1.job > ${POST_DIR}/$1.${PeriodDateEnd}.out 2>&1 &"
371	${libIGCM_POST}/$1.job > ${POST_DIR}/${Script_Post_Output}.out 2>&1 &
372	if [ $? -gt 0 ] ; then
373	echo "IGCM_sys_QsubPost : erreur " $@
374	IGCM_debug_Exit "IGCM_sys_QsubPost"
375	fi
376	IGCM_debug_PopStack "IGCM_sys_QsubPost"
377	}
378
379	#D-*************************
380	#D- File transfer functions
381	#D-*************************
382	#D-
383
384	#D-#==================================================
385	#D-function IGCM_sys_RmRunDir
386	#D-* Purpose: rm tmpdir (dummy function most of the time batch
387	#D- scheduler will do the job)
388	#D-* Examples:
389	#D-
390	function IGCM_sys_RmRunDir {
391	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
392	if ( $DEBUG_sys ) ; then
393	echo "IGCM_sys_RmRunDir :" $@
394	IGCM_debug_Print 1 "Dummy call, let the user do that."
395	# IGCM_sys_Rm -rf ${RUN_DIR}
396	fi
397	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
398	}
399
400	#D-#==================================================
401	#D-function IGCM_sys_Put_Dir
402	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
403	#D-* Examples:
404	#D-
405	function IGCM_sys_Put_Dir {
406	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
407	if ( $DEBUG_sys ) ; then
408	echo "IGCM_sys_Put_Dir :" $@
409	fi
410	if [ $DRYRUN = 0 ]; then
411	if [ ! -d ${1} ] ; then
412	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
413	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
414	return
415	fi
416
417	typeset status
418
419	# Only if we use rsync
420	#IGCM_sys_TestDirArchive $( dirname $2 )
421	#
422	#USUAL WAY
423	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
424	status=$?
425
426	if [ ${status} -gt 0 ] ; then
427	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
428	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
429	IGCM_debug_Exit "IGCM_sys_Put_Dir"
430	else
431	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
432	fi
433	fi
434	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
435	}
436
437	#D-#==================================================
438	#D-function IGCM_sys_Get_Dir
439	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
440	#D-* Examples:
441	#D-
442	function IGCM_sys_Get_Dir {
443	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
444	if ( $DEBUG_sys ) ; then
445	echo "IGCM_sys_Get_Dir :" $@
446	fi
447	if [ $DRYRUN = 0 ]; then
448	typeset NB_ESSAI DELAI status i
449	# number of tentative
450	NB_ESSAI=3
451	# time delay between tentative
452	DELAI=2
453
454	#
455	# USUAL WAY
456	i=0
457	while [ $i -lt $NB_ESSAI ] ; do
458	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
459	status=$?
460	if [ ${status} -gt 0 ] ; then
461	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
462	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
463	sleep $DELAI
464	else
465	break
466	fi
467	(( i = i + 1 ))
468	done
469
470	if [ ${status} -gt 0 ] ; then
471	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
472	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
473	IGCM_debug_Exit "IGCM_sys_Get_Dir"
474	else
475	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
476	fi
477	fi
478	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
479	}
480
481	#D-#==================================================
482	#D-function IGCM_sys_Put_Rest
483	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
484	#D- File and target directory must exist.
485	#D-* Examples:
486	#D-
487	function IGCM_sys_Put_Rest {
488	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
489	if ( $DEBUG_sys ) ; then
490	echo "IGCM_sys_Put_Rest :" $@
491	fi
492	if [ $DRYRUN = 0 ]; then
493	if [ ! -f ${1} ] ; then
494	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
495	IGCM_debug_Exit "IGCM_sys_Put_Rest"
496	fi
497
498	typeset status
499	#
500	# USUAL WAY
501	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
502	status=$?
503
504	# #RSYNC WITH NETWORK SSH CALL
505	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
506	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
507
508	# #RSYNC WITH NFS USE
509	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
510	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
511
512	# status=$?
513	# IGCM_sys_Rsync_out $status
514
515	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
516	# (( status=status+$? ))
517
518	if [ ${status} -gt 0 ] ; then
519	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
520	[ -f ${1} ] && ls -l ${1}
521	[ -f ${2} ] && ls -l ${2}
522	[ -f ${2}/${1} ] && ls -l ${2}/${1}
523	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
524	IGCM_debug_Exit "IGCM_sys_Put_Rest"
525	else
526
527	if [ X${JobType} = XRUN ] ; then
528	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
529	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
530	fi
531
532	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
533	fi
534	fi
535	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
536	}
537
538	#D-#==================================================
539	#D-function IGCM_sys_Put_Out
540	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
541	#D-* Examples:
542	#D-
543	function IGCM_sys_Put_Out {
544	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
545	if ( $DEBUG_sys ) ; then
546	echo "IGCM_sys_Put_Out :" $@
547	fi
548
549	typeset status
550
551	if [ $DRYRUN = 0 ]; then
552	if [ ! -f ${1} ] ; then
553	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
554	IGCM_debug_PopStack "IGCM_sys_Put_Out"
555	return 1
556	fi
557	#
558	IGCM_sys_MkdirArchive $( dirname $2 )
559	#
560	if [ X${JobType} = XRUN ] ; then
561	if [ X${3} = X ] ; then
562	IGCM_sys_Chmod 444 ${1}
563	fi
564	fi
565	#
566
567	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
568	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
569	status=$?
570	IGCM_sys_Rsync_out $status
571
572	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
573	(( status=status+$? ))
574
575	if [ ${status} -gt 0 ] ; then
576	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
577	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
578	IGCM_debug_Exit "IGCM_sys_Put_Out"
579	else
580	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
581	fi
582	fi
583	IGCM_debug_PopStack "IGCM_sys_Put_Out"
584	return 0
585	}
586
587	#D-#==================================================
588	#D-function IGCM_sys_Get
589	#D-* Purpose: Get a file from ${ARCHIVE}
590	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
591	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
592	function IGCM_sys_Get {
593	IGCM_debug_PushStack "IGCM_sys_Get" $@
594
595	typeset DEST status dm_liste
596
597	if ( $DEBUG_sys ) ; then
598	echo "IGCM_sys_Get :" $@
599	fi
600	if [ $DRYRUN -le 2 ]; then
601	if [ X${1} = X'/l' ] ; then
602	eval set +A dm_liste \${${2}}
603	else
604	dm_liste=${1}
605	fi
606	eval DEST=\${${#}}
607
608	# test if the (first) file is present in the old computation :
609	IGCM_sys_TestFileArchive ${dm_liste[0]}
610	status=$?
611	if [ ${status} -gt 0 ] ; then
612	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
613	IGCM_debug_Exit "IGCM_sys_Get"
614	return
615	fi
616
617	#USUAL WAY
618	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
619	status=$?
620
621	if [ ${status} -gt 0 ] ; then
622	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
623	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
624	IGCM_debug_Exit "IGCM_sys_Get"
625	else
626	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
627	fi
628	fi
629	IGCM_debug_PopStack "IGCM_sys_Get"
630	}
631
632	#D-#==================================================
633	#D-function IGCM_sys_GetDate_Monitoring
634	#D-* Purpose: get the last year for which the monitoring has been computed
635	#D-* Examples:
636	#D-
637	function IGCM_sys_GetDate_Monitoring {
638	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
639	if ( $DEBUG_sys ) ; then
640	echo "IGCM_sys_GetDate_Monitoring :" $@
641	fi
642
643	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
644
645	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
646	}
647
648	#D-#==================================================
649	#D-function IGCM_sys_Dods_Rm
650	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
651	#D-* Examples:
652	#D-
653	function IGCM_sys_Dods_Rm {
654	if ( $DEBUG_sys ) ; then
655	echo "IGCM_sys_Dods_Rm :" $@
656	fi
657	return 0
658	}
659
660	#D-#==================================================
661	#D-function IGCM_sys_Dods_Cp
662	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
663	#D-* Examples:
664	#D-
665	function IGCM_sys_Dods_Cp {
666	if ( $DEBUG_sys ) ; then
667	echo "IGCM_sys_Dods_Cp :" $@
668	fi
669	return 0
670	}
671
672	#D-#==================================================
673	#D-function IGCM_sys_Put_Dods
674	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
675	#D-* Examples:
676	#D-
677	function IGCM_sys_Put_Dods {
678	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
679	if ( $DEBUG_sys ) ; then
680	echo "IGCM_sys_Put_Dods :" $@
681	fi
682	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
683	}
684
685	##############################################################
686	# REBUILD OPERATOR
687
688	#D-#==================================================
689	#D-function IGCM_sys_sync
690	#D-* Purpose: flush buffer on disk (dummy function on Ada)
691	#D-* Examples:
692	#D-
693	function IGCM_sys_sync {
694	IGCM_debug_PushStack "IGCM_sys_sync" $@
695	if ( $DEBUG_sys ) ; then
696	echo "IGCM_sys_sync :" $@
697	echo "Dummy call, let the system do that."
698	fi
699	IGCM_debug_PopStack "IGCM_sys_sync"
700	}
701
702	############################################################
703	# Activate Running Environnment Variables
704
705	#D-#==================================================
706	#D-function IGCM_sys_desactiv_variables
707	#D-* Purpose: set environement variables prior to execution
708	#D-* Examples:
709	#D-
710	function IGCM_sys_activ_variables {
711	IGCM_debug_PushStack "IGCM_sys_activ_variables"
712	if ( $DEBUG_sys ) ; then
713	echo "IGCM_sys_activ_variables"
714	fi
715
716	# --------------------------------------------------------------------
717	#D- MPI specifications
718	# --------------------------------------------------------------------
719
720	# --------------------------------------------------------------------
721	#D- Other specifications
722	# --------------------------------------------------------------------
723
724	IGCM_debug_PopStack "IGCM_sys_activ_variables"
725	}
726
727	############################################################
728	# Desactivate Running Environnment Variables
729
730	#D-#==================================================
731	#D-function IGCM_sys_desactiv_variables
732	#D-* Purpose: unset environement variables after execution
733	#D-* Examples:
734	#D-
735	function IGCM_sys_desactiv_variables {
736	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
737	if ( $DEBUG_sys ) ; then
738	echo "IGCM_sys_desactiv_variables"
739	fi
740	# --------------------------------------------------------------------
741	#D- MPI specifications
742	# --------------------------------------------------------------------
743
744	# --------------------------------------------------------------------
745	#D- Other specifications
746	# --------------------------------------------------------------------
747
748	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
749	}
750
751	############################################################
752	# Build MPI/OMP scripts run file (dummy function)
753
754	#D-#==================================================
755	#D-function IGCM_sys_build_run_file
756	#D-* Purpose: build run file (deprecated)
757	#D-* Examples:
758	#D-
759	function IGCM_sys_build_run_file {
760
761	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
762
763	}
764
765	############################################################
766	# Build MPI/OMP scripts
767
768	#D-#==================================================
769	#D-function IGCM_sys_build_execution_scripts
770	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
771	#D-* Examples:
772	#D-
773	function IGCM_sys_build_execution_scripts
774	{
775	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
776	if ( $DEBUG_sys ) ; then
777	echo "IGCM_sys_build_execution_scripts " $@
778	fi
779
780	typeset nodes listnodes init_node start_num init_exec comp ExeNameIn ExeNameOut
781	typeset node_num_current node_current comp_proc_mpi_loc comp_proc_omp_loc
782	typeset num_corempi nombre_restant_node nombre_restant_comp
783
784	if ( ${OK_PARA_MPMD} ) ; then
785
786	if [ -f run_file ] ; then
787	IGCM_sys_Rm -f run_file
788	fi
789	touch run_file
790
791	if ( ${OK_PARA_OMP} ) ; then
792
793	# Hosts treatment
794
795	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
796
797	i=0
798	rm -f hosts
799	IGCM_debug_Print 1 "sys $( hostname ), Hosts avaible :"
800	for nodes in `cat hosts.tmp` ; do
801	host[$i]=$nodes
802	echo "${host[$i]} slots=1 max_slots=1" >> hosts
803	IGCM_debug_Print 1 ${host[$i]}
804	i=$((i+1))
805	done
806	rm -f hosts.tmp
807
808	listnodes=${host[*]}
809
810	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
811
812	# Initialisation
813
814	init_node=y
815	node_num_current=0
816	start_num=0
817	init_exec=n
818
819	# Build run_file
820
821	# First loop on the components for the coupler ie oasis (only if oasis3)
822	# the coupler ie oasis3 must be the first one
823	for comp in ${config_ListOfComponents[*]} ; do
824
825	if [ "X${comp}" = "XCPL" ] ; then
826
827	eval ExeNameIn=\${config_Executable_${comp}[0]}
828	eval ExeNameOut=\${config_Executable_${comp}[1]}
829
830	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
831	echo "" >> script_${ExeNameOut}.ksh
832	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
833	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
834	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
835	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
836	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
837
838	init_node=n
839
840	(( nombre_restant_node = NUM_COREPERNODE - 1 ))
841	node_num_current=0
842	node_current=${host[${node_num_current}]}
843
844	EXECUTION="${EXECUTION} -H ${node_current} -np 1 ./script_${ExeNameOut}.ksh"
845
846	init_exec=y
847	start_num=1
848
849	fi
850
851	done
852
853	# Then loop on the components (except for oasis)
854
855	for comp in ${config_ListOfComponents[*]} ; do
856
857	eval ExeNameIn=\${config_Executable_${comp}[0]}
858	eval ExeNameOut=\${config_Executable_${comp}[1]}
859
860	# Only if we really have an executable for the component :
861	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
862
863	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
864	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
865
866	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
867	# echo "set -vx" >> script_${ExeNameOut}.ksh
868	echo "" >> script_${ExeNameOut}.ksh
869	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
870	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
871	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
872	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
873	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
874	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
875	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
876
877	node_num=0
878
879	# We define the number of MPI process to be assigned for the component
880
881	nombre_restant_comp=${comp_proc_mpi_loc}
882
883	# Loop on the allocated nodes
884
885	for node in ${listnodes} ; do
886
887	# We go to the current node
888	if [ ${node_num} = ${node_num_current} ] ; then
889
890	node_current=${host[${node_num_current}]}
891
892	# If first time on the node : initialisation
893
894	if [ ${init_node} = y ] ; then
895	nombre_restant_node=${NUM_COREPERNODE}
896	fi
897
898	# Test on the number of OMP threads
899
900	if [ ${comp_proc_omp_loc} -gt ${nombre_restant_node} ] ; then
901	(( node_num = node_num + 1 ))
902	node_num_current=${node_num}
903	init_node=y
904	continue
905	fi
906
907	# Number of MPI process to assign
908
909	(( num_corempi = nombre_restant_node / comp_proc_omp_loc ))
910
911	if [ ${num_corempi} -gt ${nombre_restant_comp} ] ; then
912	num_corempi=${nombre_restant_comp}
913	fi
914
915	(( nombre_restant_node = nombre_restant_node - num_corempi * comp_proc_omp_loc ))
916	(( nombre_restant_comp = nombre_restant_comp - num_corempi ))
917
918	if [ ${init_exec} = y ] ; then
919	EXECUTION="${EXECUTION} : -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
920	else
921	EXECUTION="${EXECUTION} -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
922	init_exec=y
923	fi
924
925	(( start_num = num_corempi + start_num ))
926
927	else
928
929	(( node_num = node_num + 1 ))
930	continue
931	fi
932
933	# Test on the number of core/process remaining on the node/component
934
935	if [ ${nombre_restant_node} = 0 ] ; then
936	(( node_num = node_num + 1 ))
937	node_num_current=${node_num}
938	init_node=y
939
940	if [ ${nombre_restant_comp} = 0 ] ; then
941	break 1
942	fi
943	else
944
945	node_num_current=${node_num}
946	init_node=n
947
948	if [ ${nombre_restant_comp} = 0 ] ; then
949	break 1
950	fi
951	fi
952	done
953	fi
954	done
955
956	else
957
958	# Then first loop on the components for the coupler ie oasis
959
960	## the coupler ie oasis must be the first one
961	for comp in ${config_ListOfComponents[*]} ; do
962
963	eval ExeNameOut=\${config_Executable_${comp}[1]}
964
965	# for CPL component only
966	if [ "X${comp}" = "XCPL" ] ; then
967	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
968	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
969	fi
970	done
971
972	# Then second loop on the components
973
974	for comp in ${config_ListOfComponents[*]} ; do
975
976	eval ExeNameOut=\${config_Executable_${comp}[1]}
977
978	# Only if we really have an executable for the component and not the coupler ie oasis:
979	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
980	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
981	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
982	fi
983	done
984	IGCM_sys_Chmod u+x run_file
985
986	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
987
988	fi
989
990	else # Only one executable. launch it.
991
992	for comp in ${config_ListOfComponents[*]} ; do
993
994	# Only if we really have an executable for the component :
995	eval ExeNameOut=\${config_Executable_${comp}[1]}
996	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
997
998	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
999	echo "" >> script_${ExeNameOut}.ksh
1000	if ( ${OK_PARA_OMP} ) ; then
1001	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1002	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1003	fi
1004	if ( ${OK_PARA_MPI} ) ; then
1005	# Default : mpirun used if nb_proc gt 1
1006	# pour sortie out/err par process
1007	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
1008	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1009	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1010	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1011	else
1012	# Default : mpirun is NOT used if nb_proc eq 1
1013	# pour sortie out/err par process
1014	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1015	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1016	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1017	EXECUTION="time ./script_${ExeNameOut}.ksh"
1018	fi
1019
1020	IGCM_debug_Print 1 "sys $( hostname ) : script_${ExeNameOut}.ksh contains"
1021	cat script_${ExeNameOut}.ksh
1022
1023	fi
1024
1025	done
1026
1027	fi
1028
1029	IGCM_debug_Print 1 "sys $( hostname ) : execution command is"
1030	IGCM_debug_Print 1 "$EXECUTION"
1031
1032	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1033	}
1034
1035	#D-#==================================================
1036	#D-function IGCM_sys_check_path
1037	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1038	#D-* do not point to an important use directory. Stop immediately in that case.
1039	#D-* Examples:
1040	#D-
1041	function IGCM_sys_check_path {
1042	IGCM_debug_PushStack "IGCM_sys_check_path"
1043	if ( $DEBUG_sys ) ; then
1044	echo "IGCM_sys_check_path"
1045	fi
1046
1047	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1048	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1049	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1050	IGCM_debug_Exit "This will stop the job"
1051	fi
1052	IGCM_debug_PopStack "IGCM_sys_check_path"
1053	}
1054
1055	#D-#==================================================
1056	#D-function IGCM_sys_check_quota. Dummy call here
1057	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1058	#D-* Examples:
1059	#D-
1060	function IGCM_sys_check_quota {
1061	IGCM_debug_PushStack "IGCM_sys_check_quota"
1062	if ( $DEBUG_sys ) ; then
1063	echo "IGCM_sys_check_quota"
1064	fi
1065	IGCM_debug_PopStack "IGCM_sys_check_quota"
1066	}
1067
1068	#D-#==================================================
1069	#D-function IGCM_sys_CountJobInQueue
1070	#D-* Purpose: Check if job_name is currently
1071	#D- running or in queue
1072	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1073	#D-
1074	function IGCM_sys_CountJobInQueue {
1075	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1076	if ( $DEBUG_sys ) ; then
1077	echo "IGCM_sys_CountJobInQueue"
1078	fi
1079	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1080	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: